Here is a very interesting idea:
“Encyclopaedia of the Solar System
” Academic Press Ed Weissman, McFadden and Johnson.
Catastrophic collisions ultimately provide a new influx to the NEO population. The catastrophic disruption of a ~ 170-km-diameter MBA ~ 160 Myr ago led to the creation of the Baptistina family. Over time, dynamic processes (i.e., Yarkovsky Effect) changed the orbits of fragments produced in this collision such that they could strike the terrestrial planets. It has been suggested that one such fragment may have hit Earth and led to the extinction of the dinosaurs 65 Myr ago (K-T boundary
MBA = main belt asteroid
Thus collisions between asteroid family members in the main asteroid belt are (comparatively) numerous, and can send sizeable fragments into the inner Solar System.
The point is that "killer asteroids" do not have to be rarities from the outer SS, but can be produced at almost any time (relatively) close at hand