Sure, asteroids did some of the work. But a few came from the fringes of our Solar System — and were kicked in by other stars.