It is theoretically possible for the sun to fly through a cloud of space debris and gravitationaly capture some of that material, but it is extremely unlikely. Basically, what you have is two bodies traveling through space on different orbits, approaching each other from infinity at escape velocity plus, and as they come together they will gravitationally interact. The result is either a collision, or the two will slingshot around each other and head off in different directions. For the two bodies to wind up in orbit around a common center of gravity is dynamically impossible unless there is some third body interacting with the other two to absorb some of the excess momentum and energy. This is why an approaching space vehicle cannot be "captured" into orbit by another planet...it has to either fire its rockets to slow down, or it has to lose excess energy through atmospheric braking, or by slingshotting around a satellite of the planet.

Several bodies in the solar system probably actually did this, some of the retrograde moons in the outer planets come to mind, but this is probably a result of a collision long ago that knocked a planet or satellite over on its side or even shattered it into fragments and sent stuff flying around chaotically. In the resulting confusion, several of the pieces interacted in just the right way and was "captured". In general, interacting bodies either collide, or are ejected away from each other, the conditions have to be just right for them to wind up in orbit, and it usually requires an external boost or nudge to balance the energy and momentum books.

As for the retrograde rotation of Venus and Mercury, this is due to tidal forces caused by gravitational interaction with the Sun, similar to the way the moon is tidally locked onto the earth, so the same side is always facing us. Its a different mechanism all together

I'm not saying that capture is impossible, just highly unlikely. In the early solar nebula, there was all sorts of confusion, tangled orbits, collisions, ejections, and stuff bouncing around. No doubt some captures took place, but they could easily be identified today because of their retrograde, highly elliptical orbits and backwards rotation. But in general, the vast number of solar system objects revolve AND rotate in a counter-clockwise direction (as viewed from the north ecliptic pole). This is a consequence of the original rotation of the protoplanetary disk. After the system stabilized, subsequent collisions may have knocked over or spun backwards some rotating bodies, but they are few and far between, and most are small planetoids and Oort Cloud and Kuiper Belt objects.

Now consider the sun flying around the orbital path of its great revolution around the galactic nucleus, it, and its surrounding disc stars are flying together like a cloud of buckshot. If this stream of stars suddenly plows into another star stream left behind in the wake of some colliding dwarf galaxy in the past, the sun and any of the stars or debris in that stream will be approaching at colossal relative velocities (even though their velocity relative to the galactic center may not be all that high. A capture is highly unlikely.

Astronomers have spent a lot of time studying the "proper motion" of stars, that is, their motion relative to the sun.
We can measure the line-of-sight velocities with spectrometers (looking for red and blue shifts) and their velocities across the line of sight by carefully measuring the motion of their images on photographic plates taken decades apart. The studies have shown two distinct populations of stars, the disc stars rotating along the disc with the sun, and other stars that belong to a population of randomly distributed orbits around the galactic center. This tells us that when the galaxy first started off as a roughly spherical cloud of gas, the "halo" stars formed in a huge spherical volume. After the gas disc flattened out due to its rotation, stars formed later remained in the plane of the galaxy. From this point of view, we see the sun's disc population companions traveling along with us at relatively low velocities, say ,several tens of km/sec (relative to us). The halo stars flying through the disc right now from all over the place, above and below the disc seem to be moving at hundreds of km/sec. Actually all these stars are moving at approximately the same speeds (relative to the galactic center) but the ones floating along with us in the stream appear much slower because we're moving along with them.

Again, that Wikipedia article on Stellar Populations covers this material very well. The fact that stars could be divided into distinct populations due to their kinematic behavior, and that this was highly correlated with their chemical composition as determined through spectroscopy, gives us clues as to just how the galaxy formed billions of years ago.