There are 2 things we know about asteroids or EGA's: they have a very small gravity and they require a solar orbit for spacecrafts, no matter how close they are.

With a very low gravity, we have no idea what would happen once an astronaut steps or anchors himself on it. Will there be a lot of dust moving around? Much more than on the Moon?
We should first land a probe on one of them, before (but even then, can we be sure the same things will happen on any other asteroid?).

NASA has never shown enthusiasm for very long missions aboard ISS (even though they could have and that way could have saved money, after Columbia's loss).

Going to the Moon would mean much shorter trips, exploration on the surface of a world anybody can see (and relate to).