I do agree with you to a point.
It would take a very large rock to wipe out Humanity. One big enough to shatter the planet. The people who talk about us going the way of the dinosaurs are idiots, IMO. Dinosaurs and people have very little in common, in terms of survival. Our species would easily survive the strike of a 6 mile wide asteroid. If turtles, snakes, crocs, gators, birds, sharks, and countless other dumb species can do it, so can we. With that being said however, we should still prepare to keep one from hitting should it be detected.