Not quite so black-and-white?
I'm sure that back in the depths of the cold war, there were a bunch of smart guys who went into aerospace specifically to design better fighters, bombers, and missiles to push back the "Red Menace".
Right now, though, I agree with your sentiments entirely.
The challenge is to get something useful done with this "diversion" money. If we can progress to the point where meaningful manned space missions are possible, and even the first steps towards an off-Earth colony, then it all might become "worthwhile" .