It was far more than just a handful of astronauts and cosmonauts.

Hundreds died when the rocket explosion that became known as the
Nedelin Disaster happened in the former USSR in 1960:

http://www.russianspaceweb.com/r16_disaster.html

I recall several technicians dying while preparing various Space Shuttle missions.

And then there was the sad case of an engineer committing suicide after the
Apollo 1 tragedy in 1967 by parking his car on the railroad tracks (with his family
aboard) in front of an oncoming train.

So it's been a lot of folks on all sides sacrificing their lives for the dream on
many levels, not just the ones who go into space.

But that is ultimately what exploration and evolution are all about.