If you sail off to far, you’ll fall off the End of the Earth
The human race is a very clever and adaptable species. We evolved in the warm African Savanna, yet we have adapted to survive in every climate on the Earth. I don’t think space will be any different. We learn, adapt and grow.
If it makes you feel any better, eventually the nay-sayers are almost always proved wrong.