re: What bothers me...
It's physics. Remove the need to return at high speeds and you will remove the worry.
But, I point out during Columbia's reentry we weren't particularly worried. And any experimental aircraft has an inspection at each landing and orbit is a "landing" where they never bothered to inspect the spaceplane.
BTW, the CEV doesn't remove high speed reentry. A space elevator would, but then you would need to shield against radiation.