If aliens had been here, surely there would be evidence on the moon. Artifacts. Footprints. Something. But we haven't found anything. So I agree with what you are saying, that interstellar travel is extremely difficult, maybe next-to-impossible. ETIs simply don't undertake the task often enough to leave a footprint, so to speak.

So there is a good possibility that:

1. There may be trillions of ETIs in the visible universe.
2. It is next-to-impossible for them to interact in any manner that we currently understand.
3. The universe maximizes and preserves diversity in this way.
4. It is unknown if there is anything "more" to this existence, or even if current conditions that give rise to life are indefinite or merely a temporary window of opportunity amid randomness and chaos, both spatial and temporal.

And after saying all that, it's hard to say if I've said anything at all! LOL.