The ability to pitch a rock
It depends on your vision of space travel and colonization. You may be right in the way you apparently see it.
For me, I see colonies as drop off points that could not be a threat to Earth for centuries. They will be too busy making a go of it to think about what is going on back on Earth. Some may be dependent on Earth for trade and support but others will need to be totally self sufficient . It could be many generations before a colony would have the resources for even localized space travel, much less interstellar travel.
The time might come when the galaxy becomes small enough for political squabbles but it would be so far in the future the old differences would be forgotten.