No I don't see...
...other than human stupidity and short-sghtedness there's no impediment to indefinite expansion, just to exponential expansion.
But that assumes no fundamental change in human "nature". Personally I think advances in our understanding of the brain will eventually allow us the option of flesh, robot or software as modes of existence. Which we choose will depend on what we discover about ourselves when we're no longer driven by arbitary and insentient built-in goals.
Adam