Large black holes probably don't grow significantly from other black holes or stellar bodies instead they grow from gas. I assume there could be a shortage of something similar to gas at the sub-atomic level.
But perhaps there is something that feeds these mini blackholes usually at some uniform rate, and without being fed, they would evaporate.

Also in the marco universe gravity is the significant force, and matter clumps together into galaxies, etc. In micro because of the other forces one could have a more uniform distribution. But even if use our galaxy as model, and removed anything smaller than say our moon, there would not be significant growth due to moon size object or larger joining together.

It seems to me the problem would be evaporation rather than excessive amounts of joining to together