Your argument is valid, except for the fact that
There is a misuse of the word “hole,” black or otherwise; a “hole” is usually emptier inside than its surroundings.
The “hole” such as you describe, would in my opinion, be more like “dark matter,” after all, it is made out of matter.
I would describe the center of a galaxy as a black hole, because the galaxy is expanding and leaving “emptiness” behind.
Which brings me to the conclusion that “emptiness” has “potential (for) energy.”