I agree - but here's a thought
Given that the first humans all had the same skin color (which was closer to brown than black - why do we call it 'black' anyway?) as consciousness arose among the first primitive tribes, the idea of skin color as an identifier, and later as a social classifier, would have been completely meaningless. No one would have pointed out skin color as meaning anything any more than a cloudless sky being blue meant anything.
Perhaps in 1,000 years or so the human race will be back in the same situation. Hope so.