That's a bit misleading, even if true.
In Europe and England in those times, many children were raised by nannies, nurses, or governesses, or shipped off to boarding schools. The children's poetry and stories of the time pretty much assumed this.
"I’d think shame to stick to nursie as that shadow sticks to me!"
No doubt this created a lot of problem children, and problem adults, but some of the greatest people of the time were raised no differently.
In the same way, great people have come from the poverty, as well as criminals. When you finally get down to it, choices have more to do with it than anything else, a conclusion that is not comfortable for many people. I suspect a lot of men had backgrounds similar to Hitler's without becoming tyrants.