"Where's the line?"
Tell me about it. That's the problem that leads to abuses by social workers who suck at their job, and makes all the hair fall out of anguished social workers who are responsible and conscientous.
At what point do you decide a kid is better off away from their parent? The wrong choice one way breaks up a family. The wrong choice the other way can end up with abused kids, or dead ones. There aren't any red lights that flash on top of a bad parent's head to make it easy.