That simple notion is the first Rule of Toxicology. It's also the name of a good book discussing the issue.

"The Dose Makes the Poison: A Plain Language Guide to Toxicology, 2nd Edition"

(Also used as the text for college courses like this one.

The notion that "the dose makes the poison" is at the foundation of toxicology. Some chemicals can be completely harmless at low levels but extremely toxic at high levels. The central question for any given chemical is at what point it becomes dangerous.

As in everything else in life, there are opportunity costs associated with environmental cleanup. Simply put, the money spent on the cleanup is no longer available to do other things, including things that might have a more pronounced affect on public health. Forcing an environmental cleanup to points well below the threshold of danger results in spending a lot of money for no noticeable health effect. Indeed, that money could've been used to better effect in saving lives, such as better prenatal care for low income mothers, better childhood innoculation programs, improved highway safety, etc.

What is happening all too often is that we're being driven by environmental hysteria instead of science. Some are trying to address this issue but they seldom get the coverage of the scare mongers, e.g. Alar or arsenic in drinking water.