More than radiation
Uranium and plutonium are also the heaviest metals with half-lives measured in any more than seconds - as such they are also chemically toxic in addition to the ionizing radiation. I have no idea what the LD-50s are - I wonder where they might be found, and if looking for them on the Web would raise a red flag in Ridge's office...
Anyway, that's what makes the 'dirty bomb' so insidious, combined with the radiation and - as you astutely point out - the panic.
E'