Perhaps the biggest impact would be economic
It'd cost a fortune to do the clean up following a dirty bomb detonation. Would we do like at Love Canal and buy up everyone's property? Would we follow the Superfund model where legal fees exceed the amount actually spent on clean up? To what level would people demand the site be cleaned, anyway? A dirty bomb in a major city would almost certainly lead to billions in lost economic activity and clean up costs.