Rare or not, it would only take one
If your argument was that supernovas are rare and therefore not a threat, I would still want evidence that there weren't any giant stars in the neighborhood that might go supernova. The rarity of the event would be secondary to the proximity.
Unfortunately, comets are not readily visible like stars are. Yes, impacts are rare, but how close is the nearest one. If you can't tell me where the comet is, you can't tell me if the next impact will be today, tomorrow, or in 1000 years. "Rare" by itself doesn't reduce the threat. Need more information.