Threat of damage on the ground
Good point on why canons were not a good idea. They are one chance while the target is passing over vs picking our shot with aircraft. As for damage to the ground, according to the pentagon, the balloon carried equipment equivalent to three buses. That could have easily destroyed a house or done serious damage.