It's not so much the proximity, but the size.
How big is it, Raoul?
Some consider "space rocks" only 10-meters in diameter 'asteroids'.
Now!...If the 'asteroid' in question
were about 100-200 meters in diameter
at least, and maybe between 12-100 million metric tons of mass [minimum], then I would be concerned.
For example:
KE = 1/2MV^2.
M = [min] 1.2 x 10^10kg
V = [min] 12,000 m/s
KE = 1/2[1.2 x 10^10]1.44 x 10^8
KE = 10^18 joules.
A hydrogen bomb of about 2.4 MT yield
releases about 10^16 joules of energy.
So a 100-meter diameter "space rock"
would unleash the energy of 100 H-bombs.
Jacob