In a rocket launch, the sooner you get out of the atmosphere, the better. The extra Isp you pick up from an airbreathing engine has to be balanced against both the thermal and drag issues you mention, and also the problem of slowing the air down to the point where it can be burned. A rocket hauls its oxidizer along with it. An airbreather has to accelerate the oxygen up to close to the vehicle's speed to burn it properly, at least in the near-term concepts I've seen.

Trying to get a vehicle that uses atmospheric oxygen up to near-orbital velocities is as difficult and pointless as trying to develop a supersonic submarine.

On the other hand, I have seen workable concepts using airbreathing for the very early launch regime, and moving to pure rocket soon after that.