Remember, as modified transport planes, they don't have a lot in the way of armor. That's one of the main reasons why they switched from 7.62 mm miniguns to 20 mm Vulcans to the new 25 mm cannon - to increase standoff range to make them less vulnerable. The heavier guns also have the added benefit of being able to take out harder targets like armored vehicles.

I found this article on gunships last night and it lists several Spectre losses from ground fire and SAMs. Five AC-130s (4 AC-130As and 1 AC-130E) were lost in the Vietnam war. The article says the one shot down during the Gulf War was hit by a SAM.

Here's a better article describing what happened to the plane, "Spirit 03".