Far more stupid than racist
The comments were just thoughtless noise and he may deserve to lose his leadership position. However racism is overtly working to deny rights and/or opportunity, not some dumb statement designed to please the audience.
By this criteria, if every similar statement made out of the limelight were made public, 90% of congress would be branded racist. People say the darnedest things when they are out of earshot