I wouldn't worry about that.
I don't believe he believes that.
If he does, I don't think he's aware of it.
If he's aware of it, he's too smart to repeat it around unfriendly witnesses.
It's either a deliberate smear by his foes or, at worst, it's reverse propaganda (where you accuse your man of an outrage he can easily disprove, and he comes off smelling like a rose and his enemies as liars).
Besides, snobbery and elitism are not restricted to any political agenda.