Obama addressed this during the election.
If he constantly defends himself against these insane accusations, it draws attention, time and resources away from the real business of the country. It is also obvious that nothing he can say is going to make a difference to the pathetic dummies who are still chirping about this issue. They'll still be chirping long after his term(s) is done. Some people just have to chirp.