There are many Africans of high intelligence, but I do think that your notion of climate could be pretty close. I believe that the ice ages did help to concentrate "intelligence" genes among the Eurasian populations. During the various ice ages vegetation was extremely sparse and easy food sources were very hard to come by. Add to this extremely cold weather and the need to develop technologies to cope, and one can imagine how those less cunning and resourceful would be at a very big disadvantage. Considering we're talking about fairly small and isolated communities, it's understandable that over thousands of years the gene pools would be skewed towards the higher end of the spectrum.

Another idea that I find very plausible is that Africa is devoid of large river systems. Throughout history advanced cultures developed along large rivers. These rivers are believed to have been used as highways to connect larger numbers of people who would have normally never met. This allowed for the exchanging of knowledge and ideas. The only large river in Africa is the Nile, and we can see what kind of cultures developed along its banks.

So climate and geography probably has played a large part in Africa's backwardness.