Good eye
You're half right. In compact mode, the big graphics in the header aren't loaded, but the four small graphics that make up each buttons set are still loaded. There are some arcane technical reasons for that, and some of the fixes I can think of at the moment might actually slow it further (paradoxical, I know, but trust me on that, lest I be provoked into speaking Geek at you).
Still, I'll file that one away as a possible optimization if I can find a way to do it.