In WW2, what mattered was to get the occupants back into their country. Once that was done, were there attacks against the liberators in the occupied zones ,from the locals? None! We were attacked by V2s until complete cessation of hostilites, but nobody attacked the liberators in our countries. We were freed, after all!
In Iraq, you can't kick the former regime supporters away as they're part of the country.
So, even though Saddam has been captured months ago, occupying the country is felt as
1) An occupation and everything that goes around it (fear of being stormed by troops at night because a jealous neighbour denounced you, for instance).
2) A humiliation as it shows at best that you can't take care of your own country or at worst that your culture, your upbringing are worthless.
3) A liberation; certainly a number of Iraqis are much better off now.

But it's always a minority of people that is bringing chaos into the country and affecting the political decisions.

The question is: if it's indeed a minority, then why not leave as soon as possible and let the country choose its own direction, even without US troops out there?

If the US doesn't do this, it means it is influenced by a minority of iraqis. It can also mean the removal of Saddam wasn't the objective, but regime change.
For the moment, it looks like the Sunnis and some Chi-ites are uniting against the occupants. If the occupants are gone, there will be a civil war or an Islamist state.
The price to pay for removing a dictator isn't what we expect.

Raoul