You have a good point..
They have intimidated the population into not divulging information about them and they don't represent the major portion of the population.
And they haven't declared war against the government. They wouldn't take over if the government were overthrown.
In Afghanistan, for instance, if the US left Karzai, as corrupt as he is, would be gone in an instant and the country would disolve into 5 major tribal areas. It is the people we are fighting against, not some militant sub-sect.