We have this funny peace of paper called the CONSTITUTION.
It specifies four year terms, with a maximum of two. Only impeachment can legally remove a President before his term expires, and that is a criminal matter, not just a vote-of-confidence thing.
In short, our Constitution just does NOT work that way.
