Or marrying a non-favorable female
Like the Duke of Windsor who married a divorced American woman.
I had wondered if Prince Charles marriage to what'shername might not get him passed over but evidently now. When he is King, there won't be a Queen. I believe they call a King's wife who is ineligible for the title of Queen, Princess Royale.
At any rate, Harry is behind his brother in line for the throne. Prince William is a good kid, too. Princess Di did a good job with them. And Prince Charles hasn't ruined them, so far at least.