So both young men and women would both start out life with older spouses; at about age 40, they divorce and marry a younger partner.

Later, both men and women at around age 60 would have to give up their younger (now around 40) spouses and start shopping for mates in the 60+ range.

Say Jack (20) marries Jill (40) and Sally (20) marries Sam (40).
Twenty years later, Jack (40) marries Jane (20) and Sally (40) marries Steve (20). This leaves Jill and Sam (now both 60) free to marry each other.

The drawbacks, following the usual societal stereotypes and prejudices: young men would get stuck with unattractive women, and old women would have to put up with immature men. By the time they reach 60 they'll eventually wind up marrying the person they wished they'd wed at age 20.

The mind boggles.