I think he was a pessimist at heart, yes.
He spent the better part of his life believing that people would take the easy way unless things became unbearable. That's why he was such a brilliant diplomat.
He was the one who brokered the compromise on slavery, even though he himself founded the first abolitionist movement in the British colonies.
He understood that one could dream big, but that expectations were a different animal altogether.