Infrastructure costs money, which means higher taxes.
And the rich, the people who pay most of the taxes (at least, that's what they claim) don't like paying for them. They feel that if the poor want roads and bridges (or schools and hospitals), let THEM pay for it.
Of course, once roads and bridges are built, the rich will be happy to use them to move their products to market, to travel to their vacation homes, and to bring police to protect them from the mob.