It certainly has an effect.
But shut down one of the refineries because of any one of a number of factors, and you will see the price of gasoline skyrocket, even if the price of crude falls.
The supply of gasoline is limited by what the refineries can produce. If demand exceeds that supply, for even a short period of time, then the price of gasoline will quickly increase.
The biggest factor for the economy (resulting from higher crude prices) is that the balance of trade suffers: If you want to see the US trade deficits continue to get worse, oppose drilling for new sources of oil and/or discourage conservation efforts.