You won't fly to Mercury by first going to Mars.
You won't fly to geosynch by first going to Mars. You won't fly to Jupiter or any of the other outer planets by first going to Mars. You won't go to a comet by first going to Mars.
You're going to start from the Earth. From LEO, you can go anywhere, manned or unmanned. That's where a space infrastructure is needed. Infrastructure on Mars is needed to support exploration on Mars, but it isn't useful anywhere else. That doesn't mean that Mars infrastructure shouldn't be built (far from it), just that it shouldn't be oversold.