One thing to keep in mind is that on Earth we measure temperature 3 feet above the ground and in the shade. In a nearly airless world such as Mercury and the Moon [even Mars] we instead measure the surface ground temperature.
And because we have volcanoes in Earth, obviously we have small areas on Earth with a hotter surface ground temperatures on than Mercury. Some humans work near volcanoes. Humans also work near fire and other intense heat which hotter than temperature on Mercury get- fireman and steel workers.

Now, the continent of Antarctic resides almost completely within the Antarctic circle; the continent of Antarctic is larger than the US. So the land area on Mercury at both poles is probably more area than 3rd or 4th largest country on Earth. In addition Mercury's rotation is such that one could walk fast enough to keep the Sun at the horizon- therefore if you moving the entire surface of Mercury could available at cooler surface temperatures.
The solar energy at Mercury is intense- it is about 9 times that on Earth, but solar energy on earth is fairly weak. If the solar energy on earth so twice or three times stronger, then Earth would be Venus-like, but assuming we could handwave that away, we could then have cars that ran on solar panels. On Mercury having solar panel driven vehicles could quite doable in terms of energy density.

For quiet living one could have a house or base in Mercury polar region in which one had constant sunlight available for constant power. The main excitement or extreme danger would be from solar flares- obviously they will be more intense at Mercury distance and one would have less warning time- a min or two. Bigger problem is you are mobile and rather stationary.

The main problem with Mercury is it's "distance" from Earth in terms of delta-v and time. Without considering orbital mechanics in terms of distance, Earth orbit is 149,600,000 km from the Sun. And Mercury is 57,910,000 km from Sun, so at it's closest it is about 90 million km. Whereas Mars is orbit is 227,940,000 km, so Mars in terms simply drawing shortest line gets nearer- it's closest is 70 million kilometers. But with interplanetary travel without warp speed or even sub-warp speed, the gravitational affect of the Sun is a significant factor- one has to take into account that planets are in orbit around the Sun. Or in other words if you had the capability of traveling in vehicle capable of 20-30 km/sec [not Star Trek, but way beyond current capability] simple straight line distance could be significant.
In simple terms with same technology capability that gets humans or machines to Mars in 6 months, it will take a year or more to get to Mercury. If one could get to Mars in two months, that might quite a different situation in regards to getting to Mercury. Or saying it differently, 10-20 years after first sending human to Mars, sending humans Mercury might be twice as dangerous or perhaps about same level of danger involved.

Specifically regarding "The main problem with Mercury is it's "distance" from Earth in terms of delta-v and time." What is more important than simple distance is the orbital velocity of the planets:
Earth:
Mean orbital velocity (km/sec) 29.79
Mercury:
Mean orbital velocity (km/sec) 47.88
Mars:
Mean orbital velocity (km/sec) 24.13
http://www.solarviews.com/eng/homepage.htm

Mercury's year- time it take to complete one orbit is 1/4 of the time [87.969 days] as compared to Earth and it's velocity is 1.6 times faster.
Rather doing the math, this calculator:
http://www.exodusproject.com/NavComp/index.html
Shows it takes total of 17 Km/sec from Earth orbit. 7.5 km/sec to leave earth orbit and 9.6 at Mercury orbit and 3/1/2 months. It of course it is same in terms of total delta-v and time to leave mercury to get to Earth: 17 km/sec. but takes 9.6 km/sec to leave Mercury and 7.5 Km/sec at earth which could be aerobraked unlike at Mercury.
This is the least delta-v to go directly to Mercury, though one could more or less go directly towards Mercury and take one gravity assist from Venus on the way- thereby requiring less time and/or delta-v- in other words from Earth you could reduce the 7.5 km/sec leg, but still would more or less still have the 9.6 Km at Mercury.
The least delta-v to Venus from Earth it's 2.5 Km/sec, but takes 4.8 month get to Venus. So if instead you used somewhere around 5 Km/sec, intersect Venus sooner, you could probably get to Mercury in less than 5-6 months. Of course we talking around total 15 Km/sec from Earth Orbit- which would probably get to Mars in less than 3-4 months.

Mercury: Equatorial escape velocity (km/sec) 4.25
Mercury's Delta-v to Venus is 6.7 km/sec and gets there in 2.5 months.
So, for something to leave Mercury and get to Venus requires around 7 km/sec delta-v. And probably have that launch window every 6 months or so, and to intersect earth, maybe every two years. If rocket has greater than 7 km/sec delta-v it of course can get to Venus in a shorter time and have more launch windows with Venus and Earth.

So, that pig, known as the Shuttle could launch from Mercury and get to Mars, if it intersected and got gravity assist from Venus and Earth.
The Shuttle loses about 2.5 km/sec due to gravity losses and Earth's atmosphere- On Mercury these losses would less than .5 km/sec. And total delta-v of about 10 km/sec. Now, if you launched the Shuttle from the Moon, it could get to Mars quicker than from Mercury. But modify the Shuttle a bit- tweeked it- saw off some wings, etc:), and the race was for Jupiter or beyond- from the Moon and say, both vehicles only has 12 Km/sec- one would have closer race between the Moon vs Mercury, because to get to Jupiter from the Moon, one is going have to lose orbital velocity to hit Venus, then hit Earth for another gravity assist.

Now, with 12 km/sec from the Moon one could go directly to Jupiter- you need 8.8 Km/sec and it takes 2.7 years. And with Jupiter's long year you could had a launch window for this just over every earth year. According to that earlier calculator:
http://www.exodusproject.com/NavComp/index.html

Now with the Cassini-Huygens which did flyby of Jupiter it did it's last gravity assist from Earth on 18 August 1999 and reached Jupiter by 30 dec 2000- so it took about 1 1/2 years to get from Earth to Jupiter. And it seems it had two gravity assists from Venus and one from earth
http://saturn.jpl.nasa.gov/mission/gravity-assists.cfm
I'm not sure what it's delta-v was relative to Earth.

But I have some idea of like of delta-v you could get from Venus and Earth from Galileo mission:
"The additional delta-V needed to get to Jupiter was provided by the planetary flybys (2.0 km/sec (4,320 mph) from Venus, 5.2 km/sec (11,600 mph) from the first Earth flyby, 3.7 km/ sec (7,992 mph) from the second Earth flyby)."
http://www2.jpl.nasa.gov/galileo/faqnav.html#faster1

So it seems to me, that if you used rocket that gave around 12 km/sec from surface of Mercury you flyby Venus in a about month time, then go on pass Earth for gravity assist, could gain about 6-7 km/sec from both gravity assists and then get to Jupiter in around a year or so.