You're comparing satellites whose primary mission was carrying sensors to a satellite whose primary mission is communications relay, with a secondary navigation capability. MTO apparently won't carry any sensors like Mars Odyssey and Mars Reconnaissance Orbiter. It will carry a laser communications experiment and a rendezvous demonstration experiment. The laser experiment is very interesting. Here are some details. If it works, a 5 watt laser will be able to send back 10 times the data of current RF systems. You won't need nearly as big a satellite to host such systems.

According to this source

MTO is also to function as a Mars-based navigation spacecraft to help guide robotic spacecraft approaching Martian orbit, as well as navigate new rovers on the surface. And it's also to release a Martian subsatellite that MTO will fly with to demonstrate Mars orbit navigation and rendezvous, which is important for future sample-return operations.

The NASA Jet Propulsion Laboratory in Pasadena, Calif., is heading the MTO project in connection with the Goddard Space Flight Center in Greenbelt, Md., and the Massachusetts Institute of Technology's Lincoln Laboratory. NASA plans by early March to award a contract for the basic MTO spacecraft, says JPL project manager Roger Gibbs.

The satellite bus and telecommunications relay system will cost about $550 million, including launch, while the Goddard/Lincoln laser communications system will cost $170 million. The rendezvous demonstration will cost $80 million.

It appears that MTO will cost approximately $800 million. While the laser experiment is very interesting, the whole MTO system is quite expensive to put all of your bandwidth in a single package, e.g. Mars Observer.

According to this source

The spacecraft design concept is currently being studied. At launch, the spacecraft is expected to weigh about 2 tons, most of which would be propellant to be expended getting into Mars orbit. On orbit, the spacecraft would span more than 7 meters (23 feet) across. Its most prominent feature, besides the large solar arrays required to power it, would be a large dish antenna with a diameter in the range of 2 to 3 meters (7 to 10 feet). From Mars orbit, that narrowbeam antenna would communicate with Earth-based antennas in NASA's Deep Space Network in both the X-band and the Ka-band of radio frequencies.

For the same 2 tons of launch mass, you could have perhaps 3 smaller satellites separated 120 degrees apart in the same orbital plane. Each would have a substancial communications capability. You'd also have better navigation capabilities, better coverage, and much better redundancy than you'd likely have from a single satellite system. For several scenarios, having continuous coverage at a high data rate is better than having partial coverage at a very high data rate.

Sometimes, as the saying goes, "quantity has a quality all it's own."