From the MOC link I provided above:

"MOC has considerable capabilities to compress images to optimize on-board storage and downlink utilization. Three different forms of compression are available. During acquisition of images (i.e., in realtime), a predictive compression technique can be employed, in either lossless or "lossy" form, to acquire more data than would otherwise fit in the MOC buffer. The predictive compression is implemented in hardware for the NA camera, and the same algorithm can be applied in software to the WA. Lossless compression factors from 1.5 to 2.5 are possible; with loss, the factors are larger but image quality is degraded, primarily by contouring that results from insufficient bits to represent the actual image gray levels. Once data are in the buffer, they can be compressed (decompressed and then re-compressed if previously compressed) using two transform compression techniques: Walsh-Hadamard (WH) and Discrete Cosine (DC). WH compression can be used to compress images when time is critical, as it is somewhat faster than the DC transform. Compression factors of 3 to 5 are possible with acceptable degradation (mostly in the form of "blockiness" associated with the loss of high frequency information). DC provides the highest compression factors with the least degradation, but is also the slowest, taking nearly 30 minutes to compress a typical image (2048^2). Compression factors of 10:1 show little visual evidence of degradation, although they may have increased radiometric noise (in the 1-3 DN range). Visual degradation of DC transformed images includes "blockiness" associated with the 16 X 16 pixel transform blocks that become obvious when high frequency information is reduced in compression. Figure 3 shows some examples of applying various compression factors to a representative scene."

Going to a higher compression algorithm would mean losing more data. The MGS algorithms were selected to return the highest quality data within the limitations of memory and processing capability. Furthermore, these algorithms are burned into UV-EPROMs, making it difficult if not impossible to change them. As it is, they can do lossy compression similar to JPEG at the expense of losing some data and taking longer to process. Everything is a tradeoff. We can't change any of the hardware currently on the MGS, so there are limits on what we can do to change things.

Buffer size and processing time are additional constraints that we can't change. An unprocessed 2048x2048x8 bit image takes 4 MB. The initial compression can reduce that by a factor of 1.5 to 2.5, meaning the intermediate image size is 1.6 to 2.67 MB. One constraint is whether the MOC is capable of compressing one image while capturing another. If it isn't, then you're limited by the buffer size on how many images you can take at a time. Even if the system can do simultaneous processing, the limited buffer size only allows for a small number of images at a time, probably less than 10. There's a further constraint - orbital period. MGS's period is about 2 hours, so only 2-3 images can be processed for the highest compression ratio and still be transmitted on the next transmission opportunity. Rest assured that JPL and Malin have calculated the optimum image acquisition, compression, and transmission schedule possible for MGS.

On the subject of things we can't easily change, the same goes for the data rates. Data rates are dictated by signal-to-noise ratios. If we were designing a new vehicle, we could put a higher gain antenna and perhaps a more powerful transmitter on the spacecraft to boost the data rate. Of course, a higher gain antenna weighs more and requires more precise pointing due to a narrower beamwidth. A more powerful transmitter requires more electricity, probably generates more heat, and odds are weighs more. Nothing is free.

Unfortunately, we can't change the MGS's antenna or transmitter so we're stuck with what's up there now. The MGS downlinks were designed based on the capabilities of the Deep Space Network. Since we can't change the MGS, the only way to substancially increase the data rate from MGS would be to enhance the DSN by enlarging the antennas, upgrading the electronics, and/or perhaps by employing some new processing. None of that is cheap and I don't know where we could get the funding. It might be possible to point more than one antenna at MGS but the DSN is already being pushed pretty hard to support the current mission load.