Interesting question. Solid rocket motor casings have to be able to withstand the internal pressures of combusion, so perhaps they're more rigid. That might transmit more vibrations (would there be internal shock waves that might cause vibrations?) such as flight loads to the crew.

In the case of the Shuttle, thrust is at its highest when the SRBs are burning. Once they drop off, the thrust is much lower. That might account for at least some of the difference in vibration levels.

As for using an SRB as the first stage of a CEV design, there might need to be some tailoring of the internal propellant geometry to better control flight loads. For example, the first few segments might be hollow core for high initial thrust. After that, the segments could be solid resulting in a lower, steady thrust level. If the SRB thrust level stayed constant for the entire burn, the acceleration levels near burnout could get pretty high.