That is, they are mounted on a pivot that allows them to be swiveled a few degrees in several directions. A rocket with a single engine gimbals in all directions, while sometimes multiple engine rockets have outer engines that only gimbal in limited directions.

The gimballing is done by hydraulics or powerful electric actuators that can change the direction of thrust very quickly. The rest is just high-speed calculations in the guidance system which tracks the attitude and other factors of the rocket during the launch. Flexible bellows in things like fuel supply lines accommodate the movements.

This is a video of an SSME (Shuttle engine) in a test stand, exercising its gimballing. It's a Liquid Oxygen/Hydrogen engine, with a near-invisible exhaust at the outlet. The Saturn, which burned LOX/RP-1 (sort of kerosene) had a brilliant yellow-white exhaust at the outlet (SSME vs. F-1 Test Videos)

It's possible to adjust attitude with varying thrust from multiple engines, but a lot harder.

Other techniques that have been used include graphite vanes in the exhaust (V-2) or injecting a liquid into the side of the exhaust stream to deflect it, often used in solid propellant rockets that don't gimbal easily.