The phenomenon you describe, polar wandering, and the precession of the equinoxes are two totally unrelated and independent phenomena.

Polar wandering is the location of the earth's axis actually moving randomly relative to the crust of the earth, it is at most a few meters a year as you point out, although this does not rule out catastrophic shifts in the past. You can think of it as the earth wobbling due to shifting masses inside, or perhaps by the crust slipping on the mantle. I don't think anyone really knows why it happens and I don't know how well polar wandering in the past has been mapped.

Precession of the equinoxes is caused by the earth's axis actually moving relative to the distant fixed stars, The planet is like a spinning top leaning over 23.5 degrees from the ecliptic plane, but the top is wobbling, each wobble taking about 25.8K years, so the axis traces a big circle on the celestial sphere. That circle just happens to be passing near the star Polaris now, in about another 13K years the axis will be pointing at a spot 47 degrees distant from it. Precession is very well understood and predictable, and is caused by the gravitational tug of the moon on the earth's equatorial bulge.

Precession would not cause the pole to move relative to features on the earth's surface, it's the whole earth that's wobbling, not just the axis.