I don't see the genetic mixing to be as pronounced as you say.
It probably happened to a certain extent, but if it were as common as you suggest then there would be no regional differences in populations at all. The simple fact is that most people stay put. They are born and die in a fairly small geographical area.
A few brave souls would wander out into the bigger world, but how long would it take for their small genetic contribution to be smothered by the genes of a native population? Given high infant mortality rates and the harsh conditions of moving into foreign lands, I think the odds would be against them.