A simple answer
Others have already touched on it. There's no reason a supernova explosion (the origin of many neutron stars) would have to be symmetrical, and I suspect it wouldn't take much of an off-balance detonation to give the remnant a pretty good velocity.