Erosion
In some cases, "orphan" rocks like this, sitting out in the middle of a desert, are the result of the surrounding area eroding away, and not the rock having been "set" there. A million years ago, maybe there was a crater, and nothing says this came directly from space. It could have been knocked out of another crater impact.
I've seen examples of this kind of thing on Earth, and I used to know the official name for the process. I'm sure those here with better geological knowledge can elaborate on this.