Which opens up a whole new can of worms.
There have been several skeletons found that have traits from both groups, and yet no trace of Neanderthal genes has been found in present day populations. Could the offspring have been proverbial mules? Maybe interbreeding was possible, but the result was a sterile hybrid. It is interesting that all of the hybrid skeletons found to date have been from children. Another possibility is that the occurrences were so few and far between that over the course of several hundred generations any trace of Neanderthal was bred out of the population.
It still makes me wonder every time I see someone with a thick brow ridge or short stocky limbs.