The slippery slope argument doesn't apply.
Regardless of what one thinks of the moral or psychological effects of either legalization or decriminalization of prostitution, it would not make rape legal. If anything, it would allow more protection for the workers through a stronger contractual agreement, because they would be able to organize and negotiate out in the open instead of pleading with their pimps in motel rooms.
Until late last century, a woman couldn't even bring a rape complaint against an abusive husband, but we can now, and the charge can stick. There's no reason to think that sex by contract for other reasons would become more prone to rape when the victim can make the accusation in broad daylight without fear of being prosecuted herself (or himself).