Peer pressure and cultural background are all that matters, along with personal conscience, not any absolute external standard of morality.

For most of mankind's history slavery has been moral, and moral arguments have been used to support it, including in the USA less than 200 years ago; the most politically advanced and egalitarian society on the planet. Remember the Civil War? Very similar arguments were used to justify Jim Crow laws. At least the Greeks and Romans were honest about it: "You lost the war, Jack. You and your descendants are property now." Our moralists preferred to tell their victims: "You are sub-human and you're better off this way than free. God must have wanted it that way or he wouldn't have let it happen."

Once you reject the idea of a morality-dictating God who sets arbitrary rules and then punishes transgressors and rewards the obedient, there is absolutely no justification for any morality, except what we decide on existentially as individuals and as communities. Public morality evolves historically and culturally, and religion does play a role. I reject Christianity but I do not reject all Christian morality. Just some of it.

Your bringing up gays is interesting, I have no problem with gays doing their thing to or with each other, but I am very much opposed to they (or straights) forcing sex on those who don't want it. And children are off limits, as is the common cultural standard in my society.
The Graeco-Roman culture of the ancient Mediterranean, which we often point out as the sources of our cultural values, and which was highly regarded by Americans and Europeans in the 18th and 19th centuries, didn't even have a word for homosexuality in their language, and were pedophiles and pederasts as well. They did share our distate for rape, except for slaves, of course. The other source of our cultural values, Judaism, rejected homosexuality but embraced the principle of scapegoating, which is the whole justification for Christianity.

Does this mean there should be no morality? Of course not, we do have an individual conscience and sense of morality. Mine is based on an intense dislike of watching other people suffer, even if I don't like them very much, and a personal refusal to benefit from their suffering. Maybe it's the way I was brought up, maybe it's in my genes, maybe I just got dealt this hand by sheer luck. But I can't justify it as part of some abstract moral absolute and I certainly am not doing it because I want a ticket to heaven and protection from hell. Which by the way, makes me morally superior to the traditional moral absolutist.

As for your last paragraph, accepting that logic inevitably forces you to accept its logical consequence: that any morality, no matter how vile and reprehensible, is superior to a "sophisticated" amorality simply because it provides you with some kind of survival advantage. That is wrong on two counts: One, it denies that all individuals and communities on this planet already possess a right and capacity to self defense, and morality has nothing to do with it. It is part of our biological equipment. Even amoral beasts will defend themselves. Second, if bad yet strong moral beliefs are superior to weak but tolerant amoral ones, then your entire concept of moral absolutism is exposed for the sham and fraud it is. It reduces morality to sheer power over others and the justification to do to them what you think you are entitled to.