On morality:

The main issue with this idea of punishment and reward, in the sense that you mean them, is that their results depend entirely on the criteria by which you are punished or rewarded.

Shifting the goalposts from "it's pessimistic" to "it depends on criteria".

I was being simplistic to keep it less verbose. I didn't talk about consistency, for example, even if it also matters.

By "reward/punishment" I mean something closer to behaviourism than to law. I could've called them "negative stimulus" and "positive stimulus" too, it ends the same.

Say, the law says being gay is illegal and the punishment is execution, does that mean it’s immoral?

For the bigots implementing and supporting such idiotic laws? Yes, they consider being gay immoral. That's the point for them.

For gay people? It throws them into a catch-22, because following such a law is also punishing: they're forced into celibate and prevented from expressing their sexuality and sexual identity.

So even considering your example, rooting morality into punishment and reward still works. And you can even retrieve a few conclusions out of it:

1. Moral is subjective. No such thing as an universal set of moral values. (NB: the ability to set up those moral values still needs to be there, in order to get a moral agent.)
2. What 2+ people consider morally good might enter in conflict.
3. Trying to reason with bigots and hoping to change their views solely through logical arguments is 100% futile.
4. Since people promoting those laws are forcing gay people into a catch-22, they're treating gay people as existentially immoral.

Back to AI. Without ability to be rewarded/punished, not even a hypothetical AGI would be a moral agent. At most it would be able to talk about the topic, but not generate a consistent set of moral rules for itself. And no, model feeding, tweaking parameters, etc. are clearly not reward/punishment.

Being moral boils down to making certain decisions, the method by which they are achieved is irrelevant if the decisions are “correct”.

That's circular reasoning, given that the "correct decisions" will be dictated by moral.

Most moral philosophies agree that moral decisions can be made by applying rational reasoning to some basic principles (e.g. the categorical imperative).

That does not address the request.

I'll rephrase it: since you disagree that moral values ultimately come from reward and punishment, I asked where you think that they come from. For example, plenty of the moral philosophies that you mentioned root their moral values into superstitions, like "God" or similar crap.

Language and reasoning:

We [humans] reason through language

That's likely false, and also bullshit (i.e. a claim made up with no regards to its truth value).

While language and reasoning do interact with each other, "there is a substantial and growing body of evidence from across experimental fields indicating autonomy between language and reasoning" (paywall-free link.

That already dismantles your argument on its central point. But I'll still dig further into it.

and these models capture and simulate that.

They don't even capture language as a whole, let alone a different system like reasoning.

They output decent grammar and vocab. But they handle notoriously poorly meaning (semantics) and utterance purpose (pragmatics). They show blatant signs of not knowing what they are outputting.

You can test this by yourself by asking any LLM-powered chatbot of your choice about some topic that you know by heart. Then looking at the incorrect answers, and asking yourself why the bot is outputting that wrong piece of info ("hallucination").

Example here.

I watched the video, honestly I don’t find anything too surprising.

Except the fact that it directly contradicts your claim.

The idea that AI could be immoral because it holds the stances of its developers is invalid, because it doesn’t. It is trained on a vast corpus of text, which captures popular views and not the views of the developers.

Emphasis mine.

This is the cherry of the cake because it shows that you're wasting my time with a subject that you're completely clueless about. I'm saying that those systems are amoral, not immoral.

Sorry to be blunt but I'm not wasting my time further with you.