Usually known as GIGO - Garbage in, Garbage out.

Would be really cool if people finally understood the basic priciples of what they call AI, to realize this is not a criticism of AI but of society's output sourced for training.

The male form of "student" in German is generic and includes both male and female students.

That's not as easy as it sounds, because machine translation systems don't typically understand the grammar or vocabulary of the languages they work with. All it "knows" is that in the examples of English and German texts that say the same thing, the word "student" corresponds to „Student“ more often than it does to „Studentin“. Depending on the word you're translating, you'll also see the opposite problem, where a word like "nurse" is translated as feminine because that's what occurs more often in the training data, because the training data comes from a particular culture with its own gender biases. It's not an indication that anyone thinks women don't matter, or that men don't matter, just that the technology has some pretty serious shortcomings.

Looking at the Google Translate app for Android, I see it lists both German words as translations of "student" by itself, because someone has already done the work to handle simple cases correctly. It gets tricky when you try to translate whole phrases or sentences, because to do it correctly in general you have to make sure articles, adjectives, pronouns, etc. are changed to match the gender of the noun you'd prefer to be translated with a particular gender. When you're translating a sentence with nouns of multiple genders, it's likely to get confused and change the genders of the wrong words. The longer the text you're trying to translate, the worse the problems get.

I'm sure getting gender right most of the time can be done, but having worked with machine learning systems, natural language processing systems, and the kind of people who build them, I'm certain the developers are aware of the shortcomings and would love to fix them, and my intuition is that getting gender right most of the time for one language is probably as complex as the whole existing translation system, and that work would need will need to be repeated for every language that's supported. It's the kind of feature I might expect to see in a high-end translation system, like maybe a system designed as an assistant for human translators, but not in one that's offered for free. In other words, the real problem is that companies offering free machine translations don't have any financial incentive to spend the money that would be needed, and companies that do spend the money don't have an incentive to share their technology.

Maybe LLMs will offer a cheaper path to better translations because they're basically a much more sophisticated version of the same technology. OTOH a lot of people seem to hate LLMs with a fiery passion, so I don't know if using an LLM would actually make people happy even if it works a lot better.

I'd also like to add that the same kind of problem occurs with grammatical concepts other than gender. If you translate Chinese into English, you'll mostly get singular nouns and present-tense verbs for exactly the same reasons you get more masculine words in German. Machine translation is just generally crappy when one language lacks information that's mandatory in another language. The only thing that's different about gender is that people are more apt to be offended when the system fills in missing information with naive assumptions.