it's a tech product that runs a series of complicated loops against a large series of texts and returns the closest comparison, as it stands it's never going to be dangerous in and of itself.

That's not how it works. I really don't get what's with people these days being so willing to be confidently incorrect. It's like after the pandemic people just decided that if everyone else was spewing BS from their "gut feelings," well gosh darnit they could too!

It uses gradient descent on a large series of texts to build a neural network capable of predicting those texts as accurately as possible.

How that network actually operates ends up a black box, especially for larger models.

But research over the past year and a half in simpler toy models has found that there's a rather extensive degree of abstraction. For example, a small GPT trained only on legal Othello or Chess moves ends up building a virtual representation of the board and tracks "my pieces" and "opponent pieces" on it, despite never being fed anything that directly describes the board or the concept of 'mine' vs 'other'. In fact, in the Chess model, the research found there was even a single vector in the neural network that could be flipped to have the model play well or play like shit regardless of the surrounding moves fed in.

It's fairly different from what you seem to think it is. Though I suspect that's not going to matter to you in the least, as I've come to find that explaining transformers to people spouting misinformation about them online has about the same result as a few years ago explaining vaccine research to people spouting misinformation about that.