You sure you can train a model deterministically down to each bits? Like feeding them into sha256sum will yield the same hash?

Even if you trained the AI yourself from scratch you still can't be confident you know what the AI is going to say under any given circumstance. LLMs have an inherent unpredictability to them. That's part of their purpose, they're not databases or search engines.

if I were to download a pre-trained model from what I thought was a reputable source, but was man-in-the middled and provided with a maliciously trained model

This is a risk for anything you download off the Internet, even source code could be MITMed to give you something with malicious stuff embedded in it. And no, I don't believe you'd read and comprehend every line of it before you compile and run it. You need to verify checksums

As I said above, the real security comes from the code that's running the LLM model. If someone wanted to "listen in" on what you say to the AI, they'd need to compromise that code to have it send your inputs to them. The model itself can't do that. If someone wanted to have the model delete data or mess with your machine, it would be the execution framework of the model that's doing that, not the model itself. And so forth.

You can probably come up with edge cases that are more difficult to secure, such as a troubleshooting AI whose literal purpose is messing with your system's settings and whatnot, but that's why I said "99% of the way there" in my original comment. There's always edge cases.

Ironically, as far as I'm aware it's based off of research done by some AI decelerationists over on the alignment forum who wanted to show how "unsafe" open models were in the hopes that there'd be regulation imposed to prevent companies from distributing them. They demonstrated that the "refusals" trained into LLMs could be removed with this method, allowing it to answer questions they considered scary.

The open LLM community responded by going "coooool!" And adapting the technique as a general tool for "training" models in various other ways.

That would be part of what's required for them to be "open-weight".

A plain old binary LLM model is somewhat equivalent to compiled object code, so redistributability is the main thing you can "open" about it compared to a "closed" model.

An LLM model is more malleable than compiled object code, though, as I described above there's various ways you can mutate an LLM model without needing its "source code." So it's not exactly equivalent to compiled object code.

Well, LLMs are quite literally trained on language, so asking it to simulate a conversation between a hotel clerk and a guest who is upset that they can’t find the hair dryer is pretty much what it’s best at doing.

You can even build the dialogs with students. Have them introduce a scenario for the LLM to manufacture, then have the students suggest variables to apply, such as the clerk being hungry and in a bad mood while the guest is actually drunk after returning from a club in order to see how the language changes, then have the students act it out for laughs.

Agreed, but there have been big projects that have been open source. I can imagine* an AI (LLM) being developed fully FOSS. It would be rare, but I can see it happening if a big foundation got behind it. Maybe Mozilla, or another that tries to keep the spirit of their mission statement.

*Imagine: I'm not too familar with all of the current, public, and free models out there, just a few. This was just me making a hopeful guess about if it might be actually happening now.

I did read this part, and while this is generally true, there are use cases of such large models. Some of them require the input of personal data (find bugs in my code, formalize this email, scan this picture for text and translate it, draw an anime version of this picture of my friend tom)

So people being weary of security implications of such large models are certainly not

in a huge circle jerk that never ends, but refuses to understand how it all works.

Sure you can just call them all dumb using ai like the mainstream (putting in personal data) and attribute it to an unwillingness to understand, but this doesn't match the reality. Most people don't even understand how an operating system functions, which components work online and which offline and who can access which of their information, let alone know how "AI" works and what the security implications are.

So If people ask those questions, hoping there are alternatives they can use safely your answer "no, u just dumb, machine can't harm you, its not magic, just don't put in data in"

Is not only rude but also missing the point. Most usefull/fun/mainstream ways DO in fact, put in data.

You explaining basic models also doesn't help, as the concern here is not mainly/only the model, but american spy institution to access all prompts you did put in, maybe categorizing you in personality clusters dependent on your usage of language or assigning tags on which political stance a users has (and with entities like the NSA I could imagine far worse)

Also "A model is a model" Is not very accurate in such cases. When someone has control and secrecy over each aspect of the model, it would be very well possible for entities like the NSA to manipulate the content the models puts out in arbitrary directions. A government controlling and manipulating information the public receives is a red flag for a lot of people (rightfully so IMHO)

How are people supposed to get better in digital privacy topics if you just tell them to shut up and insult them when they aks questions trying to learn? You acting like you are in your Elfenbeinturm of genius isn't helping anyone.