This is tough. If it was just a sicko who generated the images for himself locally... that is the definition of a victimless crime, no? And it might actually dissuade him from seeking out real CSAM....

BUT, iirc he was actually distributing the material, and even contacted minors, so... yeah he definitely needed to be arrested.

But, I'm still torn on the first scenario...

Fuck that guy first of all.

What makes me think is, what about all that cartoon porn showing cartoon kids? What about hentai showing younger kids? What's the difference if all are fake and being distributed online as well?

Not defending him.

One thing to consider, if this turned out to be accepted, it would make it much harder to prosecute actual csam, they could claim "ai generated" for actual images

I hate the future

I find it interesting that the relabeling of CP to CSAM weakens their argument here. "CP generated by AI is still CP" makes sense, but if there's no abusee, it's just CSM. Makes me wonder if they would have not rebranded if they knew about the proliferation of AI pornography.

Can't generate Abuse Material without Abuse. Generative AI does not need any indecent training to be able to produce indecent merial.

But it is a nice story to shock and scare many people so i guess the goal is reached.

oh man, i love the future, we havent solved world hunger, or reduce carbon emissions to 0, and we are on the brink of a world war, but now we have AI's that can generate CSAM and fake footage on the fly 💀

Quick things to note.

One, yes, some models were trained on CSAM. In AI you'll have checkpoints in a model. As a model learns new things, you have a new checkpoint. SD1.5 was the base model used in this. SD1.5 itself was not trained on any CSAM, but people have giving additional training to SD1.5 to create new checkpoints that have CSAM baked in. Likely, this is what this person was using.

Two, yes, you can get something out of a model that was never in the model to begin with. It's complicated, but a way to think about it is, a program draws raw pixels to the screen. Your GPU applies some math to smooth that out. That math adds additional information that the program never distinctly pushed to your screen.

Models have tensors which long story short, is a way to express an average way pixels should land to arrive at some object. This is why you see six fingered people in AI art. There wasn't any six fingered person fed into the model, what you are seeing the averaging of weights pushing pixels between two different relationships for the word "hand". That averaging is adding new information in the expression of an additional finger.

I won't deep dive into the maths of it. But there's ways to coax new ways to average weights to arrive at new outcomes. The training part is what tells the relationship between A and C to be B'. But if we wanted D' as the outcome, we could retrain the model to have C and E averaging OR we could use things call LoRAs to change the low order ranking of B' to D'. This doesn't require us to retrain the model, we are just providing guidance on ways to average things that the model has already seen. Retraining on C and E to D' is the part old models and checkpoints used to go and that requires a lot of images to retrain that. Taking the outcome B' and putting a thumb on the scale to put it to D' is an easier route, that just requires a generalized teaching of how to skew the weights and is much easier.

I know this is massively summarizing things and yeah I get it, it's a bit hard to conceptualize how we can go from something like MSAA to generating CSAM. And yeah, I'm skipping over a lot of steps here. But at the end of the day, those tensors are just numbers that tell the program how to push pixels around given a word. You can maths those numbers to give results that the numbers weren't originally arranged to do in the first place. AI models are not databases, they aren't recalling pixel for pixel images they've seen before, they're averaging out averages of averages.

I think this case will be slam dunk because highly likely this person's model was an SD1.5 checkpoint that was trained on very bad things. But with the advent of being able to change how averages themselves and not the source tensors in the model work, you can teach new ways for a model to average weights to obtain results the model didn't originally have, without any kind of source material to train the model. It's like the difference between Spatial antialiasing and MSAA.

Good