I work in this field a good bit, and you're largely correct. That's a great analogy of trying to remove salt from a stew. The only issue with that analogy is that that's technically possible still by distilling the stew and recovering the salt. Even though it would destroy the stew.

At the point that pii data is in the model, it's fully baked. It'd be like trying to get the eggs out of a baked cake. The chemical composition has changed into something else completely.

That's how building a model works today. Like baking a cake.

I'm order to remove or even identify pii data in ML models or LLMs today, we'd need a whole new way of baking a cake that would keep the eggs separate from the cake until just before you tried to take a bite out of it. The tools today don't allow you to do anything like that. They bake you a complete cake.