For clarity, the game I had in mind there is Magicka. You're not directly punished just for using magic, but all spells carry risk through something like area effects, ricochet, trapping yourself behind obstacles you place, mis-typing input combos, idiosyncratic targeting mechanisms, trying to cast electric/fire magic while you're soaked with water/oil, etc.

Trying to hit the lich with a thunderbolt? Congratulations, you just exploded your party's archer because you forgot they were at the highest elevation of any character in range.

Also for clarity, I don't think this system is appropriate for a ttrpg. It really needs time constraints to work properly, or harder-to-avoid risks that start to verge into "direct punishment" territory, which itself has a high risk of just not being fun. I think the latter option is generally best saved for the most devastating (or just eldritch/necromantic) spells. You can summon an army of beholders, but the material component is your character's own eyes.

Personally I quite like them, but I feel like they should be long-term effects rather than just "you rolled a 1 so you miscast and hurt yourself." The long-term effects can add a lot to RP potential for characters that I think makes it interesting.

Like, as an example, say magic ages people faster, so every time you level up you also age a few years. Or, to take from Dark Sun, magic gradually transforms you into various magic creatures, taking away the caster's humanity. Both could lead to some interesting RP that isn't super common in most games. (Or, well, the former isn't, the human/inhuman thing comes up in WoD stuff all the time)

Obviously in such a case you would require a different system to limit magic use for balance though, so we return to the question of slots or mana ~~or scene abilities like in 4e or Lancer~~