For clarity, the game I had in mind there is Magicka. You're not directly punished just for using magic, but all spells carry risk through something like area effects, ricochet, trapping yourself behind obstacles you place, mis-typing input combos, idiosyncratic targeting mechanisms, trying to cast electric/fire magic while you're soaked with water/oil, etc.

Trying to hit the lich with a thunderbolt? Congratulations, you just exploded your party's archer because you forgot they were at the highest elevation of any character in range.

Also for clarity, I don't think this system is appropriate for a ttrpg. It really needs time constraints to work properly, or harder-to-avoid risks that start to verge into "direct punishment" territory, which itself has a high risk of just not being fun. I think the latter option is generally best saved for the most devastating (or just eldritch/necromantic) spells. You can summon an army of beholders, but the material component is your character's own eyes.