I think that really depends on why the app made the system hang.
Can you reproduce it consistently? If so, you could try out different forms of isolation, like flatpak, docker, a VM. And there are linux distros focused on each of those, but you can try a solution on whatever distro you're running.
If for some reason your system hangs due to resources (which is the only case I have ever experienced), that can be limited through cgroups and such. The only resource I don't know how to limit is GPU compute.