That’s how we get helium.

Edit: Because someone has downvoted me, now I have to explain it.

Helium is a noble gas; it doesn't bond with anything. It's also lighter than air (at ground level), so if it was just free, it would rise. So how come I can go buy a tank of it and fill balloons?

Radioactive uranium and thorium decay, and when they do, they release an alpha particle. An alpha particle is two protons and two neutrons. This is the same as a helium nucleus. When these alpha particles are released, they capture electrons from the environment, and now you have helium.

When high concentrations of uranium and/or thorium are found near natural gas seams, it's possible for the helium to become trapped in the top of the seam. It's then possible to collect that helium as a byproduct of natural gas wells.

Almost all of the world's helium is produced in an area around San Antonio, TX.

We've actually discovered a few of these! Though, nothing quite so catastrophic as you might be thinking.

Natural nuclear reactors exists:

https://en.wikipedia.org/wiki/Natural_nuclear_fission_reactor

Though I wonder if this is just a big meteor...