Because they didn't actually live in caves that often.

The image of these people living in caves arises from the fact that caves are where the preponderance of artifacts have been found from European Stone Age cultures. However, this most likely reflects the degree of preservation that caves provide over the millennia, rather than an indication of them being a typical form of shelter. Until the last glacial period, the great majority of humans did not live in caves, as nomadic hunter-gatherer tribes lived in a variety of temporary structures, such as tents[4] and wooden huts (e.g., at Ohalo). A few genuine cave dwellings did exist, however, such as at Mount Carmel in Israel.>

Wikipedia - Caveman

Not an expert, but it seems like you're conflating issues of spelunking with cavemen living 5 meters from the entrance.

Having been to a lot of Palaeolithic sites and studied cave formation and the like...

They usually used just the cave mouths or even rock shelters. Caves "breath" and if they "breath" in then the environment in a cave mouth can be dry and sheltered from the rain. You'll also find that the most frequently used caves are either higher up the side of a hill providing good views over the area to help hunt game and/or they're close to a spring.

They did venture deeper into caves but usually for ritual purposes and those that were decorated for these purposes seem to have been chosen for relative ease of access (you just need a torch and a good guide) but also because the journey itself to the ritual site was part of the ritual (we can only speculate but it may be part of a rebirth ritual and/or the effect would have been enhanced by drugs, music, etc).

It might be that you have a mistaken view of cave dwelling.

Most such settlements only used relatively shallow rock-face caves, which eliminates the majority of your listed risks.

Without reliable light sources, you'll appreciate that people weren't delving kilometers deep into the ground. Caves were simply a way to protect from the elements and to provide a choke point to better defend against wild creatures.

There's an additional point that other commenters haven't brought up. Sure, all those dangers you mention exist, and tons of people suffered and died of them over the tens of thousands of years that humanity was in this period. But enough people still survived.

They considered the caves to be safer alternatives to whatever else they were using before they found the cave. And they hardly ever stayed in one permanently, instead migrating around to find food.

Another thing that sticks out to me is the rule against fire. (...) The heat from fire is enough to interfere with the stone composition, which in turn threatens to collapse a cave.

Is that really the case? I thought it would more about filling the cave with smoke or wasting oxygen, not causing a cave to collapse. Perhaps I was imagining a different cave? Tom Hanks did keep fire in his cave in Cast Away.

Anyway, very interesting subject. I always wanted to have a cave of my own, especially with an entry hidden behind a waterfall :D

Caves can be safe, there are many different cave landscapes. Everything from tight squeezes to huge chambers the size of a football field. Some are big, some are small, some complex, some relatively straight and simple. There are many caves that have guided tours. They even have gatherings and weddings inside.

I just want to add that gases are probably also a huge prohibiting factor to living in a cave. All of the second-hand cave emergencies I've ever heard of are due to poisonous air.