I have a lot of indirect experience/knowledge with caves. I don't have to be one of the people who directly explored one to know one of the first things one learns about caves is how ill-suited for wandering around they are. Slopes can change on a dime, it's incredibly rocky inside, and they fill with whatever falls from the sky. Imagine hiding in one of these, as someone with less intelligence than us, and not expecting to even stub a toe, let alone fall or suffocate. I assume it must be a concern because all the cave hobbyists I know (even if I only know a couple) say they have to split up based on their physical skills (so one chooses a slope-free while the other a rain-free one as would be the case for me as someone who never learned), though I admit I'd be amused if cavepeople didn't ignore these slopey checkpoints and instead it caused them to make some inventions.
Another thing that sticks out to me is the rule against fire. When cave explorers (not me) venture in, the most important rule of all time they learn is that it's an absolute cardinal sin to light so much as a match in a cave, let alone a torch (in contrast to Indiana Jones movies where that's the first darn thing he does). The heat from fire is enough to interfere with the stone composition, which in turn threatens to collapse a cave. Imagine having just discovered fire, and you go running to show your family but everyone dies before you can say "hey Mario look what I made". I wouldn't expect a caveman to know about that rule, but I would expect them to feel tempted to find out the hard way.
Finally, there's the fact they're filled with disease. Most notably from the cave animals; while things like ticks and rabies are not common in cave animals, they do happen. If that wasn't enough to outright stigmatize dwelling in a cave, even the environment itself is viral. There are caves where the reservoirs are like 100% condensed bacteria. That's got to send awful mixed messages to seek refuge from an oasis and suddenly you have a fear of water like me.
How did caves become such a go-to and one where nobody is depicted as having any serious accidents in?
Not an expert, but I do have some insights on this. The reason that spelunking doesn't usually allow fires, even at the entrance, is because the smoke can damage the cave environment. Smoke, especially from things manufactured in a different environment, can contain a lot of wild stuff that interferes wjth an ecosystem. Caves, especially ones big enough for tours, have incredibly unique ecosystems that are ripe for study by biologists.
Additionally, smoke contains unique particles that settle on surfaces. We can study the materials deep in some caves like Mammoth Cave and find evidence of smoke and ash. This is strong evidence for archeologists that ancient people were able to travel deep into this cave.
tl;dr: smoke damages ecosystems and makes archeology difficult. Don't smoke in a cave.
(would write more, but using phone keyboard)
It isn't just the smoke though, it's the heat, as I was explaining here when I got this reply. As an on-site out-observer, that much from all of this I do understand (which formed the basis for what I don't).
It's a matter of statistics. We know that it can cause a collapse, but depending on the cave it's not often. We don't want people to get hurt, so we're going to warn people against generating heat like that. But with a ton of caves, it would have happened irregularly enough for people to be mostly fine.
That was 7ft flames in a sandstone cave. Yes it can happen. No it wouldn’t happen often enough to have killed every single human living in a cave system.
Not every one in every part, no. It's just a hazard per section.