The problem is it really depends who you ask. On road.cc, a fairly reputable bike magazine/website, you can find this: link

1. Do you need to change your helmet after a certain amount of time, even if you haven't had a crash?

Yes, but that's not necessarily because the expanded polystyrene foam degrades over time, as April Beard, Bontrager's (link is external) product manager for helmets, explains: "It really depends on how good of care you take of the helmet. There is no evidence that the EPS liner will deteriorate from age. Still, there are things such as solvents, chemicals and environmental exposure that can degrade the performance of the helmet."

Paul Caswell, the senior brand manager for Giro and Bell helmets' distributor ZyroFisher agrees: "We normally recommend a new lid every three years depending on usage as even with no impacts there is a constant knocking and pressing of the EPS as the helmet is stored, dropped, placed on hard surfaces etc. Due to tiny impact after tiny impact over time, the EPS will gradually lose its volume making it less able to deal with the energy in the unfortunate event of an impact. Of course, the more the helmet is used, the more it will deteriorate, so one rule for all does not work here, but three years is a good guide."

Helmets.org, a non-profit consumer-funded program providing information about bicycle helmets, largely backs up the manufacturer's stance. They cite data from an MEA Forensic study in 2015 that found that the foam liners of used but not crashed helmets retained their performance over many years, with some of the helmets tested being 26 years old.

When you actually look on helmets.org I am not sure what part of the manufacturer’s stance they imply that this program backs. link

Personally, I like to replace it every 4 years because I use mine a lot and I would rather fall for a selling trick than risk a higher chance of brain injury. In any case, a system like MIPS or wavecel would, in my opinion, justify buying a new helmet anyway if you don’t have that on your old one.

At a guess, it could be because the materials just naturally degrade over time. Any foam will start disintegrating, sun exposure could weaken the plastic, and the straps will fray.

I ride a motorcycle as my commuting vehicle and I find my helmet needs replacing every 5 years max, as the inner foam lining has compressed to point where it is no longer snug to my face and therefore unsafe.
Push bike helmets would be made of similar materials, just with less coverage.
So as @wander1236@sh.itjust.works says, it would be materials degradation.