It may be counterintuitive, but stakers don't actually have influence over the consensus mechanism. It's actually the other way around. Consider it this way; the stake that a staker puts up is a hostage that the staker is providing to the blockchain. If I stake a million dollars worth of Ether, I'm basically telling the blockchain's users "you can trust me to process blocks correctly because if I fail to do so you can destroy my million dollar stake." I have a million dollars riding on me following the blockchain's rules. That's literally why it's called a "stake."

The people who are actually "in charge" of which consensus rules are in use are the userbase as a whole, the ones who pay transaction fees and give Ether value by purchasing it from the validators. If some validators were to go rogue and create a fork that was to their liking but not to the liking of the userbase, the rogue validators would be holding worthless tokens on a blockchain that nobody is using. You can see the effects of this by the way the blockchain is continuing to update in ways that are good for the general userbase but not necessarily for the validators - MEV-burn, for example, is a proposal that would reduce the amount of money that validators could make but there's no concern that I've seen about the validators somehow "rejecting" it. If the userbase wants it the validators can't reject it without losing much more than they could hope to gain.

Ironically, proof-of-work is more vulnerable to this kind of thing. If a proof-of-work chain were to fork and a substantial majority of the validators didn't agree with the fork then they could attack it with 51% attacks. The forked chain would need to change its PoW algorithm to stop the attacks, and that would destroy all the "friendly" miners along with the attackers.

Validators in a PoS blockchain could also launch attacks at a contentious fork, but they'd burn their stake in the process whereas the validators that did what the userbase wanted would keep theirs. So there's a powerful incentive to just go along with the userbase's desires.