The more steps in the instruction pipeline the more ways there are for there to be an error where some result doesn't get erased when undoing stuff from the wrong branch. It's basically like telling someone to move into a new house and get settled then stopping them six hours in and trying to make sure you get all their stuff out.

It wasn't branch prediction alone, it was the cache combined with branch prediction. The problem is that even discarded outcomes fill the cache with data. Those older vulnerabilities also had the problem that the access permissions check was done after the branch prediction. It's probably too expensive to do when it's not even clear yet whether the branch is going to be taken (that's just speculation on my part though).