Ideally you should change the type if you do such a thing

But that's not always possible or desirable. For example, maybe you're largely just passing it through to another library, and that library made a breaking change.

So I don't think we should have semver always handled automatically, it should instead prompt the developer with its best guess, and the developer would ideally only move from patch -> minor, patch -> major, or minor -> major, and never the other direction. But the developer should always be involved in picking the version. So don't just throw it into CI and call it a day, but instead have a CLI tool that suggests it and requires developer approval before making the PR for the version bump.