You are vastly underestimating the prevalence of chromosomal variations. They are common, especially among cis women.

I like the way you phrased that at the end. Sexes are categories that relate exclusively to the concept of progeny. If you're not able to reproduce, you're already kind of excluded from the sex binary. If we break the human concept of sex down to its constituent parts, it is just "can procreate". The categories are useful in some contexts, but to state them as universal or to try and extrapolate them so widely is significantly disruptive and unhelpful. Humans are and always have been more than our reproductive anatomy. Your doctor and anyone you want to reproduce with are really the only people who need to know whether you fit into either category.