This is the same problem as with race. Colorblindness won't help anyone, because there already is a system of structural discrimination in place. The same is true for sexism and gender biases. Just wishing to be in a neutral and fair place won't bring us there. I would really like to be able to skip all the racism, sexism and other discrimination systems as well. But it will take work to change them. And not only that, I would argue that we need to abolish capitalism on the way, too. Because patriarchy and racism are deeply intertwined with capitalism. So, unfortunately no easy task...

Oh, btw it's also interesting that you seem to convolute sex and gender. Because you write "it really shouldn't matter much what's between one's legs." Well, depending on the context it doesn't (you rarely see the genitals of others). Just a reminder that penis ≠ man and vulva ≠ woman.