I don't think only gender neutral bathrooms is the way to go. There should be an all gender bathroom and a woman's bathroom. Unfortunately we live in a world where women may still feel understandably uncomfortable with men in the bathroom with them.

Note: To be clear I'm not saying the women's bathroom should be for cis women only, it should only exclude men.