The Eaton Center in Montréal has been building gender-neutral bathrooms and it seems to be working quite well. Bonus is there's actually proper stalls that go from floor to ceiling with no gaps in them. IMO this is where the privacy matters, and then the sinks area is a lot more visible and less likely to be assaulted there.

Meanwhile in high-school in the boys restroom I've had someone jump up the stall on me multiple times because they thought it was funny.