It probably depends on slight differences in regional dialect. Where I am from I would say woman and women are often pronounced pretty similarly, while man and men are easier to tell apart.

In my region, woman is often pronounced with an 'uh' sound, like womuhn. It's pretty easy to confuse with women. I have noticed that people in my area will sometimes vary up the 'wo' part of women and woman depending on which one they are using. So women becomes 'wimen', and woman becomes 'wumuhn'.