Yeah it's odd. I think there's also something going on here that a lot of people seem to be (willfully) obtuse about which is that it's not looks, but sexual attraction that is being discussed.
I see some people arguing against the guys reaction because "attraction isn't just looks", and yeah everyone knows that, which is why it makes it so much more hurtful to say you wouldn't consider your partner for a sexual relationship.