I don't think that is the root cause here, if that was true the people opposed to abortion wouldn't feel so strongly about it because they aren't really affected by other people having abortions (no, not even in their religious world view).

In fact, the percentage of the population directly affected by abortion legislation one way or the other is likely relatively small (not tiny, but probably not more than 20%) due to age and other life circumstances that make it unlikely for them to need one. The vast majority of the rest are people who "merely" have empathy with those who do and that includes men and women alike.