As someone formerly in the same boat, I think belief in the Abrahamic religions makes it hard to identify with the plights of others, because if you believe in a just, loving god, then "those people" have the religion and hardships that they do for a reason (and the reason is usually either "it's part of God's plan" or "they made bad decisions").

When you base your entire worldview on a faulty premise, you can use sound logic to get all the way to libertarianism without a problem. Once I reexamined and discarded my belief in the Christian god, it was like flipping a switch; I went from douchey religious Libertarian to bleeding-heart socialist almost literally overnight.