Disclaimer: I'm not celibate.

I think, the real difficulties emerge when presented with opportunity. This is why some people, who didn't think they would, cheat on their partners and some actually regret it. It can take a lot of willpower to refuse to have sex, because it's genetically ingrained in most people. Every single one of their thousands of ancestors had sex for them to exist today.