I relate to this. My father isn't as openly hostile as yours sounds, but he's a narcissist. One year, he decided he was to busy to visit my kids, his grandkids. He's retired.
For me, there is an emotional tug that will always be there, not for him, but for a father that loves me. Rationally, I remind myself of why I haven't talked to him in 5 years. It's gotten easier over time, but it still flares up occasionally.