You are 100% right. The quality dropped without him. He was a good antagonist.

I know he's apparently a piece of shit to work with, but that doesn't change the fact that the stories were better with him.