What do you find annoying about him? I think he's absolutely fine, though I don't follow him too closely, so I'm probably closer to that kid who watched a season of Cosmos and saw a handful of tweets than anything.

I looked a bit into him, and it looks like, as with many (most?) celebrities, there are sexual misconduct allegations. I don't know the details of any of those cases, so I can't really tell if it's legitimate, if he's largely being targeted due to his celebrity, or a mixture of both. Regardless, I don't think that matters at all when discussing his work on TV, running a planetarium, or doing science outreach programs. Nobody is perfect, so we should seek to separate the message from the messenger.

I'm much more interested in the "ragebait debates" thing you mentioned, but unfortunately his Wikipedia article doesn't mention "controversies" or whatever. That said, this has a lot more relevance to his work as a public science figure.