Have you seen the world recently? We are literally at the point where states are policing interstate travel because people might have an abortion. Standing in opposition to actual fascism is an ethical imperative, and people who equivocate in the face of true evil should be condemned, at minimum.

Some people just want to enjoy some spare time listening to podcasts about plants, gardening and the migration patterns of African swallows.
Some people want to listen to things they can nod along to and agree with.
And I guess some people want to listen to things that challenge their opinions, views etc.

I don't know why people listen to JRE. I think - back in the day - there was interesting and decent stuff.
Now there are interviews with anti-vax people. And they aren't challenged or corrected. So they say their piece along side actual scientists talking about cool planet discoveries or whatever.
My biggest gripe is that Spotify bought the show, and tries to shove it down my throat all the time. I can hide artists, albums and songs... but not podcasts

I'm not opposed to listening to people I disagree with, but that's not really the concern in this case. As others have pointed out, he doesn't really present a balanced perspective on anything, he just invites guests with extreme views, skewing heavily to the right, and then allows them to present their views with zero critical analysis. He just gets high and nods along like "wow bro so true" while lunatics like Jordan Peterson calmly explain some bullshit that can be summarized as "I don't like trans people existing, and I think we should put a stop to it."

You will also not find him entertaining any extreme ideas from the left, which if he were truly the unbiased "just asking questions" guy he purports to be, he would do at least occasionally.

So really it's a combination of the demonstrably false premise of him being an "unbiased centrist," his overt ignorance of the subjects he discusses, and his outsized influence on the minds of his listeners.

He's not just presenting ideas that I and others of the left disagree with, he is tacitly endorsing harmful and toxic ideas to his legions of fans, who then rabidly defend him in every venue online.

Edit: to clarify, I don't think trans rights is an implicitly political subject, but rather one that has been politicized and turned into fuel for the culture war. It's a minor thing but I felt I should make it clear since I used that as an example

Them: Let's kill all jews!!

You: hmm, iiiiiinteresting. Guess I'll challenge my views tonight!