It wasn't a computing power issue: simply plugging a hat into the same exact pi B2 fixed the issue.
I've tried to Google for an article about this but I didn't get lucky. Essentially it was something about how the jack output was driven using pwm that was causing hissing and crackling if I remember right.
And I concur that it's a good everything streamer if you use the HDMI output or a hat with an audiophile-grade audio output.
Edit: the latest pis might still have a better-driven jack output but I still won't trust it unless I test it thoroughly.
This is the way!