I didn't care until it consistently loaded faster.
That's now my new baseline, and anything slower than 'instant' is annoying.
I would care if that was no longer the case, because I don't like being constantly annoyed.

That said, I don't think the page loading speed is noticeably different between major browsers.
The addons, customisation, privacy and resource usage are where it's at.

I'm just hoping that some competition to chromium stays afloat.