I almost dumped Firefox on Android for Chromium based one due to this reason before realising the culprit, many sites were super sluggish and FF often lost cache when I changed between apps.

I am now just using the built in setting, many sites have native dark mode and the ones that don't - I realised it does not bother me too much.