Depends on what you want. I did a lot of side quests on my first character, but I had to because I had the habit of just accepting everything - until I ran into the 20 sidequest cap… This forced me to do side quests when I wanted to continue the main quest, because I would at least want to be able to accept the side quests that are outside & harder to find, if I were to come across them.

On my second character I did exactly that. Just did the main quest and accepted all the side quests that were on my way but away from any waypoint. And I gotta say, the campaign boss fights were so much easier on a lower level. I'm not super sure whether I finish all side quests on that character though. The exp you get for them is typically pretty low and I don't have a need for the herbs or leather packs they give you at this stage anymore either.

So yeah, my recommendation is to focus on the main quests & just pick up stuff you find outside the towns. Do sidequests when your quest book gets too full but otherwise only afterwards.