First of all, you can assume the server can infer this in a number of ways - there is actually no way to fully block it, but we can try.

The main issue for privacy is that it makes your browser behave in ways that are a bit too specific (i.e. less private by comparison with the rest of the browsers in the known universe).

As for techniques the site can use

• javascript can test the geometry of something that was rendered to draw conclusions - was this font actually used? test several options and check for variations
• measure font work between network events i.e. generate a site that makes the browser use unique links for 1) fetches a font 2) renders text and 3) only then another fetch - measure the time between 1) and 3) and draw conclusions. Repeat for test cases and draw conclusions - e.g. is the browser really fast using monospace vs custom huge font? not a great method, but not completely worthless
• some techniques can actually do some of this without Javascript, provided you can generate some weird CSS/HTML that conditionally triggers a fetch

By the away not downloading the fonts also makes you "less private". Some of this is a stretch but not impossible.

Now for a more practical problem. Lots of sites use custom fonts for icons. Which means some sites will be very hard to use, because they only display buttons with an icon (actually a letter with a custom font).

FWIW these two lines are in my Firefox profile to disable downloads and skip document provided fonts:

user_pref("gfx.downloadable_fonts.enabled", false);
user_pref("browser.display.use_document_fonts", 0);

If someone has better/different settings please share.

Finally the Tor browser folks did good work on privacy protections over FF. Maybe their issue tracker is a good source of inspiration https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/18097