This depends on your farmer's market entirely.

Plenty of the non-fruit/veg stands are also small shops trying to go somewhere with more foot traffic.

I know that I buy from resellers, actual farmers have a farm to run and don't have time to run a stall. The produce is grown by actuall small farmers because the taste is markedly different

Such assholes of course also exist, but here on the local farmers market, the one selling vegetables indicates what is stuff he grew himself, vs things he bought from elsewhere. So if you want to buy things from the actual farmer standing there, you're perfectly capable of doing so (alas, just selling those things isn't really an option. Not enough people come to such markets if they can't find pretty much every vegetable they want...)

And regarding the price in this meme, that's also a complex story. Can indeed be inefficiencies of smaller scale distribution of the same stuff. But on the other hand there is organic and "organic" farming. There is always a large push by the large scale organic farmers to keep the requirements of being organic as low as possible. So yeah, there is a big difference between large scale farms that just make the bare minimum requirements for being called organic, vs smaller scale farms that actually try to make their farm respect nature (which is kind of the point of being an organic farmer).

And in the end, it's like all things in life, want to do it properly? Then spend time on actually learning who's who, what's what, if there is a farmers market, if they're serious, i'm sure you can visit their farms. You can learn about the methods they use, and the impact of those methods, and then compare it to other methods of production, and see if it's worth it for you, and see if the quality difference is worth it.