Mostly the neighbors who don't pay their dues or needlessly hold up needed improvements.

Our HOA board does a great job, they are volunteers who live in the neighborhood. Our dues used to be $100 a year, plus $150 if you wanted to use the pool. Then the pool needed to be replaced (it was 45 years old) and we didn't have the money for it, so we changed the dues to $250 and everyone gets access to the pool. That still didn't raise enough money, so they tried to get everyone to agree to a one-time assessment of $1000 but too many people complained. So they've now raised the dues to $350, likely going up to $450 soon.

The good thing about the higher dues is we will now be able to afford a lawyer to go after the 10% of the homes that aren't paying their dues. And maybe even force the people who are hoarding cars in their yards to have to move them.

Edit to add from a follow up comment;

The change to the dues required a change to the bylaws, which required 100% participation in voting, and required 60% approval. The change passed with over 85% approval. Most HOA dues in our area are $600 - $1000 per year, and many of those don't have a pool. There are also many places around that don't have an HOA. The HOA was created at the same time the pool went in, anyone already living in the neighborhood who didn't want to use the pool didn't have to join the HOA, we still have a few 'grandfathered' homes and I have no issue with them, it's the people who moved into the neighborhood knowing there's an HOA and a pool and refuse to pay their share.