Libertarianism has nothing to do with private vs. public companies. Libertarians are not pushing for only private companies.

Libertarians are for Free Market, but that doesn't mean only private companies.

Say you're paying 10 cents per gallon of water from a private company. If the government can do it and charge you 5 cents per gallon that's what should happen in a "Free Market"

What's not good is when the government charges you 15 cents a gallon and a private company can do it for 10 cents but is blocked from doing it because it's not a "Free market".

If we said USPS is the only mail carrier. UPS or FedEx can't exist and you've got to pay whatever USPS wants to charge. That's not a free market. Same as saying USPS can't exist because UPS and FedEx control the sector. Not free.

In your example it sounds like the government was losing a lot of money with the public water and instead of them charging more and trying to fix everything they just gave up and passed it off to a private company. It's less that the private company is trying to milk you for all your money but "we don't have thousands of taxpayers to subsidize our expenses and inflation exists"

Imagine being happy paying taxes just so your local government can get rid of your public water. They probably increased local taxes to pay for all the public water infrastructure. Are they going to decrease your local taxes now that they aren't supplying you that service? Your local government screwed you, but you see the new company trying to supply you water as the bad guys.