I am not the OP, nor am I a lawyer, but I believe I am informed enough to answer these.

How does it work in the case where the person was outside of the USA at the time, seeing as they were not on USA "soil" at the time? It's just that one of the parties (in this case the federal government) has to be on USA soil?

Yes. In this case, the alleged offense (the cost demanded for renouncing citizenship) took place by the US federal government on American soil, which is why they can use through American courts.

The reason why they probably wouldn't be suing through the court system of the country they immigrated to is because other countries do not have the authority to dictate how much money the US is demanding. But at the same time, there's technically no reason to pay the US either if you never plan on going back there, given that the US has no power to arrest people in foreign soil...unless the two countries have an extradition treaty in place (and much of the first world does). The US would then have to sue for extradition within the court system of the other country first, and then you'd be facing a lawsuit in the US over unpaid fees.

The threat of the latter is also assuming the fee justifies the court expense spent pursuing it, which I doubt it would. I met a lot of American expats in China who technically owe the US government thousands of dollars in unpaid taxes/fees/etc but aren't even worried about going back to visit because the government would be spending far more pursuing legal action than they stand to make from the suit. The only time one should be worried is the rare example where the government might want to make an example of someone, or if you're a mob boss or something and that's the only concrete offense they can jail you for.

And how does that work if, say, you're standing on the USA side of the Mexican border and you throw a brick at someone on the Mexican side? Could the Mexican citizen in this case file a lawsuit in a USA court?

Now ain't that the tricky scenario. A similar case actually came up recently, with Hernandez v. Mesa and it was ruled at the time by the conservative-stacked Supreme Court that the US government was not responsible for prosecuting a crime where the victim was not in the US and not an American citizen. But the fact that there were dissenting opinions from all of the non-conservative judges, who are themselves legal experts on the constitution, shows that this is a very contentious gray area.

I guess the takeaway from this is that the person in this hypothetical scenario would be better off filing suit from Mexico and pushing for extradition, as the two countries have an extradition treaty.

Court jurisdiction can become a really complicated question, but citizenship of the parties has nothing to do with it. If a court has jurisdiction, doesn't matter if the plaintiffs reside on Mars.