I think it is easy enough to argue without making it circular. As for "good", I don't think an objective absolute and universal definition is necessary.
The argument would be to consider it an optimization problem, and the interesting part, what the fitness function is. If we want to maximise happiness and freedom, any pair of people is transient. If it matters that they be kind to you, it is the exact same reasoning for why you should be to kind to them. Kinda like the "do unto others", except less prone to a masochist going around hurting people.
I'm not really arguing against this tho (perhaps the choosing part, but I'll get to it). I'm saying that a goal post of "axiomaric universal good" isn't all that interesting, because, as you say, there is likely no such thing. The goal shouldn't therefore be to find the global maximum, but to have a heuristic that is "universal enough". That's what I tried to make a point of, in that the golden rule would, at face value, suggests that a masochistic should go around and inflict pain onto others.
It shouldn't be any particular person's understanding, but a collectively agreed understanding. Which is in a way how it works, as this understanding is a part of culture, and differs from one to the other. Some things considered polite in the US is rude in Scandinavia, and vice versa. But, regardless, there will be some fundamentals that are universal enough, and we can consider that the criteria for what to maximise.