saying that the British should inherit it is a very weak argument

Yes, I am not making that argument, inheritors mush be at least somewhat related.

Although, in case you're talking about, the indigenous people's artifacts will likely end up in the country of their conquerors and oppressors, which is also a shame