Who do you recognize as the authority to make that decision though? If the locals are currently ruled by a terrorist group or Nazis or whatever, do they get to decide? What about the locals that disagree with the government currently in power?
And an answer of 'if we just didn't needlessly meddle' might be the ideal, but it's ignoring the realities that we have meddled and some countries are unlikely to stop doing so. We have to accept the world we have not the one we wished we had.
What if some of the locals want it taken away for protection, but the government wants it destroyed?
There's no clear 'owner' in many cases. I think it places where it's uncertain, then we should prioritize saving the artifacts over the ones that seek to destroy them.