It looks like it's not a case of Sweden asking the US to drop the exemption, rather Sweden can simply decide that there is no exemption for that case and only have to tell the Americans that they are doing so.

In specific cases of particular importance to Sweden, Swedish authorities may withdraw the waiver by providing a statement in writing to the competent U.S. forces authorities not later than thirty (30) days after receipt of the notification described in Paragraph 2 of this Article.

Paragraph 2 says that America has a duty to notify Sweden about anything relevant