You're correct in relation to precious metals and other commodities that attain some form of communally accepted value independent of a centralized authority. In that case someone has to physically come take them from you by force or theft as the other poster noted. Or alternatively turn peoples minds collectively away from perceiving value in it to accomplish a similar effect by different means.

As for other physical currency, that can be "revoked" by a central authority like a central bank at will. If they deem it worthless and disable use of it in commerce tomorrow, it's not long before the average person agrees and it becomes only as valuable as the base paper or coin representing it. There is no need to physically seize it, but it has a similar effect as revocation or seizure would in terms of ongoing usefulness of its previous purpose to you.

This can also occur slowly over time via currency manipulation. The USD has lost somewhere in the range of 96% of its value or purchasing power since 1913. This isn't complete revocation, but again, accomplishes a similar goal without the need for physical confiscation.