This assumes some concept of location independent of the surrounding matter.

We like to think that way because we live on a mostly-not-changing clump of rock and dirt.

There is nothing to define location other than what’s nearby.

The only alternative anyone ever proposes is larger clumps of matter further away. Relative to the sun. Relative to the center of the Milky Way. Center of the Milky Way is the most “legit” Nonmoving Point we can think of.

But maybe the legitimacy of the nonmoving point is based also on its nearness to you. Perhaps the thing that defines the wormhole’s position through spacetime is inertia and gravity.

Hard to see what else it might be, other than “it doesn’t move” which, the entire point above being, doesn’t really exist