Imagine there being 2 portals on the shovel one on the front and one on the back Whatever passes through remains in the same place. THIS is the actual analoge for the rusty hole. Entering the first portal the shit gains the speed of the shovel but since the exit portal is also traveling with the same speed but facing the opposite direction, it effectively cancels out the previous speed gain. Imagine it like the first portal adding one unit of speed to it and the exit one adding minus one unit of speed because of the opposite direction.

So now with one portal on the shovel and the other somewhere on a wall the shit gets accelerated and you will feel a force acting on the shovel that decelerates it's swing.

Velocity is relative and this is why it works. If velocity wasn't relative then Einstein would be wrong and many of the effects we see in the real world could never happen.