Weeeeell, not exactly.

It is true that things are all relative to each other. But think about it this was, if you fly out in a Starship in one direction for a light year, then turn around exactly 180 degrees and fly back, you wouldn't arrive back at earth, right? Mainly because things are accelerating due to gravity. And acceleration breaks symmetry.

It would of course depend on how time travel works, but since time and space are linked, it wouldn't be unreasonable to assume that you follow the same trajectory in space as you move through time. But that would be a straight line in the space+time dimensions still. Think of the paths in Minkowski diagrams.