Say Jane passes Bob, travels some distance away, then turns around and comes back. For both her outward trip and return trip, her experience and Bob’s experience are symmetric—but when Jane accelerates (by turning around), she changes reference frames. In her new reference frame, the point in Bob’s history Jane sees as simultaneous with her own changes—and the farther apart they are, the greater the time shift will be. This time shift will persist for Jane’s return journey, since she’s no longer changing reference frames.

Bob, on the other hand, never perceives a comparable shift in Jane’s history, since he never changes reference frames.

To illustrate the frame shift, let’s say Jane is four light-years away and moving at a relative speed that gives a Lorentz factor of two. So just before turning around, she sees a red-shifted signal from Bob in which he’s moving at half speed. She knows the signal took four years to reach her, so she’s looking at Bob from two years in his past.

Immediately after turning around, she sees the same signal from Bob but now he’s blue-shifted, moving at double speed. Knowing the signal took four years to reach her, she now interprets the same signal from Bob as being eight years in his past—so the point in Bob’s history she considers simultaneous with her own just shifted by six years.

Or here’s a simpler way of looking at the asymmetry: Both Bob and Jane see the other’s signal go through a red-shifted phase and a blue-shifted phase. Jane experiences the two phases as being equal in duration; but because the light from Jane’s turnaround takes four years to reach him, Bob experiences the red-shifted phase being longer that the blue-shifted phase.

If you want to get really technical, it's because the symmetries of the Minkowski metric are the Poincaré group. Which includes only rotations, translations and boosts, none of which correspond to acceleration. Meaning it's inherently impossible to make acceleration look like being stationary because of the geometry of spacetime.

If Alice flies by Bob at some relativistic speed, then there's a very simple coordinate transform (a Lorentz boost) that flips our perspective to Alice's pov; she's stationary and Bob is moving.

If Alice were to accelerate and we did the same thing, we'd end up with a "momentarily comoving reference frame," in which Alice is only "stationary" for an instant and Bob is moving at a constant speed as before. Or we could create a non-inertial reference frame which would look nothing like Bob's perspective, but Alice would be stationary.

Physics in non-inertial frames behaves differently, as a simple example: if stationary (or constant speed) Bob dropped an object while floating in space, it would remain there. If accelerating Alice tried the same thing, it would accelerate away from her. You can test this out in an accelerating car or train or whatever and see that it's fundamentally asymmetrical even before considering SR.

In terms of things like length contraction and time dilation, these are a little more complicated mathematically, but it's just an extension of the above asymmetry when spacetime is Minkowski rather than Euclidean. The difference in observed time is clear when looking at each person's worldline, Alice's isn't straight like Bob's and so she unambiguously experiences a different proper time and proper length.

Ultimately this means that even if Alice accelerates then passes Bob at a constantly speed, they'll both see one another's clocks running slow by the same amount, when Alice decelerates and returns to compare her stopwatch with Bob's they'll have very different totals which corresponds to how much time Alice lost during her acceleration.

The tl;dr answer is that it's axiomatic. It's built in to the model, and if it weren't true, the model wouldn't work.

More broadly though, symmetry in relativity exists because there is no way of distinguishing the "truth". No one point of reference is more correct. You can't tell who is truly moving, because as you highlight, A is moving away from B is just as valid as B is moving away from A.

Acceleration breaks symmetry though, because you can measure it. You can tell whether it's A or B that's accelerating, and there is a "right" answer.

I can’t help but feel like this post was generated from a physics discussion I had the other day here.