How do you know you can't become those past selves again? If you did have that ability imagine what that would be like...

Let's say you want to bring back your middle school self. You would forget everything that came after middle school, you would become that middle schooler again. Not only that, but to truly be your past self, you would have to be in the same situation, place, point in time. Otherwise the change in environment would be reflected in a change in your mind and that change would mean you are a new person, not your past self. To truly become your past self, you would just be back in middle school and the future would still be ahead. If you pressed a button that did that, after you pressed the button you would not and could not know anything happened at all.

Going beyond that, maybe you have the ability to become your future selves as well. Maybe you have both of these abilities and you just don't know, can't know. If you did, you couldn't tell.

For all you know, in between every moment you jump around and become every version of yourself. Maybe every possible version of yourself. Maybe in between every moment you become every person that has ever lived or will ever live. Every saint and murderer, every animal and bug, every rock and star and black hole. Everything and everyone. And in all that jumping around you would eventually land back to this current moment and version of you, except one timestep in the future. And you wouldn't know any of that jumping around happened, because to be you in this moment you can't be anything else.

You are not separate from the universe, you are the universe. Past, present, and future.