Damn, you just clicked for me why I have a pretty good sense of direction. I've occasionally impressed myself and others for years, with "do you not know how we got here?" or "well we came from that direction" in spite of a generally terrible memory and a passionate dislike of geography and learning street names, etc.

But you're absolutely right, it's video games: puzzle dungeons, huge open worlds, metroidvanias, I even prefer playing with the UI and maps off whenever possible, and somehow I've never made this connection before. Incredible.