I guess I see the N64 situated in a time when big changes were happening in gaming. Besides the obvious advent of 3d, this was around the time when gaming stopped becoming arcade-dominant and transitioned towards being console-dominant, which has huge implications (arcades existed in third places while consoles were privately owned). It's for these two reasons that I wouldn't call N64 retro (or lump N64 in a different historic period than the NES/SNES if we're just using "retro" to mean "old").

I think there's a degree of millennial revisionism where the spotlight is shown on retro/whatever-you-want-to-call-that-particular-period-of-gaming console games when they weren't even the dominant form of gaming during that time period. You're not going to see video essays of The Simpsons arcade game or Alien vs Predator or Space Harrier anytime soon even though those were massively popular arcade games way back in the day. You can't really compare arcade cabinet sales vs console game sales because obviously people weren't individually buying cabinets to put in their garages but buying cabinets to put in a mall or a laundromat or a pizza place.

The dominant form of gaming was arcades. You can see this very clearly whenever movies or TV shows from the 80s and 90s reference gaming. It's almost always some kind of arcade game. The Simpsons' first reference of a console game (Bonestorm) was a Season 7 episode that aired 9 months before the N64 dropped in NA and even that reference was largely a Mortal Kombat reference (Liu Kang knockoff getting owned by a tank, Goro mirror match in a bridge stage that every arcade Mortal Kombat game had). Everything else before that were parodies of arcade games. You have something like Terminator 2 where you saw John Connor briefly playing Afterburner 2 at a mall while being stalked by the T-1000.

There is a way bigger difference between the nes and n64 than between the ps3 and ps5, hell even between the ps2 and ps5. Things moved really fast in the 90s as far as games and the technology behind them goes. A console generation is longer and also less of a defining era of gaming than it used to be because at this point putting more power into a system is yielding dismissing returns, you can only make a polygon so small until it doesn't matter anymore or takes way more time and money than it's worth. Gameplay has gotten stagnant as hell for AAA games over the last 15 years or so as well and at the moment, indie games are starting to fall down that trend hole too. The world can only take so many survival crafting games, rougelites and metroidvanias

I completely missed out on the PS1 so the rise of all these faux-retro games with polygon jitter is honestly pretty cool. Helps that we've learned how to make games feel better than the first time they looked like this.

I guess my point is that regardless of what particular label someone uses, the development of gaming can be split into various periods just like how the history of painting can be split into various periods. It's just weird to have a floating label that basically means "old." When I was a kid, "old games" were essentially just pre-1983 crash games while "modern games" were post-1983 crash games because gaming was only two decades old. But now, gaming is a little over half a century old at this point.

In the end, I think "retro" is used in gaming in the same way "classic" is used in film and movie. Casablanca and The Godfather are both classic films even though they have nothing in common outside of being old Hollywood films.