Thinking about it some more I don't think we would need to abandon the whole TTY to get a good set of the features. What is basically required for a lot of the features is more communication between the shell and the terminal. There is already some communication for basic things - like raw mode and alternative buffers, colors and even images. These are how TUI programs like vim or screen/tmux function and how you can exit them without losing what was previously in the buffer.

I wonder if markers for the prompt and start/end of command output would probably enable a lot smarter virtual terminals with only some minor additions to the VT100 protocols. Possibly some extra data could be sent as well - like optional tooltip data maybe or even supporting actions that when the user clicks something it can send a response back to the shell. Maybe like a retry button on previous commands for example.

There is quite a lot that could be done if the terminal and shells had better protocals to communicate between each other. I dont think these will change overnight though so seeing terminal emulators try out these features to find what people like/want to use IMO is a good thing to see where we can take things in the longer term.

Would we have to abandon SSH or always X forward

No we would not. At the end of the day a TTY is just a input and output pipe between the terminal and the program running on the shell with a specific protocal VT100 (or some bastardized version of that - looking at you xterm). This is what network protocals are as well - just with different protocals in play. So you can do a lot over that connection with changes to the protocals. No need for xforwarding at all.