Just get a used ultra-small form factor PC a la the Tiny, Mini, or Micro series. A higher-end one which is 7 generations old will still absolutely destroy the Pi in terms of performance.
Once I gave up (for now) on doing all this on ARM and switched back to x86, everything got way easier to actually accomplish.
I use NoMachine, but that's in a Linux-to-Linux environment.
Did a test last weekend sitting in a department store parking lot on the store's public wifi, wifi bitrate about 50Mbps both ways, 50ms between me and my homelab ... very very usable experience with quality set at 6/10.