I got interested, so I spent some time looking into what's going on here. I'm not intimately familiar with X11 or Wayland, but I figured out some stuff.

Why sudo ip netns exec protected sudo -u user -i doesn't work for X11 apps

Short answer: file permissions and abstract unix sockets (which I didn't know were a thing before now).

File permissions: when I start an X11 login session, the DISPLAY is :0 and /tmp/.X11-unix/ has only 1 file X0. This file has 777 access. When I start my wayland session with Xwayland, the DISPLAY is :1 and /tmp/.X11-unix/ has 2 files X0 (777) and X1 (755). I can't figure out how to connect to display :0, so I guess I'm stuck with :1. When you change to a different (non-root) user, the user no longer has access to /tmp/.X11-unix/X1.

Abstract unix sockets: When I start my wayland/xwayland session, it creates abstract unix sockets with ids @/tmp/.X11-unix/X0 and @/tmp/.X11-unix/X1. See ss -lnp | grep Xwayland. The network namespace also sandboxes these abstract unix sockets. Compare socat ABSTRACT-CONNECT:/tmp/.X11-unix/X1 STDIN and sudo ip netns exec private socat ABSTRACT-CONNECT:/tmp/.X11-unix/X1 STDIN.

When you do sudo ip netns exec protected su - user, you loose access to both the filesystem unix socket /tmp/.X11-unix/X1 and the abstract unix socket @/tmp/.X11-unix/X1. You need access to one or the other for X11 applications to work.

I tried using socat to forward X1 such that it works in the network namespace... and it kinda works. sudo ip netns exec protected socat ABSTRACT-LISTEN:/tmp/.X11-unix/X1,fork UNIX-CONNECT:/tmp/.X11-unix/X1. It appears having ABSTRACT-LISTEN before UNIX-CONNECT is important, I guess it would be worth it to properly learn socat. With this sudo ip netns exec protected su - testuser -c 'env DISPLAY=:1 xmessage hi' works, but sudo ip netns exec protected su - testuser -c 'env DISPLAY=:1 QT_QPA_PLATFORM=xcb kcalc' does not work. 😞

Changing the file permissions on /tmp/.X11-unix/X1 to give the user access seems to work better.

Wayland waypipe

Waypipe works as advertised. But it's still a little bit tricky because you need to have two separate processes for the waypipe client and server, wait for the waypipe socket to be created, adjust file permissions for the waypipe socket file, and set (and probably mkdir) XDG_RUNTIME_DIR.

waypipe -s /tmp/mywaypipe client &
sleep 0.1
chgrp shared-display /tmp/mywaypipe
chmod g+w /tmp/mywaypipe
sudo ip netns exec protected su - testuser -c 'mkdir -p -m 0700 /tmp/runtime-testuser && env XDG_RUNTIME_DIR=/tmp/runtime-testuser waypipe -s /tmp/mywaypipe server -- env QT_QPA_PLATFORM=wayland kcalc'
kill -SIGINT %1

Combined

into this script https://github.com/vole-dev/grabbag/blob/main/run-netns-user-wayland.bash

Another thing to solve: XWayland apps as a different user

Giving access to the wayland socket makes other users able to use wayland; however programs that rely on XWayland to work don't seem to get it:

Start Failed
Failed to initialize graphics environment

java.awt.AWTError: Can't connect to X11 window server using ':0' as the value of the DISPLAY variable.
        at java.desktop/sun.awt.X11GraphicsEnvironment.initDisplay(Native Method)

Wine

0120:fixme:kernelbase:AppPolicyGetThreadInitializationType FFFFFFFA, 0ECAFF08
0128:err:winediag:nodrv_CreateWindow Application tried to create a window, but no driver could be loaded.
0128:err:winediag:nodrv_CreateWindow L"The explorer process failed to start."
0128:err:systray:initialize_systray Could not create tray window
0114:err:winediag:nodrv_CreateWindow Application tried to create a window, but no driver could be loaded.
0114:err:winediag:nodrv_CreateWindow L"Make sure that your X server is running and that $DISPLAY is set correctly."
0114:fixme:kernelbase:AppPolicyGetProcessTerminationMethod FFFFFFFA, 0DE4FB40

env | grep -i display
WAYLAND_DISPLAY=wayland-0
DISPLAY=:0