Another part of it is the gpu bios. The gpu bios contains x86 opcodes that it expects the host system to run for gpu-specific functions like video mode switching and probably lots of other stuff. I know that Vesa bios extensions mode switching requires a pointer to the functions in the gpu bios which the cpu runs. I tried to make a platform independent Vesa driver one time and couldn't figure out how to circumvent using the gpu bios for it since the functions you're supposed to call are compiled for x86. Even the well-refined projects like Seabios still rely on the VBE pointers for non-legacy video modes.

Legacy vga does also has a bios but it's relatively not that difficult to circumvent using the bios on legacy vga cards, only issue is that legacy vga modes are mostly useless.

I think there's a newish way of doing this stuff that doesn't involve Vesa or legacy vga but I don't know what it is. This I'm sure is only one of the many problems that have to be overcome if someone wanted to hack a 1080ti onto a raspberry pi or something.