I think somebody posted a guide about passing GPU function from the host to a guest VM (for example) with vfio, so it's certainly possible to get a more bare-metal experience!