What nvidia drivers did you used? The open source one or the proprietary one? Because I have the rtx 3070, and I have not experience a problem using the proprietary drivers in plain old debian stable, using x11