I've actually never ridden a road bike, but I can say that I can fairly easily hold my own on my 2017 CX bike in a group ride with road cyclists on much newer and more expensive bikes. I'm sure it doesn't feel quite as zoomy in direct comparison, but in my experience that's a very relative thing.
Anecdotally, I swapped my trusty 33mm Schwalbe X One Speeds over to 40mm G One Allrounds last year, and for a few days the bike felt sluggish to me. But not only did that feeling evaporate very quickly; my reason for swapping - spongy steering on slightly deeper gravel and generally loose surfaces - went away and I can ride at least as fast or faster and safer on every surface as a result. And most of those surfaces my road cyclist friends can't even dream of touching.
There's ever more differentiation in gravel bikes. Some tuned for stiffness and speed, essentially road bikes with wider rims; others for carry capacity and comfort over long distances. I lean towards the speedy stiffness and higher tire pressures (my gravel friends mock me for it.)
While I'd certainly try a proper road bike when the opportunity arises, the choice is ultimately a very simple one for me: Whenever I'm out with road cyclists, I feel cramped by the routes that are simply off limit to the group and that would be very much on (as in: bring it!) for me. My bike can handle whatever comes my way, and the only limiting factors to speed are either my legs or my inner chicken.
If the woods and more technical passages have no appeal at all for you it's probably not a good fit, but if they do even a little (and I think they do, even if just a little), I think there's a wonderful world out there for you waiting to be explored. My heavily biased advice: Suck it up and get a cross/gravel bike. You might surprise yourself putting on 45mms one day and loving every minute.
Thanks for the opportunity to vent my love for gravel bikes.