Legit, it would depend on the chair.

Assuming no rubber or equivalent, the ride would be bumpy as hell, but doable on a well packed road, or relatively well laid stone. I've pushed patients over both surfaces, and seen plenty of patients do it themselves.

Gravel is the real pain in the ass. You get a fresh gravel path, and good luck with most wheelchairs that aren't powered and/or have narrow wheels. Way more effort.

On something like cobblestones, you might even have an easier time moving yourself without rubber tires.

For short distances, even a poorly packed but dry dirt path isn't bad.

Remember, wheelchairs annually predate asphalt by a good bit, and existed before sidewalks and such were evenly paved. Some of the older examples from the Victorian era were used on city streets that were unpaved entirely, and impassable when wet, but managed to work over the dried ruts.

Same with unpaved dirt paths that would be poorly maintained. I've had patients in chairs push themselves down paths in the woods while hunting, with things growing in the path, rocks not fully cleared, etc. That's modern chairs, but the fundamental design is more robust regarding surfaces than you'd think