I suppose, but since there's a much more limited supply of gatherable food, there's an upper limit on the time you can spend, and the size of community it can support.

Agriculture doesn't have that upper limit (well, arable land limit but that's still much more), plus it takes a ton of work to sow crops, irrigate water, and wait months for harvest. Much harder than just picking berries for an hour or two a day, which is why the transition to agriculture took so long even after it was discovered.