You could also cut down ppl, if they weren't interdependent and interacting with each other enough to realise fast enough and start retaliating.
One of the big factors making humans (and animals in general) have power over the trees is, that we are faster, both at action and adaptation, thanks to our superior mechanisms of Central Nervous System and Muscular systems.

But at the same time, any single human would be much more dependent upon trees in general, as compared to how much a single tree would be, upon us, or other animals. Since the seed stage of the plant is sturdy enough to let it choose its starting point, all it really needs is for the place it chose to remain within a reasonable range of the germination conditions (soil, water, air, insolation quality etc.) and it will be just fine.