If I'm more specific, what Vegans care about is conscious experience. They don't care if something is alive or has some form of reactive biological intelligence. Its not a loose definition of killing that's the problem, it's the killing of conscious beings.

There is no scientific consensus as to the potential for consciousness in plants/trees. Almost nobody affirms that they are. You'll find generally that when we discuss consciousness we describe beings with brains, or if we get in to gray areas, beings that at least have some form of nervous system. Since there is some level of brain plasticity, I tend to take the position that consciousness is an emergent property found in those with a nervous system at bare minimum, but absolutely and especially those with brains. That said, there are particular areas of brains that if compromised will show patterns of lost consciousness, but I just don't affirm that those areas are entirely responsible.

So if plants and trees are not conscious, and they don't experience reality, and there is no subject, then there is no one to grant rights to. If we were talking about some random planet that had no conscious life on it, a planet that for some reason could never support conscious life but could support plant life, I would have no ethical quandary with a space fairing civilization taking all of those resources and leaving the planet with not but rock.

The need for residential housing complicates the ethics of forest habitat removal but not by that much if we consider what a vegan world looks like. Roughly 37.5% of the world's habitable land could be redistributed as that land currently is required for animal agriculture that otherwise wouldn't be. Roughly the size of North America and Brazil combined. You'd have loads of land that could be reforested but also some land that could be reused for housing purposes. As for current reality, I think there's a strong argument that group housing or apartment blocks would be far better for both people and the planet.