I completely agree, the total number of users isn't really that important. The three things that count are having enough users to generate interesting content, developers who can develop the ecosystem, and people hosting instances. As long as these three things can be done sustainably then the Fediverse will be around indefinitely, and will likely outlast all the existing commercial platforms.

Too much rapid growth can also be a negative because it can disrupt the existing culture and normalize negative behaviors on mainstream platforms. When the growth is gradual then new people are more likely to adjust to the existing community norms.