More people also means more demand for things that require labour to create however. Your position is referred to as the lump of labour fallacy

That's interesting but I think you're making a couple of crucial mistakes.

First as others mentioned, production and consumption are obviously intrinsically linked. A bigger country doesn't automatically mean bigger quality of life despite having more workers, Switzerland is not richer because it's smaller when it's got roughly the same population as the poorest country on earth. But if talking proportionally, more workers per capita means more production per capita, which means more consumption per capita.

Second, to kinda go in your direction and in part because of the contractual nature of employment, the market pressure on workers wages is not a product of the number of workers, but the number of available workers. For working (not unemployed) people, the quality of life does increase as that number gets lower, but this means less unemployment, not less workers. This fact is the reason why unemployment is not a side-effect of capitalism (or the lazy nature of people or whatever else), but a necessary feature of capitalism, since capital relies on this perpetual supply drive (buyers market) for profit.

edit: This isn't to talk about immigration, this is a more nuanced subject. Immigration has been defended on progressive basis (often not genuinely, but to benefit from cheap exploited labor) and attacked on reactionary basis (surprisingly also often non genuinely, e.g. France making massive anti-immigration propaganda in the 20th from one hand while asking border to let through illegally half a million of Portuguese workers with the other, against Portugal's demands).