Kropotkin's rebuttal in Conquest of Bread is in chapter 12: objections. He expands on it quite a bit in section 12.3, but his conclusion is:

Take, for example, an association stipulating that each of its members should carry out the following contract: “We undertake to give you the use of our houses, stores, streets, means of transport, schools, museums, etc., on condition that, from twenty to forty-five or fifty years of age, you consecrate four or five hours a day to some work recognized as necessary to existence. Choose yourself the producing groups which you wish to join, or organize a new group, provided that it will undertake to produce necessaries. And as for the remainder of your time, combine together with whomsoever you like, for recreation, art, or science, according to the bent of your taste.

“Twelve or fifteen hundred hours of work a year, in one of the groups producing food, clothes, or houses, or employed in public sanitation, transport, and so on, is all we ask of you. For this amount of work we guarantee to you the free use of all that these groups produce, or will produce. But if not one, of the thousands of groups of our federation, will receive you, whatever be their motive; if you are absolutely incapable of producing anything useful, or if you refuse to do it, then live like an isolated man or like an invalid. If we are rich enough to give you the necessaries of life we shall be delighted to give them to you. You are a man, and you have the right to live. But as you wish to live under special conditions, and leave the ranks, it is more than probable that you will suffer for it in your daily relations with other citizens. You will be looked upon as a ghost of bourgeois society, unless some friends of yours, discovering you to be a talent, kindly free you from all moral obligation towards society by doing all the necessary work for you.

“And finally, if it does not please you, go and look for other conditions elsewhere in the wide world, or else seek adherents and organize with them on novel principles. We prefer our own.”

This is what could be done in a communal society in order to turn away sluggards if they became too numerous.