This is why metadata matters so much. Plenty of "end-to-end encrypted" apps only handle the content of messages, but know on the server side what people are in what places. Or even keep a historical record of who joined and left. Or just don't bother encrypting...