I thought this first too. But then I remembered an interaction where one colleague of mine told another pair who were speaking another language that "secrets don't make friends" or some such. I think it was intended as a jokey way to express that he was uncomfortable with the conversation that he couldn't understand. He also joked that they were probably talking poorly of him. I noticed this person was normalizing controlling the discussion by throwing negative or secretive intentions onto the others' discussions. In reality, they're just friends discussing something in their primary language.

Anyway, long story long, I don't think this colleague would tell us he has a problem with others speaking a language besides English, but then he'd probably follow that up with a bunch of clarifiers that indicate he does in fact have a problem with it.