I think one needs to address other issues if they are afraid that the entire group prefers the one they deem "toxic" over themself. If the entire party is willing to ditch one for the sake of another, I don't think that implies a fault with the latter.