I agree that it's not desirable to conflate the two in common usage, but I don't really see how that can be done while continuing to use those specific terms.

What constitutes toxic behaviour is culturally subjective. Many people in the first group would have been considered a part of the second not so long ago.