Not colorblind, but my guess would be the colors.

Those global overlay filters that tint the whole screen never seem to do anything for me at all.

On the other hand, the ones that change specific colours (enemy tags are blue instead of green, for example) are a huge help.

Partially red green colorblind here. There's really no pet peeves, but sometimes if I must identify the color/color accent, it takes focus.

Great question. Had to think about it and I'd say for me personally, poor implementation of color pickers is the biggest frustration.

As a technical user, I have no qualms w/ editing the default selection if it's hard to read due to colors, but I get frustrated with poor color picker implementation. For example, color swaths that don't have named descriptions when you hover over them. Even/especially the standard ROYGBIV colors on the first page of a color picker, but also to a lesser degree, descriptive hex codes on more nuanced online color pickers. I can't tell the difference and don't feel like hearing someone ask why I made the bold choice of making the sky pink.

Another issue is something like KDE's Konsole has a color picker that doesn't have clear names/examples for which aspect of the terminal is being changed, so when I wanted to change the bash custom prompt color to improve readability, I had to edit 5-6 different options, and use trial and error to fix the color.