Great question. Had to think about it and I'd say for me personally, poor implementation of color pickers is the biggest frustration.

As a technical user, I have no qualms w/ editing the default selection if it's hard to read due to colors, but I get frustrated with poor color picker implementation. For example, color swaths that don't have named descriptions when you hover over them. Even/especially the standard ROYGBIV colors on the first page of a color picker, but also to a lesser degree, descriptive hex codes on more nuanced online color pickers. I can't tell the difference and don't feel like hearing someone ask why I made the bold choice of making the sky pink.

Another issue is something like KDE's Konsole has a color picker that doesn't have clear names/examples for which aspect of the terminal is being changed, so when I wanted to change the bash custom prompt color to improve readability, I had to edit 5-6 different options, and use trial and error to fix the color.