The ad categories offered by various companies vary and I think adsense is nowhere near the closest-targetable network there is.

Try showing an ad to only Python software developers. Not IT repair shops. Not software developers writing exclusively C. I think you may be able to do that with keyword targeting on AdWords, once you avoid the bear traps you mentioned, but it's hard.

OTOH, I bet there are ad companies that will help you target "30-40 years, single, lonely" for dating ads (that might be possible even with adsense), and definitely people with specific diseases to peddle medicine to them.

Occasionally someone posts a list of categories used by one or multiple networks and they can be the most specific, or far too broad (see: python dev).

I'm extremely surprised that I haven't seen ad companies offering specifically to advertise to people working at specific companies. I'm sure it exists, just haven't seen it. This would be incredibly valuable both for job ads, industry specific ads (this would benefit from breaking down by department), and also criminals and spies trying to get people from specific companies infected with malware.

What's also important to understand is that these categories don't need to be accurate. "This person has a 80% chance to be in category X" is more than good enough. Hell, 10% would probably already work.

The right ads pay really well. A life insurance click can be worth tens of dollars, because the conversion is worth thousands. So if there is a 10% chance you're interested in buying life insurance, bombarding you with those ads makes sense.