To the best of my knowledge, how a body processes caffeine is highly individual.

My body seems extremely bad at processing caffeine, and I'll sleep worse and get other bad side effects after a few days of drinking two cups of coffee before 2 PM.

Other people drink coffee until the late afternoon and don't report such problems.

In summary: I would recommend anyone who experiences sleep problems or the feeling of being stressed to avoid caffeine for 2-3 weeks to rule it out as a source of problems. OTOH if your sleep is okay and you don't sleep bad, consume as much caffeine as you are happy about.

One more word about the research concerning caffeine: Usually the research is done with young (< 30 years), healthy (they sort out people with illnesses problems) people. AFAIK everything they say about caffeine is technically true, but unless you are young and healthy, the research doesn't tell you anything about how caffeine will impact your life/body/health.

Where is the line?

We interact with hundreds if not thousands of chemicals in every ordinary act of life. This is not just unavoidable, it's normal and natural, and has been going on for centuries if not millenia.

Are you proposing we stop cooking food (which results in chemical alterations of the underlying food). What about soap?

You have a point, but you've oversimplified it and taken it to an extreme where it's no longer a sound or balanced idea.

Having an expresso machine doesn't mean you will drink 10 cups a day. It will just make the cups you drink taste better and be more enjoyable. Personally, I avoid coffee after 6PM and don't have any problems with sleep. I'm also fine without drinking coffee, some days I prefer tea or just water.