In general I think people are referring to things that are either A) heavily proccesed and/or B) something that isn't naturally occurring. When they speak about "chemicals"

Your body is made to eat natually occuring plants and animals. Any deviation from that is risking long term issues. Effects that are very often (at least here in the states) ignored unless they just straight up kill you. And even then it'll probably take a couple decades before anyone actually does something about it. So, yes while many people misuse the word "chemical," their fear is not misplaced. You should be skeptical of things that are synthesized until they are proven to interact with the human body appropriately