It's more complex than this even. Not all drugs are that toxic. In fact some of the most addictive aren't even that toxic at all like heroin. It's things like addiction, overdose, lack of clean supply, and the side effects that make it dangerous.

Even ones that are destructive to the body aren't always because of the chemicals they leave behind. Take meth for example: it's bad because of how much strain and immediate damage it causes, not long lived toxins. In small doses it's reasonably okay and is even prescribed by doctors sometimes. At amounts addicts do with the regularity they do them the damage builds up faster than it can be repaired by the body. MDMA, Amphetamine, Ketamine, and cocaine are similar here I believe.

Disclaimer: I am not a doctor, I mainly know about drugs from doing them and researching them online.