I feel ya brother, I have ADHD too and my memory is shit, you should be good to do some psychedelics, permanent damage usually doesn't happen unless you take entirely too much or you try something like taking some every day, you need to let your brain and psyche rest.

If you're interested I would recommend checking out erowid.com and making sure whatever you do you cross check any interactions with any medications you're taking.