I suffer from this with almost any media I use and enjoy, re-watching/reading is a thing I enjoy at least ¯\_(ツ)_/¯

Sorry if this is not the answer that you are looking though.

Idk if it's an ADHD thing or just a normal people thing but I think I don't remember much about the books until I'm discussing/writing about the subject. It jogs my memory and all of a sudden I can recall significant portions of the texts. Perhaps not verbatim, but the overall vibe is there. Doubly so if it's something you're actually interested in. Don't beat yourself up too much OP, you probably remember more than you realize!

No need to beat yourself up, just make simple changes to improve. Write a one paragraph summary for yourself after each chapter, or list key points you want to remember. The next day, review it and take 10 minutes to think about how that chapter applies to your life or other things you've read.

Once you've finished the book, rewrite & combine your notes into one or two pages and flip back through the book to find a quote or two that stands out to you. Then set a reminder on your calendar one month in the future to review your summary page.

Note-taking, rewriting notes, and spaced repetition are all proven study techniques. It's a little extra time, but it's nothing compared to the time you're spending reading, and it'll make a big difference in how well you remember your takeaways from it.