After reading a non-fiction book, do you beat yourself up over not remembering all that much? This is especially painful if the book took years to complete (e.g. Anne Applebaum's "Gulag").
It's a bit ridiculous to expect to become an Encyclopedia after reading something in passing too, though.
I feel as if working with a computer and using the internet daily destroyed my attention span, which is why I'm self concious about this.
I suffer from this with almost any media I use and enjoy, re-watching/reading is a thing I enjoy at least ¯\_(ツ)_/¯
Sorry if this is not the answer that you are looking though.