I use the Fjällräven/Specialized (now seems to be rebranded as 'Hoja') hip pack for this.

It's just about big enough to fit an average book, my headphones, and some miscellaneous stuff. But if needed it's also expandable so you can fit a jacket, or groceries on the way back etc.

It has completely replaced the backpack I used to carry everywhere (which I carried because of wanting to bring a book).