I've been having a lot of vague thoughts about the unconscious bits of our brains and body, in regards to LLMs. The parts of our brains/neurons that started evolving back in simple animals as basically super primitive ways to process visual/audio/whatever input.

Our brains do a LOT of signal processing and filtering that never reaches conscious thought, that we can't even reach with our conscious thought if we tried, but which is necessary for our squishy body-things to take in input from our environment and turn it into something useful instead of drowning in a screeching eye-searing tangled mess of chaotic sensory input all the time.

LLMs strike me as that sort of low-level input processing, the pattern-recognition and filtering. I think true generalized AI would have to be built on pieces like this--probably a lot of them. Ways to pluck patterns out of complex but repeated input. Like, this stuff definitely isn't self-aware, but could eventually end up as some sort of processing library for something else far down the line.

Now might be a good time to pick up Peter Watts' sci-fi book Blindsight. He doesn't exactly write about AI in it, but he does write about a creature that responds to input but isn't exactly conscious like you or I.