Indeed, my strong suspicion AGI is wholly impossible for computers as we presently understand them.

Hard disagree, right off the bat.

We do all this on three pounds of wet meat powered by cheeseburgers. It's not special. It doesn't take quantum entanglement or any silly business like souls. We simply don't know how to fake it yet. Current neural network shenanigans are a definite step in the right direction, as we started from observed data and got spooky abstraction, via three gigs of lumpen algebra we also don't understand.

If mere computing power were going to break the problem then we’d be getting there by now, or at least getting anywhere by now.

We kind of are. LLMs are the wrong shape of model, but still went from 'that'd never work' to 'holy shit' in like two years. The next-word-guesser has plateaued at a level where it can code, write poetry, tell jokes, and summarize complex articles. Badly. A real dog's breakfast on all fronts. It can't even count... but it tries. It has extracted enough from the end products of thoughtful writing, there's a glimmer of recreating those processes. It's already smart enough to call it stupid.

The fact it almost works is what "getting there" looks like. We're not about to turn it on and be blown away by divine omniscience, as in the nightmares of the subtweeted Elizer Yudkowsky. (Or more accurately as in The Metamorphosis Of Prime Intellect.) All the money in the world won't fix these models, but they're a near miss.

It’s abundantly clear is that ChatGPT simply does not have a concept of a poem as an object. There is literally no meta-cognition about poetry going on; it no more understands what a poem is than your phone’s autocomplete does a booty call.

Oh fuck off.

You just had a conversation with a robot about the poem it wrote, and you still want to say there's zero intelligence at play? Nobody told it how to do that. It demonstrably has the concept of a sonnet; it's just bad at doing the thing. People mock these models for playing chess badly when fed screenshots of an Atari game, as if that whole sequence of events working at all isn't science fiction bullshit made manifest. You wanna declare it philosophically incapable of comprehension, because of a D+ in English Lit?

are human brains Turing Machines?

A better formulation is - could a human brain be simulated? Would an atom-for-atom scan, applying all known laws of physics, work like the real thing? I expect so. I cannot imagine a reason it wouldn't, aside from woo woo mystical nonsense. If a Turing machine can emulate a mind this way, then intelligence is computable, and there's surely a less arcane technique.

The article eventually gets there, but not before entertaining this tiresome bollocks:

This forms the broad basis of one of the most famous objections to Turing’s claims about artificial intelligence, the Ch--

John Searle was a troll. He wanted to yank out a CPU and interrogate it, when it only does what it's told. If a computer appears sentient - it's the software. A book lets any idiot respond in perfect Chinese? Cool, that book speaks Chinese. Memorizing the book changes nothing because it's already a metaphor for software. Where computation happens cannot matter. Math in your head gets the same answer as a calculator, or you've failed.

When you're chatting up Ziyi, and open the box to find Raul and a lot of paper, Raul's complete ignorance means nothing. His calculations could be emulating a Game Boy and he wouldn't follow along. He doesn't have to. If you want to examine why Ziyi's last letter was ambiguous, then close the door, write it down, and ask her.

Turing’s own work in “On Computable Numbers” establishes fundamental holes in the reach of computability by proving that it’s impossible to construct a Turing Machine that can identify whether another Turing Machine running on a given input will halt and produce an output or get stuck in an infinite loop.

Because it's a paradox, not because it's magic. The core is literally 'if true then false' versus 'if false then true.' There's not some higher class of automaton that would get the right answer, because there is no right answer.

It’s true that, if all of the atoms and cells comprising the organism can be mathematically modeled, a Turing Machine would then be able to simulate them. But it doesn’t follow from this that the Turing Machine would then generate thought. Consider the analogy of digestion. Sure, a Turing Machine could model every single molecule of a steak and calculate the precise ways in which it would move through and be broken down by a human digestive system. But all this could ever accomplish would be running a simulation of eating the steak.

There's no such thing as simulated math.

This sophistry reeks of Descartes torturing dogs and insisting they only act as though they feel pain. Thought is an abstract process - when it occurs in simulation, it still occurs. You don't need a mouth to consume information.

Consciousness, I would suggest, is similar—a biological process whose realization consists of the actual qualia being subjectively experienced.

If a brain's entire universe consists of simulated events, in what manner did it not experience them?

The response to this is generally that the time and labor involved is fundamental to art. But even more fundamental is the thought involved. At the end of the day what defines art is the existence of intention behind it

And the intention of whoever's directing the software doesn't count, apparently. In reality, even if someone only selects from countless renders of "handsome portrait," that selection is an aesthetic process. It demands and reveals their interiority. They didn't make the result, any more than someone made the songs on a mix tape - but if the mix tape's about their girlfriend specifically, that's not the musicians' doing.