Forget the brain in the box

The first time you talked to ChatGPT, Claude or Gemini, you may have felt a little foolish. You typed “Hello.” You phrased your question politely, the way you would address someone you did not want to offend. When the reply appeared, perfectly written, clear and courteous, you felt a peculiar little shiver. You thought, “This is incredible. It understood me.” And then came a less comfortable thought: am I actually talking to someone?

Let me put your mind at ease. If you had the impression that a tiny consciousness was watching you from the other side of the screen, you are perfectly normal. Everyone feels it. Even the engineers who build these machines are occasionally taken in, despite knowing exactly what is inside them.

There is nobody there. No little genius trapped in a box, no mind pondering your words, no entity lurking in a Californian server and finding you pleasant or irritating.

To understand why you believed otherwise, we need to go back to 1770.

That year, a Hungarian inventor began touring the courts of Europe with an astonishing automaton: a mannequin dressed as a Turk, seated at a chessboard, that defeated the continent’s greatest players. It became known as the Mechanical Turk. Even Napoleon is said to have tried his luck and lost. For decades, nobody could explain the marvel. Scholars argued, articles multiplied, and everyone searched for the miraculous mechanism. The truth eventually emerged, and it was comically mundane: a human chess player was folded inside the cabinet, operating the mannequin’s arm from his hiding place.

Remember that story, because it contains everything you need to know. For two centuries, making a machine appear to think required hiding a person inside it. Today there is nobody left in the box, yet the illusion is a hundred times more powerful.

Artificial intelligence does not think. It calculates.

That is a blunt truth, but above all it is wonderfully liberating. There is no reason to feel intimidated. There is no reason to fear asking a stupid question, because the machine does not care. It does not judge you. It cannot judge you, any more than your calculator can laugh at you for typing seven times eight.

So why are we taken in so easily?

Do not blame yourself. Blame your biology. The human brain is a magnificent intention-detecting machine. Since prehistoric times, our survival has depended on our ability to spot living things. A bush rustling on the savannah? Our ancestors had to assume at once that a lion was hiding there rather than a gust of wind. Better to run away needlessly a thousand times than to be wrong once. The cautious had children; the sceptics had far fewer. We are the descendants of the cautious. We see minds everywhere. When the car breaks down on the morning of an important appointment, we are convinced it has done so deliberately.

So when words appear with the fluency of a real conversation, that ancient reflex goes into overdrive. It projects humanity onto the machine before you have even finished reading the sentence.

And to show you just how little reason you have to feel foolish, let me tell you a true story. It took place long before the internet, in 1966, in the laboratories of the highly respected MIT.

Joseph Weizenbaum, a brilliant computer scientist, created a small experimental program called ELIZA. It was the prehistoric ancestor of today’s assistants. The idea was to simulate a conversation with a psychotherapist.

Its workings were laughably simple. The program understood nothing, absolutely nothing. It spotted keywords in whatever people typed and spat out stock questions.

You wrote, “My mother annoys me.” It picked up the word “mother” and replied, “Tell me more about your family.” You wrote, “I’m sad.” It turned the sentence back on you: “Why are you sad?”

A few hundred lines of code, no more. Weizenbaum thought he had devised a good joke. He wanted to show his colleagues that conversation with a computer was bound to be superficial.

He was in for the shock of his life.

Within days, students, researchers, some of the brightest minds of their generation, even his own secretary, were spending hours in front of the screen. They confided in ELIZA. They told it about their heartbreak, their doubts and their secrets. When an appalled Weizenbaum explained that it was nothing more than a stupid piece of code, they refused to listen. His secretary, who had watched him write the program line by line, asked him to leave the room so that she could speak to the machine in private.

What happened next is less well known, and worth hearing. Weizenbaum was so shaken that he spent the rest of his career warning people about what he had discovered. Ten years later, he published an entire book explaining that the danger was not that machines might become intelligent, but that human beings would place in them a trust they had never earned. He had built the most rudimentary thing imaginable, and emerged worried about the next fifty years.

The ELIZA effect

Our irresistible tendency to attribute understanding and intentions to a machine the moment it uses our words. The program does nothing to create them: we fill in the blanks ourselves.

The stochastic parrot

The label a group of researchers gave these machines in a widely discussed 2021 paper. “Stochastic” simply means “governed by chance.” A parrot picking words at random, but rolling dice that are extraordinarily heavily loaded.

If a handful of computer rules could fool geniuses in 1966, do not blame yourself for being impressed today. Modern machines have devoured entire libraries. They have mastered grammar, humour, sarcasm and register. They can write a poem, summarise a financial report and adapt to your tone.

Yet the raw truth remains the same as it was in 1966. When AI answers you, it is not thinking. It has no ideas. The word “apple” evokes neither a colour nor a taste, nor Sunday’s apple pie. It is a string of characters linked statistically to other strings of characters.

Think of an extraordinarily gifted, magical parrot. A parrot that has listened to millions of conversations and produces exactly the right sentence at exactly the right moment. You say “Hello,” and it replies, “How are you?” It sounds polite, but it could not care less about your health. It has simply learned that “hello” is very often followed by “how are you?”

At this point, I owe you the first warning promised in the introduction to this guide. The parrot is a useful image, but it has its limits, and we reach them quickly. A real parrot repeats what it has heard. These machines constantly produce sentences that nobody has ever written before. They do not copy; they recombine. Keep the parrot for what it explains so well: the complete absence of understanding behind perfectly chosen words. But when the time comes to understand where those unprecedented sentences come from, we will have to leave it on its perch and find another image. That is the subject of the next two chapters.

One consequence remains, and it is more unsettling than it first appears. The ELIZA effect works both ways. We know to be wary when the machine appears compassionate. We forget that the same caution is needed when it flatters us. When it begins with “Excellent question,” that is not an assessment but a turn of phrase. It finds you neither brilliant nor dull. It is stringing together words that it has learned go well together.

You are not speaking to an oracle. You are operating a highly sophisticated tool made of mathematics and lines of code. A learned parrot that has read almost the entire internet and retained none of it, at least not in the sense that you retain something.

The day you truly forget the brain in the box, something will change in the way you use it. You will stop labouring over how to phrase things. You will ask the same question three times in a row without feeling guilty. You will tell it bluntly that its answer is poor, because there is nobody whose feelings need sparing. You will take back control and dare to ask it anything.

But to master this parrot, we need to lift the bonnet. We need to understand the feat by which it turns our doubts, poems and questions into something it can manipulate: numbers.

That is exactly what we are going to explore. Take a deep breath. We are about to turn words into mathematics, and you will see that it is not nearly as complicated as it sounds.