Why it sometimes talks nonsense

Ask your assistant to polish a covering letter or translate a message and all will be well. It will excel. But start asking it for precise facts, dates, references, a summary of a book or a point of law, and sooner or later you will run into trouble.

One day, it will tell you something completely false.

Not a minor approximation. A fact invented from beginning to end, embellished with an admirable wealth of detail and delivered with the serene confidence of a professor at the end of a long career.

The phenomenon has a name, an attractive but misleading one: hallucination. We will see later why the word is poorly chosen, but let us begin by understanding where the phenomenon comes from.

It all comes down to one sentence I left you with at the end of Chapter 3.

The machine seeks coherence, never truth. It cannot tell the difference.

Remember its sole function: to guess the next word, to produce a sentence that sounds right, one that resembles something a human might have written. Yet the form of a sentence and the truth of a sentence have nothing to do with one another.

Take this one: “Napoleon won the Battle of Waterloo using attack helicopters.”

To a historian, it contains two absurdities in a single sentence. To a language model, it is magnificent. The subject is in its proper place, the verb correctly conjugated, the object neatly positioned, the rhythm impeccable. Nothing in the mechanism allows it to distinguish that sentence from an accurate one. Nothing was designed for that purpose.

When you ask a specialised question, the machine begins its calculation. If it has read a great deal about the subject, the path naturally leads to the correct answer, because the right words are also the most probable ones. But if the subject is rare, if your question is poorly phrased, or if you ask for a precise reference in an obscure field, it finds itself in a hazy region of its map.

And there, it carries on. That is the point.

It does not stop, apologise or say that it has no idea. It calculates the most probable word for a convincing sentence, then the next one, then the next. It produces something with the shape of truth, the scent of truth and the music of truth, but which is false.

You are probably wondering why it does not simply say, “I don’t know.” That is the right question, and the answer lies in the previous chapter.

Remember the training. For months, people rated its answers, awarding higher scores to those that were useful, thorough and helpful. Now imagine an examination in which a correct answer earns one point, a wrong answer earns none, and a blank answer also earns none. What does a rational student do when faced with a question they cannot answer? They take a chance. They have nothing to lose and everything to gain.

The machine sat this examination millions of times. It learnt exactly what we taught it: a plausible attempt is always preferable to an admission of ignorance. We created a student who never leaves an answer blank.

Sometimes, the price of that lesson is very high. To understand just how high, let us travel to New York in the spring of 2023.

Steven Schwartz has been a lawyer for thirty years. He represents a client who was struck by a service trolley during a flight and is suing the airline. The opposing side applies to have the case dismissed, and Schwartz needs to show that the law is on his side.

He has what seems to him an excellent idea. Instead of spending hours in the courthouse library searching through old judgments, he opens ChatGPT and asks it for comparable case law.

The assistant obliges with exemplary zeal. It supplies a list of previous rulings. It cites one case against a Chinese airline and another against an American carrier. It gives dates, case numbers, judges’ names and entire quotations from the decisions. Everything is in the correct format. Everything sounds perfectly right.

Schwartz is delighted. He incorporates this fine work into his submission and sends it to the court.

What follows unfolds over three days. The airline’s lawyers search the official archives for the rulings. They find nothing. The judge searches in turn. Nothing there either. The cases do not exist, the numbers lead nowhere, and the quotations were never written.

But that is not the most interesting part. What matters most is what happened in the meantime. Faced with such impeccable references, someone at the firm had nevertheless begun to wonder. They went back to the machine and asked whether the cases were real.

It confirmed that they were. Without the slightest hesitation.

Why? Not out of malice, as you now know. When asked for case law, the machine positioned itself in the neighbourhood of aviation law and calculated what needed to follow to produce a plausible legal text. Claimants’ names that sounded right, numbers in the proper format, judges’ language imitated to perfection. A masterpiece of coherence, a disaster of truth.

And when asked whether it was true, it did exactly the same thing: it calculated that, in a polite conversation, the most probable continuation after “are these cases real?” was to reassure the person asking. It did not check. It has nothing to check. Once again, it guessed what came next.

Schwartz was sanctioned, his firm publicly humiliated, and his name will remain attached to the case for a long time.

Hallucination

The established term for an invented answer presented as fact. Some researchers prefer “confabulation”, which describes the phenomenon more accurately: filling a gap with a coherent story, without any awareness of doing so.

Grounding, or RAG

The technique of giving the machine relevant documents before it answers, allowing it to draw from them instead of its statistical fog. This is what assistants do when they search the internet.

Sycophancy

Its tendency to agree with you. You encountered it in Chapter 4, and it explains why the machine confirms its own inventions when you ask it to verify them.

It is time to break the metaphor, and this time the problem lies in the official term itself.

“Hallucination” suggests a temporary malfunction, a fever, an accident in an otherwise healthy machine. The image is wrong from beginning to end. The machine does not go off the rails when it invents case law. It does exactly what it does when giving you the capital of France: it assembles the most probable words. The mechanism is strictly identical in both cases.

What changes is not the state of the machine, but our luck. When a subject is extensively documented, what is probable and what is true coincide, and we call the result a good answer. When they diverge, we call it a hallucination. The machine has never changed modes, and more importantly, it has no way of knowing which of the two situations it is in.

Turn that sentence around and you have the real lesson of this chapter: error is not the anomaly. Accuracy is the fortunate side effect.

You may be thinking that all this will eventually be solved, that tomorrow’s larger and better-trained models will stop making mistakes. It is a reasonable hope, and one we must abandon.

Several research studies have shown that this is a mathematical consequence of the method rather than a teething problem. As long as a machine produces the most probable continuation, it can produce a continuation that is probable and false. We can make these errors rarer, hunt them down, require the model to cite its sources and train it to recognise its ignorance. We will never eliminate them. This is not a disease that the next version will cure. It is the nature of the animal.

Hence one rule, the commandment that governs this entire guide: always check.

You still need to know where to concentrate your efforts, because the risk is not the same everywhere. Broadly speaking, this is how it is distributed.

document provided low risk · well-documented subject moderate · reference, figure, quotation, precise date maximum

If you provide the text to be summarised yourself, you confine the machine within narrow boundaries and the risk falls. It does not disappear: the machine can still invent a logical connection that is absent from the original or attribute a conclusion to the author that was never drawn. Ask it about a broad, extensively documented subject and it will almost always manage well. But ask for a precise reference, a figure, a quotation or the number of a section of law, and you are playing roulette. That is exactly where you need to check, and, by a cruel irony, it is also the part of the answer that looks most authoritative.

Never use it as a search engine. A search engine retrieves documents that exist. An AI creates a new text every time. One is a researcher; the other is a smooth talker.

The real trap, in the end, is the perfection of its delivery. When a human invents something or makes a mistake, something gives them away. They hesitate, search for words, qualify their claims, avoid your gaze. The machine never stammers. It writes an absurdity with the same careful punctuation, the same flawless syntax and the same courtesy it uses to state the obvious. Its confidence never wavers because it is not confidence at all: there is nobody there to doubt.

Learn, then, to read it with benevolent suspicion. It is a remarkable tool for structuring an outline, opening up new avenues, rephrasing, correcting and escaping the blank page. It is not an oracle, and it never will be.

The next time it produces a fact that seems a little too good, remember the New York lawyer. Thank it politely, then go and check.

One final obstacle remains before we move on to practice. It is less obvious than this one and far harder to detect, because it comes neither from mathematics nor from chance. It comes from us.

We are going to talk about the distorting mirror.