Words turned into numbers

We have just seen that a machine does not think, feels nothing and has not the faintest idea what the words it uses actually mean.

So why does it never answer “table” when you ask about a “chair”? How can it write a poem about sadness without knowing what a tear is? If it does not understand the meaning of our sentences, how does it manage to create such a perfect illusion that it does?

The answer comes down to one word: geometry.

To a machine, the meaning of a word is nothing more than its position in space.

This may be the most dizzying idea in modern AI, yet you will grasp it in five minutes. Forget the code and picture a map.

Imagine that you had to arrange every word in the English language on a very large sheet of paper, with just one instruction: words about the same thing must sit next to one another.

You would write “dog” right beside “cat”. A little farther away, you would place “cow” and “horse”. You have just created the neighbourhood of domestic animals. The word “car”, meanwhile, has no business hanging around there. You take it to the other side of the page, near “lorry”, “road” and “engine”. Welcome to the transport district.

Now, where do you put “vet”? Somewhere between the animal neighbourhood and the medical district, naturally. And “kennel”? Right beside “dog”, but leaning slightly towards the houses.

Do this with the tens of thousands of words in the dictionary and you end up with a fascinating map, where the distance between two words tells you how closely related their meanings are. “Apple” and “pear” would live across the landing from each other. “Apple” and “tractor” would live in different countries.

That is exactly what engineers do when they teach language to machines. Except that they do not work on a two-dimensional sheet of paper.

Two dimensions are nowhere near enough. “Apple” would need to sit near “pear” for the fruit, near “computer” for the company and near “Somerset” for the cider, and a sheet of paper cannot be in three places at once. So the machine uses a map with hundreds, sometimes thousands, of dimensions. Impossible for the human mind to picture, child’s play for a computer. Every word receives precise coordinates, an address, a kind of GPS location in a space that nobody will ever see.

One question remains, the one everyone asks at this point, and it is the right one: who draws the map?

Nobody. That is where it becomes beautiful.

No team of engineers placed the words one by one, dictionary in one hand and mouse in the other. The map draws itself, following a principle formulated by a British linguist as far back as the 1950s, long before computers could make any use of it: you shall know a word by the company it keeps.

Take a word you have never seen before, “wampf”, and read these three sentences. The wampf slept in front of the fire. She took the wampf outside before nightfall. The neighbour’s wampf barks constantly. You did not need a definition. You simply looked at the company the word keeps, and you knew where it belonged.

The machine does the same thing across billions of sentences. It notices that “dog” and “cat” often appear surrounded by the same words, so it moves their two addresses a little closer together. It notices that “dog” and “bolt” never cross paths, so it pushes them apart. Repeat the operation billions of times and the map organises itself. Neighbourhoods emerge without anyone ever drawing their boundaries.

One detail in passing, because it explains a great deal of these machines’ strange behaviour. Strictly speaking, it is not words that receive an address, but pieces of words. Before any calculation begins, the text is chopped into fragments: common words make up a fragment on their own, while rare words are carved into several pieces. “Antidisestablishmentarianism” is sliced up like a salami. That is why an AI can stumble over the number of letters in a word while writing you a flawless essay: it does not see letters, it sees slices.

Token

One of these fragments of text. In English, a token averages roughly three quarters of a word. When an AI service charges “ by the token ”, it is charging by the slice of salami.

Embedding

The coordinates of a token on the map. There is no need to look for a plainer term: everyone uses the English one.

Vector, vector space

The list of coordinates, and the entire map. Two imposing expressions for an idea you have just understood in ten minutes.

And this is where something extraordinary becomes possible. Once words have become numbers and occupy precise positions, the machine can do mathematics with language. It can add and subtract ideas.

The most famous experiment dates from 2013. Researchers take the coordinates of the word “king”. They subtract the direction of the word “man”, add that of the word “woman”, then see which word lies at the destination.

king − man + woman = queen

Well, almost. They had to tell the machine one thing: it was not allowed to answer with any of the three words in the question. Without that instruction, it stops at “king”, which remains its nearest neighbour, and calmly hands your own question back to you as the answer.

That detail does not spoil the story. It makes it even more revealing. The machine had traced the right line and reached the right region of its map without ever suspecting that it was being asked a riddle. It does not know what royalty is. It does not know what gender is. But in its space, the distance between a king and a queen is the same as the distance between a man and a woman. It followed a parallel line, nothing more.

Try it again with cities. Paris, minus France, plus Italy. The machine moves across the map and lands on Rome. It has no idea what a capital city is. It has merely noticed that the relationship between a country and its principal city appears on its map as a movement of the same length and in the same direction.

This changes everything. By turning vocabulary into geometry, we gave the machine the power to manipulate concepts without ever needing to understand them.

When you ask it to translate, it does not consult a bilingual dictionary. It sees that “dog” and “chien” live in the same neighbourhood, amid the same constellation of ideas, and concludes that one is equivalent to the other. The same goes for synonyms: it does not ponder their meaning, it looks at who lives on the same street.

Think of a blind librarian who cannot read a single line on the shelves but knows the position of every volume down to the last millimetre. Ask for a book on gardening and he will retrieve it with his eyes closed, because he knows it sits between the one about tomatoes and the one about rakes.

It knows nothing, yet it knows where everything is.

Now for the crack in the picture. Because there is one, and I promised to show it to you every time.

A map is fixed. But consider this sentence: the barrister requested an adjournment. Now this one: the avocado was overripe. In French, the same word, avocat, means both things, yet the two addresses have nothing in common. If every word had one fixed and permanent position, the machine would have to choose once and for all, and would be wrong half the time.

That is precisely what happened with the earliest systems. A word had one address and one address only, and misunderstandings were simply part of the bargain. Today’s models do something different: a word’s address is recalculated in every sentence according to the words around it. The French word avocat has no permanent position. It has a temporary one, negotiated with its immediate neighbours. The map no longer merely exists; it redraws itself slightly every time you press Enter.

So keep the image of the map. It remains true where it matters: meaning is a matter of position and distance. Just strip away the glossy paper. This is not an atlas but a living map, shifting as your sentence unfolds.

And when the machine gets something wrong, when it brings together two ideas that had no business being neighbours, this is very often where you should look. Two concepts have become neighbours by accident because people on the internet wrote them side by side a thousand times. The map invents no prejudices. It records ours with depressing fidelity. We will return to this in Chapter 6; it deserves a subject all to itself.

You now know that your assistant does not manipulate ideas but addresses. One question remains, and it takes us to the heart of the reactor.

How does it write sentences that hold together? How does it arrange those words to tell you a story or explain a theorem, instead of merely spitting out a list of next-door neighbours?

The answer is the trade secret that allowed these machines to conquer the world in a matter of months. Brace yourself: we are about to enter the machine that guesses what comes next.