The distorting mirror

We readily credit computers with an enviable virtue: objectivity. A machine has no moods. It does not get out of bed on the wrong side, vote for anyone, follow any religion or come from any particular background. It calculates, and calculation does not lie.

That is perfectly true of a calculator. It is false, and rather disturbingly so, of generative artificial intelligence.

In fact, you already have everything you need to understand why. Return to the infinite library of Chapter 4. The machine learnt to speak by reading billions of pages written by human beings. But we are not objective, and neither are our texts. They are steeped in beliefs, shortcuts, habits and clichés that we no longer even notice.

The machine does not sort. It counts.

And remember what counting means to it. In Chapter 2, we saw that it arranges words according to the company they keep. If, in the texts it swallowed, “nurse” appears a thousand times near “she” and “surgeon” a thousand times near “he”, those words move closer together on its map. There is no intention and no opinion involved. Only a distance shrinking, calculation after calculation.

Bias, then, is not a speck of dust that fell into the engine. It is the engine working normally.

It has no prejudices of its own. It has ours, transformed into geometry.

The clearest illustration of this trap came from Amazon, and the affair has become a textbook case.

The company received so many applications for technical roles that it wanted to automate the screening process. The idea seemed not only practical but virtuous: entrust selection to a cold machine, incapable of favouritism and free from the reflexes of a human recruiter. At last, candidates would be judged on their abilities alone.

To train it, engineers gave it the CVs of people hired during the previous ten years, with one simple instruction: look at the profiles of those who succeeded here, and find more like them.

The machine set to work. It began rejecting applications from women.

At first, the engineers searched for a bug. There was none. Nobody had programmed anything of the kind, and the system had not even been asked to consider the applicants’ sex. But it was an excellent statistician, and in ten years of records it had spotted an unmistakable pattern: the overwhelming majority of successful candidates had been men. Its mathematical conclusion, reached without the slightest malice, was that being male formed part of what worked there.

What followed is even more troubling. Because sex was not stated, the machine learnt to infer it. A CV mentioning that someone had captained a women’s team was penalised. The names of certain women’s colleges caused the score to fall. Nobody taught the machine to look for these clues. It discovered them by itself because they were statistically useful for reproducing the past.

Amazon abandoned the project. But remember the lesson, because it applies to everything else: the machine did not invent the imbalance. It measured it, then proposed perpetuating it with an efficiency no human recruiter could have matched.

You will encounter the same mechanism every day in subtler forms. Ask for a story featuring a doctor and a nurse, then observe who receives a name and who remains “the young woman”. Ask an image generator for a company executive and count the grey suits worn by men in their fifties. There is no racism or sexism involved: only an average, faithfully reproduced.

The word “average” brings us to the crack in this chapter’s metaphor, and it concerns the title itself.

A mirror reflects. Faithfully, adding nothing and taking nothing away. If the machine truly were a mirror, it would produce company executives in the exact proportions found in the real world, with visible minorities represented in their proper measure. That is not at all what it does.

It does not reflect. It averages. And an average crushes.

what is most common becomes the norm · what is less common becomes the exception · what is rare disappears

That is the real mechanism. The probable becomes the normal, and whatever exists only in small numbers simply ceases to exist. This is not a reflection of the world but a world ironed flat, with its margins removed. Keep the mirror for what it explains well, namely that all this comes from us rather than the machine. But remember that it lies about one essential point: this reflection is more of a caricature than the original.

Algorithmic bias

A statistical pattern in training data, reproduced and usually amplified by the system. Here, the word “ bias ” carries no moral meaning; it is a statistical term.

Automation bias

Our tendency to place greater trust in an answer because it came from a machine. The more dangerous of the two, because it resides in us.

Alignment

The training described in Chapter 4, when used to correct these tendencies. This is where the decisions discussed in the rest of this chapter are made.

Faced with this problem, designers naturally responded. Indeed, this is a central part of alignment: teaching the machine to reject certain associations, vary its examples and stop treating the masculine as humanity’s default setting.

But correcting a reflection first requires an answer to a question that has no technical solution. Correct it towards what?

Towards measured reality, with all its current imbalances? Towards a balanced representation that corresponds to no existing statistics? Towards the world we would like to see? Each of these options can be defended. None is neutral, and none follows from a calculation. They are choices, and somebody has to make them.

The most widely discussed incident in this area occurred in early 2024. A major image generator had been adjusted to make its representations more diverse, addressing a real and documented problem. But the instruction was applied mechanically, without regard for historical context. The result was that German soldiers in 1943 and the Founding Fathers of the United States appeared with a variety of backgrounds bearing no relation to the facts. The company suspended the feature within days.

What matters in this episode is neither the objective, which addressed a genuine problem, nor the clumsy execution, which is obvious. It is this: at no point was there an obvious setting to turn to. Correcting nothing meant endorsing overrepresentation. Correcting it meant deciding something on behalf of hundreds of millions of people. There is no position from which no choice is made.

And the difficulty does not end with images. How much space should minority positions receive on a scientific question? How should a conflict be presented when its story differs from one country to another? Which subjects should a system refuse to address, and according to whose customs, those of California, France, Japan or Saudi Arabia? Every answer will satisfy some people and scandalise others, and no setting can please everyone.

Hence the only honest conclusion I can offer you.

There is no such thing as a neutral AI. Left raw, it carries the habits of those who wrote its texts. Carefully aligned, it carries the values of those who adjusted it. Either way, it wears spectacles. The only real questions are who ground the lenses, and whether anyone told you.

Does that mean we should turn away from it? Of course not. That would be like refusing to read a newspaper because it has an editorial position. The sensible response is not rejection, but knowing which newspaper you are reading.

The most insidious problem remains, and I have saved it for last because it does not concern the machine. It concerns you.

A colleague tells you that a candidate is less suited to a role, and you ask why. A piece of software awards the same candidate 62 out of 100, and the urge to question the judgment evaporates. The number looks like a fact. It is not. It is an opinion fed through a mincer. Yet our critical faculties, which switch on automatically when faced with another person, remain dormant when faced with a screen.

This is automation bias, and it compounds every other bias. A biased machine that is questioned remains a tool. A biased machine that is believed becomes a judge.

That is why you should treat it as an interlocutor with ideas, not as a measuring instrument. Three habits are enough for everyday use. Look at what it produces when you give it no specific instructions: that is where its default settings become visible. Explicitly ask for several points of view or a varied set of examples; it does this very well when asked. And keep decisions that affect people in your own hands.

You now know that it does not think, that it arranges words by address, guesses what comes next, has been trained, sometimes invents, and wears spectacles. The bonnet is open, and you have seen everything inside.

There is only one thing left to learn, and it is the only one that depends entirely on you: how to talk to it.