Anyone who works with AI tools knows the moment: the answer is fluent, well structured and sounds competent. And yet it is wrong. A source that does not exist. A year that is slightly off. A quote nobody ever said. This is called hallucination. The word suggests a malfunction. In fact, it follows directly from how language models work.
A language model knows probabilities, not truth
A language model does not look things up. Put simply, it calculates which word is most likely to come next, word by word. Because it was trained on enormous amounts of text, the most likely word is very often the right one.
The most likely word is not always the right one: Sydney instead of Canberra.
When the model does not know something, that process does not change. It still picks the most likely next word. The result is a sentence that sounds just as confident as a correct answer. Hesitation, a “maybe” or a hint of uncertainty does not appear on its own.
Why the AI doesn’t simply say “I don’t know”
In 2025, OpenAI published an explanation, together with a research paper. The core idea: language models are scored much like students on a multiple-choice test. If you guess, you sometimes get points. If you leave it blank, you certainly get none.
As in a multiple-choice exam, guessing earns more points on average than staying silent.
When scoring rewards guessing and penalises restraint, a model learns exactly that: guessing pays off. Hallucination is therefore also a result of how models are measured and improved.
What this means for teaching
The key insight for learners: a confident tone says nothing about accuracy. An AI answer is a draft, not a reference work.
Three checks for every AI answer.
Three questions help in everyday use:
- Can I find this in a second, independent source? Not in a second AI, but in a source that is itself backed up.
- Can names, numbers and quotes be verified? These details are especially vulnerable because they have to sound plausible but should be exact.
- What happens if I ask the same question differently? If the answer changes with a slightly different wording, that is a warning sign.
An exercise for your next lesson
Have learners work in groups and ask an AI about a topic they already know well. Each group uses the three questions to find a statement that is wrong and backs up the error with a source. The debrief usually shows quickly how convincing wrong answers can be. And it turns distrust into a technique that can be practised.
AI is not an encyclopaedia, it’s a very good guesser. Knowing that is what makes it useful.
Frequently asked questions
Why does AI hallucinate?
A language model calculates, word by word, which word is most likely to come next. When it doesn't know something, it still picks the most likely word, and the result sounds just as confident as a correct answer.
Why doesn't AI just say "I don't know"?
According to an explanation OpenAI published in 2025, language models are scored much like students on a multiple-choice test: guessing sometimes earns points, leaving it blank certainly earns none. So a model learns that guessing pays off.
How can you spot invented AI answers?
Three questions help: can I find this in a second, independent source? Can names, numbers and quotes be verified? What happens if I ask the same question differently?
