Key Takeaways
- AI models are often penalised for admitting uncertainty, which pushes them to guess instead of saying “I don’t know”; this is a key reason behind so-called AI “hallucinations”.
- “Hallucinations” often occur when models fill in gaps using patterns from training data rather than verified facts.
- Ambiguous user prompts, routing between “fast” and “deep” reasoning paths, and grounding failures with external tools all contribute to unreliable answers.
- AI models don’t actually “know” or “believe” anything; they predict what text is most likely to come next, prioritising coherence and confidence over factual accuracy.
- Calling these errors “hallucinations” can be misleading because it shifts responsibility from developers and users to the AI itself, masking the human design choices that cause them.
Did you know the main reason AI systems hallucinate is that they’re penalised for admitting they don’t know something?
This was recently confirmed by an OpenAI report, which explains that during training and evaluation, AI models are rewarded for providing answers but receive no incentive for expressing uncertainty. With this scoring system, AI models are pushed to guess an answer rather than simply reply “I don’t know” when the necessary data is missing. This pressure to respond is a key factor behind the so-called AI “hallucinations”.
However, one interesting aspect is that AI models can sometimes “hallucinate” even when they possess accurate internal knowledge. This phenomenon is usually called “CHOKE”, which stands for Certain Hallucinations Overriding Known Evidence. Even more surprising is that newer, more advanced AI models, including ChatGPT-5, Gemini 2.5 Pro, and Claude Opus 4.1, tend to “choke” more often than older models. And the reason isn’t some mysterious failure, as many assume; it’s simply a design choice with predictable consequences.
In case you were wondering, here are the technical reasons why “hallucinations” happen:
- Statistical errors during generation – LLMs are primarily built to predict the next token* based on the patterns they’ve learned from their training data. They don’t have an internal fact-checker or any real concept of what’s “true” the way humans do. Instead, they generate what seems most likely based on the prompt and context. That’s why they’re brilliant at producing convincing, well-worded responses, but not always at getting the facts right.
*A token is the basic unit of text an LLM processes and generates. It might be a whole word, part of a word, a punctuation mark, or even a space. LLMs predict text one token at a time, using patterns learned during training, and string these tokens together to form coherent words, sentences, and paragraphs.
- Increased complexity and generalisation – Paradoxically, newer and larger AI models can sometimes produce more fabricated content. As these systems grow in complexity and become better at generalising across a wide range of topics, they also develop a stronger habit of confidently “filling in the gaps” when faced with ambiguous, open-ended, or multi-step reasoning tasks. This behaviour is especially noticeable in creative or long-form reasoning, where the model must stay coherent over many steps. Unless explicitly trained with anti-hallucination safeguards in place, these systems can generate responses that sound entirely plausible yet are completely made up. This is an unfortunate side effect of their expanded reasoning power.
- Retrieval and grounding gaps – Even when an AI model has access to accurate external sources or tools, it can still fail. That’s because the system needs to: 1) recognise when it should look something up; 2) retrieve the right information; 3) integrate that information correctly into its response. If it slips up at any point, it can produce the wrong answer even when the correct data is right there.
- Ambiguous or underspecified prompts – When a user’s request is too vague or open-ended, the model’s safest move, based on its training, is to make a plausible assumption and respond. From the model’s point of view, it’s being “helpful” by providing an answer. But from the user’s perspective, the answer may simply show that the model hasn’t quite understood the request.
- Model Routing and Internal Trade-offs – Newer architectures often route queries through “fast” and “deep” reasoning paths, which are similar to the System 1 and System 2 models in psychology. These routing decisions can enhance some abilities while weakening others, leading to seemingly paradoxical behaviour, such as improved reasoning on some tasks but poorer factual accuracy on others. Recent internal tests (PDF) have demonstrated this trade-off in practice.
It’s important to remember that LLMs don’t “think” or “know” in the way humans do. Their job is to predict the next most likely piece of text based on the patterns learned during training. When a model lacks solid grounding or has been optimised to prioritise qualities like coherence, relevance, or confidence over factual accuracy, it tends to “fill in the gaps” with something that appears plausible rather than something that is right.
Seen in this light, the so-called AI “hallucinations” aren’t signs of creativity, confusion, or technical failure. They’re simply the logical, yet unintended, outcome of a system doing exactly what it was built to do, namely, generate fluent, convincing text, regardless of whether it’s factually correct.
How AI “Hallucinations” Happen
Because understanding how “hallucinations” happen is trickier than understanding why they happen, here’s a simplified look inside an AI model’s “black box”.
1 - Sequential token generation
An LLM model generates text one token at a time. For each token, it evaluates probabilities for thousands of possible options at each step based on the patterns learned during training. Even a tiny error early in the process can cascade into larger issues, producing false or inconsistent statements later on. A good example is a coding model that hallucinates a non-existent method early in the function. Even if the rest of the code is logically sound, the initial hallucination makes the snippet unusable.
Example: An LLM is required to generate a Python function that calculates the area of a rectangle.
Correct signature
def calculate_rectangle_area(length,width):
return length * width
Hallucination
def calculate_rectangle_area(length,width):
return math.rectangle_area(length,width)
Why It’s an Error
The model has hallucinated a method math.rectangle.area() that doesn’t exist in Python’s math module. While the math module is real and commonly used, this particular method is fabricated, meaning the model has generated false information based on pattern matching rather than actual Python documentation.
Resulting Unusability
If the model continues generating code based on this hallucination, the errors cascade. The model might then generate additional code that relies on other non-existent methods.
Here’s how the initial error cascades in practice when the model generates additional logic:
def calculate_rectangle_area(length, width):
return math.rectangle_area(length, width) # Hallucination
def calculate_rectangle_perimeter(length, width):
return math.rectangle_perimeter(length, width) # Another hallucination
def calculate_all_properties(length, width):
area = math.rectangle_area(length, width)
perimeter = math.rectangle_perimeter(
length, width
)
diagonal = math.rectangle_diagonal(
length, width
) # Continues hallucinating
return {
'area': area,
'perimeter': perimeter,
'diagonal': diagonal
}
2 - Mixing patterns from training data
When faced with uncertainty or missing information, an AI model may combine fragments from different examples it has seen before. This can result in coherent answers that sound right but are completely made up.
Example: An LLM is required to provide the key facts about the Roman Emperor Trajan.
| Fact | Marcus Ulpius Traianus (Trajan) was the first Roman Emperor not born in Italy. He was born in the Roman province of Hispania Baetica (modern Spain), in the city of Italica. He reigned from 98 to 117 AD and is remembered as one of the "Five Good Emperors". | |
| Hallucination | Trajan built the great Roman lighthouse at Alexandria, Egypt, as his first act upon becoming Emperor. | |
| Why It Sounds Plausible The statement draws on genuine facts (Trajan was known for major building projects, Alexandria did have a famous lighthouse, and Roman emperors often began their reigns with symbolic acts). It's logically consistent, yet entirely false. |
Why it's False The Pharos of Alexandria (the lighthouse) was built centuries earlier, around 280 BC, under the Ptolemies. |
|
3 - Internal reasoning paths
Many models route queries through either “fast” (System 1) or “deep” (System 2) reasoning paths. “Fast” routes tend to prioritise speed and fluency, often at the expense of accuracy. “Deep” routes, on the other hand, follow longer chains of reasoning, which can improve accuracy but may also amplify early errors.
Example: Who holds the record for the most Olympic gold medals in swimming? (Correct answer: Michael Phelps, with 28 medals, 23 of which are gold.)
| Fast-Path Hallucination | A model using fast-path reasoning might quickly respond: “Mark Spitz holds the record with 9 gold medals”. How the hallucination occurred: When asked the above question, the AI model quickly associated high-probability names, like Phelps, Spitz, and Ledecky with “swimming” and “gold medals”. It then picked "Mark Spitz" because it matched the prompt’s context well and halted the process without proceeding to the deeper check necessary to compare Spitz's 9 medals against Phelps' 23. |
|
| Deep-Path Hallucination | A model using deep-path reasoning might generate a longer, detailed answer: “Michael Phelps holds the record with 24 Olympic gold medals, including one retroactively awarded after a 2016 relay appeal.” How the hallucination occurred: When asked the above question, the AI model correctly identified Phelps as the record holder and attempted to calculate the exact number of medals. However, an error occurred early in the sequence when assembling the details (e.g., misattributing one specific relay victory in the training data). This early, subtle mistake (generating 24 instead of 23) was then amplified by the deep reasoning process, causing the model to fabricate a detailed but false narrative to justify the incorrect number. |
|
4 - Failure to ground external information
Even when connected to external tools or databases, an AI model must first decide when to use them, then retrieve the right information, and integrate it correctly. An error at any stage can cause a “hallucination”.
Example: An AI model is asked to generate a weather report for London, pulling live data from an API.
| Step | Expected Action | Hallucination |
|---|---|---|
| API Call | The model queries the API for London's current temperature. | The model retrieves the numerical value 71.6. |
| Data Integration | The model is supposed to check the API metadata and convert the raw data (which is in Fahrenheit) into Celsius. | The model ignores or misreads the unit metadata. |
| Final Output | The model confidently reports: "The current temperature in London is 71.6°C." (Correct temperature: 22°C) | |
5 - Resolving ambiguous prompts
When a user’s request is vague, the model fills in the blanks with details that seem likely because that’s what its internal scoring system rewards.
Example: An image generator is asked to generate a “castle in the clouds”.
| Step | Expected Action | Hallucination |
|---|---|---|
| Interpretation | The model understands the core ideas: "castle” and "clouds”. | It selects high-probability visual elements associated with both words, even if some weren’t explicitly required. |
| Feature Combination | The model combines visual fragments to match “castle-like” and “cloud-like” features.. | It merges fragments from unrelated sources, creating elements that don’t logically fit together. |
| Final Output | The model generates a dreamy, cloud-castle scene. | The output includes mismatched details drawn from unrelated sources. See the image below for reference. |
6 - Cascading effects over long outputs
In long-form reasoning or creative writing, small early mistakes can multiply over time, leading to long, coherent but false narratives.
Example: An LLM is asked to write an essay about the impact of atmospheric carbon dioxide levels on global temperatures.
| Step | Expected Action | Hallucination |
|---|---|---|
| Initial Statistic | The model should state the correct CO₂ concentration (e.g., 420 ppm). | The model starts with an incorrect figure of 700 ppm. |
| Argument Amplification | The model should build arguments based on accurate data. | Instead, it constructs an entire essay around the false figure (700 ppm) and fabricates arguments to justify it. |
Even though the essay may be grammatically flawless and use sophisticated scientific terminology, thereby creating a convincing, coherent, and well-structured argument, it is erroneous simply because it began with a false premise.
Because AI models lack a fact-checking or consistency mechanism, they don’t stop to re-evaluate whether the numbers and claims make sense. Instead, they continue reasoning as if that false premise were true, producing a cascade of secondary statements that logically “fit” the incorrect number. This process is typically called error propagation or compounding hallucination.
Why Calling It a “Hallucination” is Dangerous
Describing these errors as “hallucinations” might sound harmless, but it subtly gives AI a sense of agency it doesn’t have. The word implies that the system experiences something, that it knows or believes, when in reality, it’s just following mathematical patterns.
That phrasing matters because it shifts accountability away from the people who design, train, deploy, and use these systems to the machine itself, as if the errors were a natural by-product of its own “mind”.
In doing so, it risks obscuring the real sources of failure, which are the human choices in data selection, model architecture, evaluation methods, and deployment decisions that ultimately shape why and how AI models produce false outputs. Over time, this framing may foster passive acceptance of AI’s mistakes, prompting us to treat them as inevitable rather than fixable. And that’s where the real danger lies.
Extra Sources and Further Reading
- AI Hallucinations: What Designers Need to Know – Nielsen Norman Group
https://www.nngroup.com/articles/ai-hallucinations/
This article discusses how AI systems can produce outputs that seem plausible but are incorrect or nonsensical. It emphasises the importance of understanding these limitations in design and user experience. - AI Hallucinations: A Guide with Examples – DataCamp
https://www.datacamp.com/blog/ai-hallucination
DataCamp provides a guide with examples of AI hallucinations, explaining how they occur and offering strategies to mitigate them. It’s a practical resource for understanding and addressing this issue. - A Call to Address AI “Hallucinations” and How Healthcare Professionals Can Mitigate Their Risks – NIH
https://pmc.ncbi.nlm.nih.gov/articles/PMC10552880/
This article discusses the real-world consequences of false AI outputs in high-stakes fields like healthcare, the ethical debate around the term “hallucination” itself, and the broader issue of trust erosion in AI systems. - Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded Explanations – Cornell University
https://arxiv.org/abs/2407.08983
This paper delves into LLM interpretability, focusing on the challenge of understanding how and why an LLM makes a specific decision.
