LLM hallucinations: why they happen and what it costs the business
A practical explanation of why language models confidently state things that are not true - and how to decide whether that risk is acceptable in your case.
Last month a client showed me a demo of an internal Q&A bot. It answered fluently, cited internal policies by name, and gave confident step-by-step instructions. About a third of the cited policies did not exist. The model had filled the gaps so smoothly that no one on the team had noticed during testing.
This is not a bug that will be patched in the next version. It is a structural property of how these models work. Understanding it takes about ten minutes, and that understanding changes how you scope AI projects.
What is actually happening inside the model
A language model is trained to predict the most statistically likely next token given everything before it. It does not "look things up" the way a database query does. It has compressed a huge amount of text into weights - patterns of what tends to follow what - and at inference time it reconstructs a plausible continuation.
When the prompt asks for something the model has not seen clearly in training, or something that requires precise factual recall, the model does not say "I don't know." It continues generating plausible-sounding text, because that is the only thing it knows how to do. The result is a confident, grammatically clean statement that happens to be false.
Why the confidence makes it worse
A human expert who is uncertain will typically hedge. A model trained on confident text generates confident text by default. Users who are not expecting errors will not check. That is the real operational risk: not that errors exist, but that they are invisible.
In a low-stakes context - brainstorming, draft generation, summarising your own documents - the risk is manageable. In a context where someone acts on the output without verification - legal, medical, financial, compliance - the cost of an undetected hallucination can be very concrete.
Where the risk is highest
From what I see in practice, hallucinations cause the most damage in three scenarios:
- Citation of specific sources. The model names a real document but describes content that is not in it, or invents a reference entirely.
- Numerical precision. Dates, figures, rates, thresholds - the model gets the magnitude right and the detail wrong.
- Edge cases in process questions. "What happens if I miss the deadline?" - the model describes a plausible but invented exception path.
What actually reduces the risk
RAG (retrieval-augmented generation) helps significantly in the first scenario because the model is grounded to actual retrieved text. But it is not a complete solution - a model can still misread or misrepresent what it retrieves.
The practical controls I recommend for any production use:
- Ground the model to a defined corpus. Do not let it answer from general knowledge when domain-specific knowledge is available.
- Surface source references to the user so they can check. Do not hide the retrieval step.
- Define the cost of an undetected error for your specific use case before you deploy.
- Build human review into workflows where that cost is high - not as an afterthought, but as a designed step.
The question to ask before each AI feature
For every AI-powered feature in your product or internal tool, ask: "If this output is wrong and no one catches it, what happens?" If the answer is "nothing serious" - proceed carefully but proceed. If the answer involves a regulator, a patient, or a contractual obligation - the architecture needs a verification layer before the output reaches a person who will act on it.
The technology is genuinely useful. The risk is genuinely manageable. But managing it starts with naming it accurately, not with hoping the next model version will fix it.