Short answer: because nothing inside the model checks whether an answer is true. It scores answers for how likely they look, and a convincing lie looks exactly as likely as the truth. The rest of this page is what that means and what to do about it.
You have probably seen it. You ask an AI a question, it gives you a confident answer, and the answer is completely, perfectly, hilariously wrong. A made-up court case. A book that does not exist. A quote from a famous scientist who never said it.
That is a hallucination. The word is misleading — the model is not having a vision. It is doing exactly what it was trained to do. The problem is that what it was trained to do is generate plausible-sounding text, not be correct.
Why LLMs hallucinate
Remember from part 1 — the model is autocomplete. When you ask it "what was the verdict in Smith v. Jones, 2008?", it does not have a database of court cases. It has a sense of what court verdicts tend to sound like. So it generates a court verdict that sounds right. Fluent, confident, formatted correctly. It is also entirely fictional.
The model has no built-in concept of "I do not know this." That is not a value it has access to.
Look at what a model actually produces — a probability for every word it knows. There is no entry on that list for "I'm not sure." Every path forward is a word, so it takes the most plausible one, and plausible court cases have names, dates and citation formats. You get one. Perfectly formatted, entirely fictional.
A model that has been trained to always give an answer will always give an answer.
Why it sounds so confident
This is the part that catches people. The fluency and the accuracy come from completely different places — and only one of them was trained.
Sounding right is what the model was optimised for, across trillions of words. Being right was never scored separately. So confidence is not evidence of anything: the model is exactly as fluent when it is wrong as when it is right, because fluency is the only thing it was ever graded on.
What helps (a little)
- Retrieval. Give the model real documents to look at, and tell it to quote from those. This is what search-augmented chatbots do, and it is the single biggest improvement available — see fine-tuning vs RAG.
- Calibration. Some newer models are trained to express uncertainty. They are still not great at it.
- Your skepticism. Treat every specific factual claim — names, dates, numbers, quotes — as a hypothesis to verify, not a fact to trust.
What does not help
- Asking the model "are you sure?" It will say yes. Then it will say no if you push. It is autocomplete.
- Trusting it more because the answer was confidently formatted. Confident formatting is the easy part.
The bottom line
Hallucinations are not a bug. They are the default behavior of a system that was trained to always sound right, with no separate training to know when it does not know. Plan your usage around this. The model is a draft generator, not a fact source.
Related
- How LLMs work — the next-word loop this all falls out of.
- Fine-tuning vs RAG — the two ways to give a model facts it does not have.
Next, the final part: how to actually use AI day-to-day, without the wheels coming off.