What models actually "know"
A finance lead asks the model: “What were our Q3 sales last year?” The model returns a number, in the right order of magnitude, with the right ring of authority. The number is wrong — the model has no idea what the company’s Q3 sales were, and never did.
Earlier chapters touched the territory: What an AI model actually is showed that the same loop produces true and invented sentences, and Why models sometimes make things up named where hallucination concentrates. This chapter takes the third angle — the landscape of what a model knows well, knows shakily, and does not know at all, and the three operational patterns (verification, grounding, constraint) that production systems converge on to handle the gap.
What “knowing” means for a model
Section titled “What “knowing” means for a model”A model does not have facts the way a database does. It does not have lookup tables, search indexes, or memory cards. What it has are the patterns absorbed during training — the weights, set by trillions of next-word adjustments — and those patterns let it complete sentences in ways that often happen to be true.
When the model gets a fact right, it is not because it “remembered” the fact. It is because the patterns of language in its training data made the true answer also the most likely answer.
When the model gets a fact wrong, it is the same mechanism. The patterns made something else the most likely answer. There is no internal alarm that distinguishes “I am writing a true thing” from “I am writing a plausible-sounding thing.”
This explains a lot, including the confident wrongness about your Q3 sales.
The map of what models tend to know
Section titled “The map of what models tend to know”Imagine the model’s knowledge as a landscape. Some areas are dense and well-mapped. Some are sparse. Some don’t exist at all.
Densely mapped: common knowledge. Anything written about extensively in public sources — basic science, history, mainstream code patterns, common business concepts, popular books, well-known companies. The model has seen this material from many angles, so the patterns reinforce each other. It is usually reliable here.
Moderately mapped: specialized but well-documented topics. Medicine, law, finance, niche technical fields. There is good training data, but less of it. The model is usually correct on the basics and shaky on edge cases. It can confidently misstate a legal precedent or a specific drug interaction.
Sparsely mapped: niche, recent, or evolving topics. Specific events from the last year, new technologies, regional politics, small companies, individual people who aren’t public figures. Some training data exists, but not enough for the patterns to be reliable. The model often confidently invents details.
Not mapped at all: your business. Your products, your customers, your internal processes, your pricing, your quarter’s numbers. None of this was in the training data. The model has nothing to retrieve. When asked, it produces a plausible-sounding answer based on patterns from other businesses that look like yours. The output is almost always wrong on the specifics.
The knowledge cutoff
Section titled “The knowledge cutoff”Every model has a knowledge cutoff — the date its training data ends. After that date, it knows nothing. A model whose training data closed a year ago has no idea what has happened since then unless that information is brought to it at the moment of the question.
This is why models give outdated answers about elections, prices, leadership changes, and product releases. They are not “behind”; they are frozen at their cutoff.
Models do not usually announce their cutoff confidently. Ask “what’s your knowledge cutoff?” and the answer may be approximate, off by months, or worded vaguely. Treat the cutoff as a real constraint regardless.
Why “I don’t know” is rare
Section titled “Why “I don’t know” is rare”A trained model does not have a strong mechanism for recognizing the edges of its own knowledge. It is a next-word predictor. When asked something outside its training, it does not pause and check whether it knows the answer — it just produces the most likely next words.
For some questions, the most likely next words are an honest hedge: “I’m not sure,” “I don’t have specific information about that.” Many models have been instruction-tuned to produce these hedges in some situations. But the hedge is itself a pattern from training, not a genuine internal flag.
This is why a model can sometimes confidently answer a question it has no business answering, and other times helpfully say “I’m not sure” about something it actually knows. The difference is whether the question matches the pattern of “answer confidently” or the pattern of “hedge politely” — not whether the model truly knows.
Where the model is more vs. less reliable
Section titled “Where the model is more vs. less reliable”A rough field guide:
| Question type | Reliability | Why |
|---|---|---|
| Definitions, common concepts | High | Massive overlap across training data |
| Explanation of well-documented topics | High | Models have seen them explained many times |
| Code patterns from popular languages | High | Huge amount of training data |
| Specific numbers (recent, niche, internal) | Low | Pattern-completion fills in plausible numbers |
| Specific names, citations, dates | Low | Easy to invent something plausible-sounding |
| Recent events (post-cutoff) | None | Not in training data |
| Anything about your specific business | None | Never in training data |
| Subjective interpretations | Mixed | Often sounds reasonable; correctness is hard to check against any source |
The pattern: structural and conceptual knowledge is solid. Specifics — numbers, names, dates, anything individual — are where the model is most likely to confidently invent.
How real systems handle the gap
Section titled “How real systems handle the gap”Production AI systems that handle facts converge on three patterns, often in combination. They’re worth recognising because every serious “AI-powered” product is some mix of them.
Verification. A human reads the model’s output before it leaves the building. Drafting tools, code suggestions, internal research summaries — the model handles the draft, a person handles the verdict. This is the loosest pattern: it costs human time on every output, but it scales to almost any task.
Grounding. The relevant text is brought into the model’s context at the moment of the question, so the model summarises a source instead of guessing from training. Pasting a PDF into ChatGPT is the manual version; an enterprise search tool that pulls the right documents out of a knowledge base and feeds them to the model is the engineered version. Tools and memory walks through the machinery.
Constraint. The system only puts questions to the model where the model’s native knowledge is reliable. A drafting tool that writes generic email openers doesn’t need verification or grounding; the territory is dense in the model’s training data and the failure modes are mild. The same model asked to recite a specific company’s financial figures is the wrong tool for the job at any level of effort.
The three are not in opposition; most real deployments use all three in different places. The question for any given workflow is which pattern matches the territory it operates in.
A more accurate mental frame
Section titled “A more accurate mental frame”A common starting frame for the model is “smart, slightly fuzzy researcher.” This frame leads to the Q3-sales kind of surprise.
A sharper frame: the model is a very fluent writer who has read a great deal but has no notes, no internet, no calculator, and no memory of any specific person or organisation. It will produce coherent prose on almost any topic in almost any style. Truth, where it appears, is a side-effect of the patterns lining up with reality — not a property the model is checking against.
Holding that frame closer to the front of mind shifts what’s surprising. The model writing a clean paragraph about a topic it has seen a thousand times is the predicted behaviour, not a feat. The model inventing a name with a straight face on a question it has no real ground truth for is also the predicted behaviour, not a glitch. The model becomes more useful at the moment it stops being expected to do something the mechanism doesn’t support.