RAG explained: your AI never read the handbook

Many AI assistants for your own documents have never read the handbook. They only see a few snippets of it, picked by a search. The method is called RAG. Once you understand it, you can judge when such assistants answer reliably and when they don’t.

How RAG works

RAG stands for retrieval augmented generation: text generation supported by a search. Patrick Lewis and colleagues described the method in 2020.

It starts with preparation. All documents are cut into small snippets. Each snippet gets a numeric code for its meaning and goes into a database. Every question then goes through three steps:

1. Translate the question: your question is also turned into such a numeric code.
2. Search: the database looks for the snippets with the most similar code, usually just a handful.
3. Answer: only now does the language model come in. It gets your question and these few snippets and writes the answer from them.

The video simplifies in a few places. In technical terms, the snippets are called chunks, the numeric code an embedding and the database a vector database. The embedding is also produced by a neural network, a so-called embedding model, but it isn’t a language model that writes answers. Some systems also use a language model to rephrase the question or to re-rank the snippets found. How many snippets the model gets depends on the system.

The model only sees what the search finds

The language model never sees the whole handbook, only what the search picked beforehand. If the right snippet is missing, the model has nothing to stand on. Then it guesses, or says it can’t find anything. Connections across chapters easily get lost, because each snippet stands on its own.

Anthropic described the problem in 2024: traditional RAG systems remove context when they split documents, which often means they fail to retrieve the relevant information. A sentence about revenue growth doesn’t say on its own which company or which quarter it refers to. In Anthropic’s tests, adding context to each snippet reduced such failures considerably, but didn’t eliminate them.

The alternative: everything in the context window

The context window is everything a model can take into account when answering, a kind of working memory. According to Anthropic, current Claude models hold a million tokens. Tokens are the pieces a language model breaks text into: words, parts of words or punctuation marks.

How many pages that is depends on the text. Anthropic equates 200,000 tokens with about 500 pages; scaled up, a million tokens comes to around 2,500 pages. That’s a rough guide, not an exact conversion.

If you put everything in directly, the model really has it all in front of it. Studies show that this is often better. In 2024, Zhuowan Li and colleagues compared both approaches on several public datasets: given enough resources, long-context models performed better on average than RAG. Anthropic, too, recommends simply including the entire knowledge base in the prompt if it is smaller than 200,000 tokens.

The large window has limits

• It only scales up to the limit of the window. A whole library won’t fit.
• Every question costs more and takes longer. Billing is per input token, and all the material goes in with every question. Prompt caching lowers the cost, but not to the level of RAG.
• The fuller the window, the more likely the model misses details. In 2023, Nelson Liu and colleagues showed that models make worse use of information in the middle of a long text than at the beginning or end (“lost in the middle”). In 2025, Chroma tested eighteen models and found that performance drops as input grows, even on simple tasks. Anthropic’s own documentation calls this “context rot”.

RAG, on the other hand, has real advantages. It is cheap and fast. And because it is known which snippets were used, it can show where an answer comes from. Lewis and colleagues already named the lack of provenance as a weakness of language models without retrieval.

The rule of thumb

If it fits in the window, put it all in. If it’s a whole library, you need RAG. Or an agent that searches on its own.

An agent doesn’t pick snippets in advance. It searches with tools itself, checks what it finds and keeps searching if needed. Boris Cherny from the team behind Claude Code wrote in 2026 that early versions of the tool used RAG with a local vector database, but that agentic search generally works better and is simpler.

Check answers against the source

If you use AI assistants for your own material, such as course dossiers or regulations, a few things follow from this.

If an assistant shows which document an answer comes from, it’s worth taking a look. That is one of RAG’s strengths.

“Nothing found” doesn’t mean “doesn’t exist”. The search may simply have missed the right snippet. Try asking again with different words.

Questions that span several chapters call for caution. Comparisons, summaries of entire documents or contradictions between sections easily get lost with RAG. If the document fits in the window, it is often better to provide all of it.

A short test with a document you know well shows how reliable a particular assistant is. Three questions are enough: a detail question answered in a single section, a question that links two chapters, and a question whose answer isn’t in the document at all. Comparing the answers with the original quickly shows where the assistant is reliable. The results can also be worked through together with participants.

What exactly separates a model, a language model, a chat and an agent is explained in Model, LLM, chat, agent: four terms, one picture.

If you know what the model sees, you also know what it can’t know.