Don't bake knowledge into the model — retrieve it the moment you need it. That's Retrieval-Augmented Generation.
The problem it solves
Three walls every raw LLM hits.
A cutoff date
It only knows what it was trained on. Ask about last week's news and it guesses.
No private data
Your internal docs, database, and research files were never in its training set.
A finite window
You can't cram 500 GB into every prompt. Your whole knowledge base won't fit.
A frozen brain that can't see your data and can't hold it all at once.
The idea
Don't bake knowledge in. Retrieve it.
Keep the knowledge outside the model. At question time, pull in just the few relevant pieces — right into the context window, exactly when they're needed.
The model stays small and general. Your data stays fresh and private.
The pipeline
Index → retrieve → generate.
1Index
Done once, ahead of time. Chunk your documents, embed each chunk into a vector, and store the vectors.
2Retrieve
Per question. Embed the question the same way, then find the chunks whose vectors are most similar.
3Generate
Per question. Inject those chunks into the prompt and let the LLM answer, grounded in them.
Index once when documents change. Retrieve and generate on every question.
The flow
From documents to a grounded answer.
Phase 1 · Index
Turn documents into vectors.
Done ahead of time, whenever your documents change. Four moves:
Gather + chunk
Collect your PDFs, wikis, and records, then split each into small, self-contained chunks.
Embed + store
Turn every chunk into a vector — a numerical fingerprint of its meaning — and save it in a vector database.
One chunk → one vector like [0.12, −0.4, …]. Do it once; reuse it for every question.
Phases 2 & 3 · Retrieve + Generate
Per question: find, then answer.
2Retrieve
Embed the question the same way you indexed. Find the nearest chunk vectors — similarity search, not keyword matching — and pull the top matches out.
3Generate
Inject those chunks and the question into the prompt: "Answer using this context." The LLM now has the facts in front of it and answers, grounded.
Grounded, current, sourced — no hallucinating what it doesn't know.
A real question
"What's our refund policy for annual plans?"
Retrieved · 0.91 — …annual plans are refundable within 30 days…
Retrieved · 0.88 — …refunds are prorated after 30 days…
Retrieved · 0.83 — …contact billing to request a refund…
Grounded answer — Annual plans are fully refundable within 30 days; after that, refunds are prorated. To request one, contact billing. Sourced from your docs, not guessed.
The craft
How you chunk decides how well it works.
Chunking is the make-or-break decision. Cut wrong and even perfect retrieval fails.
Too small
Precise matches, but each chunk is stranded without its surrounding context.
Too large
Full context, but fuzzy retrieval — the right sentence hides in noise.
Hierarchical ✓
Match on small chunks, then hand the LLM the larger parent for context.
The precision of small, the completeness of large.
Levelling up
Agentic RAG: let the agent decide.
Instead of a fixed pipeline, treat retrieval as a tool the agent can reach for — and let it decide when and what to retrieve.
Decides to search
The agent chooses whether a question even needs retrieval at all.
Multiple rounds
If the first search falls short, it reformulates and searches again.
Just a tool
Retrieval slots into the agent loop from Chapter 4 as one more tool.
The most powerful pattern for complex, multi-step research.
In real code
The whole pattern, minus the frameworks.
# 1. embed the user's question — SAME model used to index
q_vec = embed(question)
# 2. retrieve: nearest chunks in the vector store
hits = store.search(q_vec, top_k=3)
# 3. build a grounded prompt from the retrieved chunks
context = "\n\n".join(h.text for h in hits)
prompt = f"""Answer using ONLY this context.
Context:
{context}
Question: {question}"""
# 4. generate the grounded answer
answer = llm(prompt)
Strip RAG to its essence and it's four steps. Step 1's comment matters: index and query must use the same embedding model.
Chapter 7 in one breath
Don't bake in knowledge — retrieve it.
RAG indexes your documents as vectors, retrieves the few most relevant chunks for each question, and generates an answer grounded in them. It's how you give an LLM current, private, trustworthy knowledge.
Index once. Retrieve and generate on every question.
Next up
Chapter 8 · Vector Databases.
Retrieval hinges on finding the nearest vectors — fast, across millions of them. Next we open up the vector database that makes that search possible.