07 RAG · Retrieval-Augmented Generation
This might be the single most important idea in practical AI today. An LLM only knows what it was trained on — not last week's news, and definitely not your company's private documents. You can't cram 500 GB into a context window. So instead of baking knowledge into the model, you retrieve it the moment you need it. That's RAG.
The problem it solves
It doesn't know what happened after training. Ask about last week and it guesses.
Your internal docs, database, and research files were never in its training set.
You can't paste your whole knowledge base into every prompt. It won't fit.
RAG's answer to all three: keep the knowledge outside the model, and pull in just the few relevant pieces at question time — right into the context window, exactly when they're needed.
The pipeline
RAG is three phases. Index your documents once: split them into chunks, turn each chunk into a vector (a numerical fingerprint of its meaning), and store them. Then, per question: retrieve the chunks whose vectors are most similar to the question, and generate an answer with those chunks dropped into the prompt. Press play.
Done ahead of time, whenever your documents change.
Similarity search, not keyword matching.
The LLM now has the facts in front of it.
No hallucinations about things it doesn't know, no stale information, no "I don't have access to that." Just accurate answers grounded in your data.
The craft
Chunking is the make-or-break decision. Cut documents too small and each chunk loses the context around it. Cut them too large and retrieval gets imprecise — you drag in paragraphs of irrelevant text. A clever fix is hierarchical chunking: store tiny precise chunks for matching and larger parent chunks for context, so you get the precision of small with the completeness of large.
Precise matches, but each chunk is stranded without its surrounding context.
Full context, but fuzzy retrieval — the right sentence hides in noise.
Match on small chunks, then hand the LLM the larger parent for context.
Instead of a fixed pipeline, let the agent decide when and what to retrieve — do multiple rounds if the first didn't surface enough, and treat retrieval as just another tool in its kit. This is the most powerful pattern for complex, multi-step research. It's the agent loop from Chapter 4 with retrieval bolted on.
In real code
Strip RAG to its essence and it's four steps: embed the question, search the store, stitch the hits into a prompt, and ask. Here it is against a vector store you've already indexed.
# 1. embed the user's question — SAME model used to index q_vec = embed(question) # 2. retrieve: nearest chunks in the vector store hits = store.search(q_vec, top_k=3) # 3. build a grounded prompt from the retrieved chunks context = "\n\n".join(h.text for h in hits) prompt = f"""Answer using ONLY this context. Context: {context} Question: {question}""" # 4. generate the grounded answer answer = llm(prompt)
Chapter 7 in one breath
RAG indexes your documents as vectors, retrieves the few most relevant chunks for each question, and generates an answer grounded in them. It's how you give an LLM current, private, trustworthy knowledge. Next: the vector databases that make that retrieval fast.
A free Google Colab notebook — build this RAG pipeline with Chroma + Gemini embeddings on the free tier.
A matching slide deck with speaker notes — press S for notes, F for fullscreen.