06 Memory · How agents remember
Here's the uncomfortable truth: a language model has no built-in persistent memory. Every conversation starts from a blank slate — like meeting someone who has amnesia each time you walk in the room. That's fine for a one-off chatbot. It's a real problem for an agent that needs to remember what it did three hours ago, or which way you like things done. The fix isn't a smarter model. It's external memory you build around it.
See it fail
Nothing makes the problem land like watching it happen. Below, a user tells the assistant their name and their preference. The assistant responds warmly — it clearly understood. Then a new session starts. Same person, same assistant, one simple question. Watch what comes back.
An LLM only ever sees what's inside its context window right now. Close the window, start fresh, and there is no hard drive quietly holding onto the last chat. Persistence is not a feature you turn on — it's an entire system you have to build outside the model.
The architecture
Once you commit to building memory, you quickly find it isn't one thing — it's four, each with a different lifespan and a different job. They map almost perfectly onto how human memory works. Tap a card to open it.
The raw input, the instant it arrives — before anything is done with it.
The most fleeting layer. It's the unprocessed input the moment it lands — the exact text you typed, the raw bytes of a file, a tool's response before it's been read. It exists for a single step and is immediately either used or discarded.
🧠 Human parallel: the split-second echo of a sound, or the after-image of something you just glanced at.Everything inside the active context window right now — the agent's short-term scratchpad.
This is the here-and-now: the chat history so far, recent tool results, the current plan. It's what lets an agent stay coherent within a single run. Its hard limit is the context window size — once the conversation grows past it, the oldest pieces fall out and are forgotten.
🧠 Human parallel: holding a phone number in your head just long enough to dial it.Long-term memory of specific events and past interactions — what happened, and when.
The record of things that happened: "last Tuesday I helped this user debug their deploy script". It survives across sessions because it lives in an external database, not the model. It isn't dragged into every prompt — it's retrieved when relevant and slotted into working memory just in time.
🧠 Human parallel: remembering a conversation you had at dinner last week.Long-term facts, preferences, and learned rules — the distilled knowledge, not the events.
Not what happened, but what's true: durable facts and rules like "this user wants concise bullets, always Python", or domain knowledge the agent has been given. Also stored externally and pulled in when useful. This is the layer that makes an agent feel like it actually knows you.
🧠 Human parallel: knowing Paris is the capital of France — the fact stays, the moment you learned it is long gone.The bridge
Working memory takes care of itself — it's just the context window. The interesting engineering is in the two long-term layers. Both episodic and semantic memory get written out to an external store, and the trick is fetching the right few pieces back at the right moment — not everything, just what's relevant to the task at hand.
The way that's done: each memory is turned into a vector — a numeric fingerprint of its meaning — and stored in a database built for finding the nearest matches fast. When a new situation comes up, the agent embeds it the same way and pulls back the closest memories. Sound familiar? It's the exact same machinery behind retrieval. Memory and RAG are two uses of one idea.
That "turn a memory into a vector and fetch the closest matches" move is retrieval — the heart of RAG (Chapter 7) — and the store it runs against is a vector database (Chapter 8). So the next two chapters aren't a detour from memory. They're the machinery that makes long-term memory work.
An agent's intelligence is only as good as what it can remember — and memory is something you architect, not something the model comes with.
Chapter 6 in one breath
A raw LLM starts every session with amnesia. Agents fix it with four layers: sensory (the raw input), working (the live context window), and two long-term stores held outside the model — episodic (events) and semantic (facts and preferences). Those long-term layers live as vectors and get retrieved on demand — which is exactly where RAG picks up next.
A matching slide deck with speaker notes — press S for notes, F for fullscreen.