A language model starts every conversation from a blank slate. Memory isn't a setting you switch on — it's a system you architect around the model.
The core limitation
A model has no memory. Every chat starts fresh.
An LLM only ever sees what's inside its context window right now. Close the window, start again, and there's no hard drive quietly holding the last chat. It's like meeting someone with amnesia each time you walk in the room.
The fix isn't a smarter model.
It's external memory — a whole system you build outside the model. That's fine for a one-off chatbot; it's a real problem for an agent that has to remember what it did three hours ago.
See it fail
Watch it forget you.
Session 1: a user shares their name and preference — the model clearly understood. A new session starts. Same person, same assistant, one simple question:
The model didn't lose the memory — it never had one. Session 2 begins empty, and Monday is simply gone.
The architecture
The four kinds of memory.
Once you commit to building memory, you find it isn't one thing — it's four, each with a different lifespan and a different job. They map almost perfectly onto how human memory works.
Two live inside the model's context. Two live outside it, in storage. That split is the whole game.
Four layers, four lifespans
Sensory, working, episodic, semantic.
👁️ Sensory
The raw input the instant it arrives — the exact text, raw bytes, a tool's reply before it's read. Used or discarded in a single step.
~1 step
🧮 Working
Everything inside the active context window right now — chat so far, recent tool results, the current plan. Bounded by context size.
This session
📔 Episodic
Long-term memory of events and past interactions — what happened, and when. Stored externally, retrieved only when relevant.
Long-term · external
📚 Semantic
Long-term facts, preferences, and learned rules — the distilled knowledge, not the events. Also stored externally, pulled in when useful.
Long-term · external
Why it feels familiar
It's how your memory works too.
👁️ Sensory
The split-second echo of a sound, or the after-image of something you just glanced at.
🧮 Working
Holding a phone number in your head just long enough to dial it.
📔 Episodic
Remembering a conversation you had at dinner last week.
📚 Semantic
Knowing Paris is the capital of France — the fact stays; the moment you learned it is gone.
The four-layer design isn't arbitrary — it mirrors a system that already works: you.
The stack
Two inside the context. Two outside it.
The catch
Working memory has a hard ceiling.
Working memory is just the context window — the chat so far, recent tool results, the current plan. It's what keeps an agent coherent within a single run. But it's bounded by the context window size.
Grow past it, and the oldest pieces fall out.
That's precisely why the two long-term layers can't live here. If it mattered an hour ago and it isn't in the window now, it's gone — unless you wrote it somewhere outside the model.
Where it lives
Episodic & semantic live as vectors.
Working memory handles itself. The real engineering is the two long-term layers — and the trick is fetching the right few pieces back, not everything.
1 · Embed
Each memory becomes a vector — a numeric fingerprint of its meaning.
2 · Store
Vectors go into a database built to find nearest matches fast.
3 · Retrieve
A new situation is embedded the same way; the closest memories come back.
Sound familiar? It's the exact same machinery behind retrieval. Memory and RAG are two uses of one idea.
Looking ahead
This is RAG. And vector databases.
That "turn a memory into a vector and fetch the closest matches" move is retrieval — the heart of what comes next.
Chapter 7
RAG
Retrieval itself — pulling the right context in on demand.
Chapter 8
Vector databases
The store those vectors live in and get searched against.
The next two chapters aren't a detour from memory — they're the machinery that makes long-term memory work.
The idea to keep
Memory is something you architect.
An agent's intelligence is only as good as what it can remember — and memory is not something the model comes with. It's something you design, layer by layer.
Chapter 6 in one breath
The model forgets. You build the memory.
A raw LLM starts every session with amnesia. Agents fix it with four layers: sensory (raw input) and working (the live context) inside the model, plus episodic (events) and semantic (facts) held outside it. The long-term layers live as vectors and get retrieved on demand.
That's exactly where RAG picks up next.
Next up
Chapter 7 · RAG.
We've seen why agents need external memory and how it's stored as vectors. Next we open the retrieval engine that fetches the right piece back at the right moment — the heart of RAG.