02 How LLMs actually work
An LLM doesn't "know" facts or "understand" your question the way you do. Underneath, it does one thing over and over: predict the next token. Everything an agent can do is built on this single, humble trick. Let's watch it happen — and meet the two dials, temperature and the context window, that shape every answer.
Step zero
Before anything else, your text is chopped into tokens — chunks that are often whole words, sometimes word-pieces. Each token maps to a number (an ID), because a neural network only does math on numbers. "Attention" might be one token or two; a space usually rides along with the word after it. Watch a sentence become tokens.
The whole engine
Given everything so far, the model outputs a probability for every possible next token — often 100,000+ of them. It picks one, sticks it on the end, and feeds the whole thing back in to predict the next. That loop, called autoregression, is all a language model does. A paragraph is just this step run a few hundred times. Step through it:
Because it only predicts likely text, an LLM can sound completely confident and still be wrong — a hallucination. That single fact is why agents need tools, memory, and guardrails, which is the rest of this course.
Dial #1
The model gives a distribution — but how boldly should it pick? Temperature reshapes those probabilities before sampling. Turn it down and the top choice dominates: safe, repetitive, near-deterministic. Turn it up and the odds flatten: surprising, creative, riskier. Same model, same prompt — drag the slider and watch the distribution breathe.
Dial #2
A model can only "see" a fixed number of tokens at once — its context window. Everything in the window (your prompt, the chat history, retrieved documents) shapes the answer. But the window is finite: as a conversation grows past it, the oldest tokens slide out and are simply gone. The model isn't remembering your earlier messages — it's re-reading whatever still fits. Add tokens and watch the front of the conversation fall off the edge.
Roughly, as of 2025–26 — a token is about ¾ of a word:
1M tokens ≈ feeding it a small library at once.
More tokens means more cost, slower responses, and models can lose focus on things said earlier in a long conversation ("lost in the middle"). Window size is a real engineering trade-off — not just a bragging right.
No persistent memory + a finite window is a real limitation. The fix is giving the model an external memory and a search tool — retrieval-augmented generation (RAG), vector databases, and the agent loop. That's Chapters 6–9.
Chapter 2 in one breath
An LLM tokenizes your text, predicts the next token from a probability distribution, and repeats. Temperature decides how boldly it picks; the context window decides how much it can see. It has no memory of its own — and closing that gap is what turns a language model into an agent.
A matching slide deck with speaker notes — press S for notes, F for fullscreen.