Next-token prediction, temperature, and the context window — the humble trick every agent is built on.
The one-sentence version
An LLM is autocomplete, scaled up absurdly.
It doesn't "know" facts or "understand" your question the way you do. Underneath, it does one thing, over and over:
predict the next token.
Everything an agent can do — tools, memory, reasoning — is built on top of this single move.
Step zero
Models don't read letters. They read tokens.
Your text is chopped into tokens — often whole words, sometimes word-pieces.
Each token maps to an integer ID, because a neural net only does math on numbers.
A space usually rides along with the word after it.
"Attention is all you need!" → 7 tokens → 15139 · 2939 · 318 · 477 · 345 · 761 · 0
The shape of the machine
Tokens in → one token out.
Goes in
The whole sequence so far, as integer IDs — your prompt, plus everything generated up to now.
Comes out
A single guess: a probability score for every possible next token — often 100,000+ of them.
One pass of the model = one token's worth of prediction.
Inside a single guess
A probability over every token. Pick one.
"The cat sat on the ___" — the model's top candidates for the next token:
The whole engine
Predict. Append. Repeat.
Pick a token from the distribution and stick it on the end.
Feed the whole longer sequence back in and predict the next one.
This loop is called autoregression — and it's all a language model does.
A paragraph is just this step, run a few hundred times.
The catch
It predicts likely — not true.
Because it only ever chases likely-sounding text, an LLM can be completely confident and completely wrong. That's a hallucination.
Tools
Let it check reality instead of guessing.
Memory
Give it facts it can look up, not invent.
Guardrails
Catch the confident-but-wrong answers.
This one fact is why the rest of this course exists.
Dial #1
Temperature: the creativity knob.
The model hands you a distribution — but how boldly should it pick? Temperature reshapes those probabilities before sampling.
Turn it down → the top choice dominates.
Turn it up → the odds flatten out.
Same model, same prompt — only the boldness changes.
Two extremes
Low temp vs. high temp.
low · 0.1
Focused
Always grabs the front-runner. Safe, repetitive, near-deterministic. Ask twice, get the same answer.
high · 1.5
Creative
Long-shots get a real chance. Surprising, varied, riskier. Ask twice, get two different answers.
Low temp ≈ a calculator. High temp ≈ a brainstormer.
Dial #2
The context window: its short-term memory.
A model can only "see" a fixed number of tokens at once. Add more, and the oldest fall off the edge:
Why it forgets you
The model has no memory of its own.
It isn't remembering your earlier messages — it's re-reading whatever still fits in the window.
Slide past the window and that context is gone for good.
The fix: give the model an external memory and a search tool.
retrieval-augmented generation (RAG) · vector databases · the agent loop
Chapter 2 in one breath
Tokens in, one token out — on a loop.
An LLM tokenizes your text, predicts the next token from a probability distribution, and repeats. Temperature decides how boldly it picks; the context window decides how much it can see.
It has no memory of its own — and closing that gap is what turns a language model into an agent.
Next up
Chapter 3 · Chatbots vs Agents.
We've got a next-token predictor with two dials and no memory. Now we give it goals, tools, and a loop — and watch a chatbot become an agent.