Lab for chapter A1
Lab: guess the next token
A toy model picks the next small piece of text, one piece at a time, and you move the dials that control the chance in that choice.
Before you start
This is a toy
The table of chances below is invented for this lab. It is not any real model's numbers, and the model in this page is a small script and a hand-written table, not a neural network. Everything runs on your device and nothing is sent anywhere.
- Token
- The small piece of text a model reads and writes. A token can be a whole word, a piece of a word, or a punctuation mark.
- Temperature
- A dial that controls how adventurous the pick is. Low means the leading choice nearly always wins. High means the other tokens get a real chance.
- Keep group (top p)
- A dial that keeps only the smallest group of leading candidates whose chances add up to the setting, and ignores the rest.
What to do
- Press Generate once. Watch the answer fill in. Each chip is one token, and the number on the chip is its position in the answer.
- Read the chips. A solid border marks a whole word, a dashed border marks a piece of a word, a dotted border marks punctuation, and a double border marks the stop token, which the model picks to say the answer is finished. The stop token is not counted in the answer length.
- Move the dials. Drag the temperature slider and the keep group slider, or focus a slider and press the arrow keys, and watch the bars in the chart change. The bars show the chances for the first token of the answer at your current settings. When keep group cuts a token, its line is struck through and shows 0 percent.
- Set both dials back to their start values of temperature 1.0 and keep group 1.00, then press Generate 10 times. Read the tally line: how many different answers came out of those ten runs?
- Set temperature to its lowest value, 0.1, and press Generate 10 times again. Compare the tally with step 4. The tally only counts your last ten runs.
- Lower keep group to 0.10 with temperature still at 0.1, and generate again. Only one bar survives in the chart, so only one answer is possible.
The lab
Answer generator
0.1 is steady, 2.0 is adventurous.
A lower setting keeps fewer candidates.
Chip key: solid whole word, dashed piece of a word, dotted punctuation, double stop.
Tokens in this answer: 0
Runs so far: 0
Chances for the first token
Press Generate to start. The bars show the chances for the first token of the answer at your current dial settings.
Last ten runs
No runs yet. Press Generate.
Ready. Press Generate.
What you should see
After ten runs at temperature 1.0 and keep group 1.00, most of your answers should be different. After ten runs at temperature 0.1, nearly all ten should be the same answer. On the page that is what happens, but your own clicks can land differently, because the chance in the choice is real.
Show what to expect
Each step has a clear leader, and the other tokens carry smaller chances. At temperature 1.0 those smaller chances are strong enough that several steps in a row almost never land the same way, so ten runs give about nine or ten different answers. At temperature 0.1 the leader's chance becomes overwhelming at every step, so ten runs collapse onto the one leading path.
At keep group 0.10 only the leading candidate survives, so every answer is the leading path no matter what the random picks try to do.
What this simulation leaves out
- A real model chooses from a very large set of possible next tokens, not a handful. The pick at each step can come from thousands of candidates.
- A real model's chances come from a neural network that reads the whole text so far, not just the last token or two. That is why a real model can stay consistent across a long answer while this toy drifts.
- The tokens here were picked by hand for this lab. A real tokeniser is a separate program with its own learned rules, and the same text can split differently on a different model.
- A real service can layer other sampling rules on top of temperature and keep group, and can change the numbers this lab shows. The lab shows one common pair of dials, not the whole picture.
- The temperature slider here runs from 0.1 to 2.0 so the effect is easy to see, and it has no 0.0 setting. The lesson quotes Anthropic's documented range of 0.0 to 1.0, so treat this slider as part of the toy.
- A real model can write a much longer answer. This one stops within twelve tokens on purpose, so the chips stay readable.
Read the lesson for chapter A1: What an AI language model is.