What an AI language model is

Watch

What a language model is

A large language model is a computer program trained on a very large amount of text. Its core skill is guessing what small piece of text comes next. A chat assistant is built on top of this kind of model, and the training that makes it follow instructions comes in the second stage below.

Tokens: the pieces of text

A token is a piece of text that the model reads and writes.² It can be a whole word, part of a word, or punctuation.

A short common word is often one token. A rare or long word can be split into several tokens, and punctuation counts too.

Byte pair encoding is one subword method for this: the model gets a fixed set of pieces, and rare or unknown words are encoded as sequences of those pieces.³

The README for OpenAI's tiktoken tokeniser says its byte pair encoding gives about 4 bytes of text per token on average in practice.² A byte is a unit of computer storage; one letter usually takes one byte.

Tokenisers: making the pieces

A tokeniser is the separate program that turns text into tokens before the model sees it.

Different companies use different tokenisers, so the same sentence can come out as a different number of tokens on different models.

Anthropic documents that its Claude 4.7 and later models use a newer tokeniser, and that the same input text produces approximately 30 percent more tokens than on its earlier models, as of 2026-10-18.⁴

One token at a time

Next-token prediction means producing chances for what the next token should be. The original Transformer paper describes a learned linear transformation and a softmax that turn the model's output into predicted next-token probabilities (a probability is a number from 0 to 1 for how likely an option is).¹

1. The model gives a chance to each option for the next token.
2. One token is chosen and added to the text.
3. The model runs again on the longer text, and the steps repeat.

The model's own output is fed back in as input. That is what autoregressive means: at each step, the previously generated tokens become additional input for generating the next one.¹ The answer is built token by token, not written out in one finished block.

Phone autocomplete is a useful analogy, up to a point. It also predicts what comes next, but it suggests a word and waits for you. A language model keeps going by itself, feeding each new token back in until a whole answer exists.

Training stage one: pretraining

Pretraining is the first stage. The model reads a huge amount of text and practises next-token prediction over and over, adjusting its parameters, the internal numbers that shape its output.

The GPT-3 paper describes an autoregressive language model with 175 billion parameters, pretrained on a large corpus of text.⁵ A corpus is a collected body of text.

After pretraining, the model's basic behaviour is to continue text. Instruction following is what the second stage works on.

Training stage two: post-training

Post-training changes how the pretrained model responds. In the InstructGPT work, people first wrote demonstrations of good answers and the model was trained on them. Then people ranked several model outputs from better to worse, and the model was further trained using those rankings.⁶

Human evaluators preferred outputs from the 1.3 billion parameter InstructGPT model to outputs from the 175 billion parameter GPT-3 model.⁶ The paper also states that making models bigger does not inherently make them better at following user intent.

Sampling: temperature and top p

Real systems choose the next token with some chance in the mix; that is called sampling. Always taking the highest scorer is one option.

Temperature is a dial for that chance. Anthropic's API documents it as the amount of randomness injected into the response, with a default of 1.0 and a range from 0.0 to 1.0.⁷ In plain terms, a low setting makes the top scorer win nearly every time, while a higher setting lets less likely tokens through more often.

Top p is a different dial. It sorts the options from most to least likely and keeps only the smallest leading group whose chances add up to the amount you chose. Options outside that group are ignored.⁷

As of 2026-10-18, Anthropic marks both settings as deprecated: models released after Claude Opus 4.6 do not support setting them, and a request that sets other values returns an error.⁷ The dials still show how sampling works.

The paper that introduced this nucleus sampling says it truncates the unreliable tail of the distribution.⁸

Research on nucleus sampling found that the decoding strategy alone can dramatically affect the quality of text from exactly the same model.⁸

You can try both dials in the lab: guess the next token. The lab uses an invented table of chances, not any real model's numbers.

Why the same question gets different answers

Chance is part of each choice, and a different token early in the answer changes the text the model sees next, so the rest can change too. The same question can therefore get different answers on different runs.

Turning the temperature down reduces that variation, but it is not a promise of identical output. Anthropic notes that even at 0.0 the results will not be fully deterministic (the same input is not guaranteed to give the same result every time).⁷

The training cutoff

The training cutoff is the date after which the model saw no more training data, so events after that date are not in its training knowledge. The gap grows over time.

Anthropic lists Claude Sonnet 5.5 with a training data cutoff of June 2026, as of 2026-10-18.⁹ On that date, the cutoff was about four months behind.

Information newer than the cutoff has to be supplied by the application, for example by a search tool putting its results into the request.

Hallucination: fluent and false

A hallucination is a confident, fluent statement that is false. It can sound natural and still be untrue.

Kalai and colleagues argue that language models hallucinate partly because training and evaluation reward guessing over acknowledging uncertainty.¹⁰

Treat an answer as a draft to check, not as the final word.

Two common misconceptions

A token is a word

No. A token can be a whole word, part of a word, or punctuation, and the pieces depend on the tokeniser. The same text can therefore have different token counts on different models.³⁴

A chat is one continuous session

No. A chat is a series of requests. The app resends earlier messages on each request, so the model sees the conversation as text each time.⁷

Terms used in this lesson

Token

A piece of text that the model reads and writes: a whole word, part of a word, or punctuation.

Tokeniser

The separate program that turns text into tokens before the model sees it.

Next-token prediction

Producing chances for which token should come next.

Autoregressive

Feeding previously generated tokens back in as input when generating the next token.

Pretraining

The first training stage: the model practises next-token prediction over a large amount of text.

Post-training

The second training stage, which shapes how the pretrained model responds.

Temperature

A dial for the amount of randomness used when choosing the next token.

Top p

A dial that keeps only the smallest leading group of token options whose chances add up to the chosen amount.

Training cutoff

The date after which the model saw no more training data.

Hallucination

A confident, fluent statement that is false.

Sources

1. Attention Is All You Need
2. OpenAI tiktoken README
3. Neural Machine Translation of Rare Words with Subword Units
4. Anthropic: token counting
5. Language Models are Few-Shot Learners
6. Training language models to follow instructions with human feedback
7. Anthropic: Messages API
8. The Curious Case of Neural Text Degeneration
9. Anthropic: models overview
10. Why Language Models Hallucinate