A language model gives a probability distribution over the next word. Given the prefix The, it might say nice with probability 0.35, dog with 0.30, car with 0.15, and so on. That distribution alone doesn't produce a sentence. Something has to decide, at every step, which word to actually write down.
That decision rule is the decoding strategy, and it matters a lot. The same model can produce dull, repetitive text, creative but rambling text, or nonsense, depending only on how we pick words from its distributions. This reading walks through the standard strategies on one small running example, so you can see exactly what each one does to the same numbers.
1Generating text one word at a time
A language model assigns a probability $P(x)$ to every piece of text $x$. We can use it in two ways: to score a given text (how likely is it?) or to generate a new text, $\tilde x\sim P(\mathcal X)$. Generating a whole sentence at once is impossible because there are infinitely many sentences, so we generate autoregressively: pick one word, append it, and use the longer prefix to pick the next one.
For a trigram model, which approximates
$$P(w_1,w_2,\dots,w_l)\approx\prod_i P(w_i\mid w_{i-2},w_{i-1}),$$the generation loop is:
- Generate the first word $w_1\sim P(w\mid\text{<s>},\text{<s>})$.
- Generate the second word $w_2\sim P(w\mid\text{<s>},w_1)$.
- Generate the third word $w_3\sim P(w\mid w_1,w_2)$, the fourth $w_4\sim P(w\mid w_2,w_3)$, and so on.
- Stop when the model produces the end-of-sentence token </s> (some systems call it <eos>).
Here $\sim$ means “draw a random sample from.” Try it with small n-gram models trained on a toy corpus.
Show training corpus
Even a good n-gram model has a hard limit: it only sees the last $n-1$ words. In “Alice could not go to work that day because she had a doctor’s appointment,” choosing she rather than he depends on a word nine positions back, far outside any practical n-gram window. Models with longer context fix that problem, but they still output one distribution per step, so they still need a decoding strategy. Everything below works the same way for any language model.
2The running example
To compare strategies, we'll use one tiny language model throughout. Generation always starts from the prefix The, and we only look at the next two words.
| $w_2$ | $P(w_2\mid\textit{The})$ |
|---|---|
| nice | 0.35 |
| dog | 0.30 |
| car | 0.15 |
| cat | 0.10 |
| house | 0.06 |
| sky | 0.04 |
| prefix | $P(w_3\mid\textit{The},w_2)$ |
|---|---|
| The nice | woman .40 · house .30 · guy .20 · day .06 · car .04 |
| The dog | has .90 · runs .05 · and .03 · is .01 · barks .01 |
| The car | is .50 · drives .30 · turns .20 |
| The cat | sat .50 · is .30 · runs .20 |
| The house | is .60 · has .40 |
| The sky | is .90 · was .10 |
Two features of this model drive everything that follows. First, the distribution after The is flat: several words are reasonable. The distribution after The dog is peaked: has takes 90% of the mass. Second, the locally best first word (nice) is not on the globally best path, as section 6 will show.
3Random sampling and greedy decoding
The two simplest strategies are at opposite extremes.
Random sampling draws each word from the model's distribution exactly as given:
$$w_t\sim P(w\mid w_{<t})$$Every word with nonzero probability can be chosen, with chance equal to its probability. Running it twice usually gives different text.
Greedy decoding always takes the single most likely word:
$$w_t=\arg\max_{w\in\mathcal V}P(w\mid w_{<t})$$It's deterministic: the same prefix always gives the same output. Here $\mathcal V$ is the vocabulary.
Run sampling 1,000 times and you'll see roughly 27% The dog has and 14% The nice woman, matching the sequence probabilities. You'll also see rare outputs like The sky was or The dog barks, picked from the low-probability tail. With six words that's harmless. With a real vocabulary of 50,000 words, the tail holds thousands of bad options whose probabilities add up to a lot, and pure sampling regularly lands in it. That produces incoherent text.
Greedy decoding avoids the tail entirely, but it has its own problems. It gives no variety, and in practice it tends toward bland, repetitive text, because the most likely next word is often a safe, generic one. It also isn't guaranteed to find the most probable sentence (section 6).
4Temperature scaling
A neural language model doesn't output probabilities directly. It outputs a real-valued score $z_i$ (a logit) for every word in the vocabulary, and a softmax turns the scores into a distribution:
$$P(w_i)=\text{softmax}(z_i)=\frac{e^{z_i}}{\sum_j e^{z_j}}$$Temperature scaling divides every logit by a temperature $T>0$ before the softmax:
$$P_T(w_i)=\text{softmax}\!\left(\frac{z_i}{T}\right)=\frac{e^{z_i/T}}{\sum_j e^{z_j/T}}$$Dividing by a small $T$ stretches the gaps between logits, so the largest one dominates more. Dividing by a large $T$ shrinks the gaps, so the scores become more similar. For a model that gives probabilities $p_i$ directly, you can use $z_i=\ln p_i$; then $P_T(w_i)$ is simply $p_i^{1/T}$, renormalized.
- T < 1 makes the distribution sharper. At $T=0.5$, nice rises from 0.35 to 0.49 and sky falls from 0.04 to under 0.01. As $T\to0$, all the mass goes to the top word, which is exactly greedy decoding.
- T = 1 leaves the distribution unchanged.
- T > 1 makes it flatter. At $T=2$, sky rises to about 0.09. As $T\to\infty$, every word gets the same probability, $1/|\mathcal V|$.
5Cutting the long tail: top-K and top-P sampling
Word distributions are long-tailed: a few high-probability words, then a very long tail of words that are each unlikely but collectively carry a lot of mass. Temperature reshapes the whole distribution but never removes the tail. The next two methods cut it off explicitly, then sample from what's left.
Top-K sampling keeps only the $K$ most probable words, sets everything else to zero, and renormalizes:
$$P'(w)=\begin{cases}\dfrac{P(w)}{\sum_{w'\in V_K}P(w')} & w\in V_K\\[4pt] 0 & \text{otherwise}\end{cases}$$where $V_K$ is the set of $K$ most probable words. With $K=3$ after The, the kept words nice, dog, car have total mass 0.80, so they become $0.35/0.80\approx0.44$, $0.30/0.80\approx0.38$, and $0.15/0.80\approx0.19$.
Top-P sampling (also called nucleus sampling) keeps the smallest set of top words whose total probability is at least $P$, then renormalizes the same way. With $P=0.75$ after The: nice gives 0.35, adding dog gives 0.65 (not enough), adding car gives 0.80 ≥ 0.75, so three words are kept.
The two prefixes show why top-P is usually preferred. With $K=3$, top-K keeps three words no matter what. After The that's reasonable. After The dog it keeps runs and and, which together had only 8% of the mass. Top-K can also cut too much: with a flat distribution over many good words, a small $K$ throws good options away. Top-P adapts to the shape: at $P=0.75$ it keeps three words after the flat The but only has after the peaked The dog.
6Greedy is not always optimal: beam search
Suppose we want the single most probable output, not a random one. Greedy decoding looks like it should find it, but it doesn't. In the running example, greedy picks nice (0.35) over dog (0.30), then woman:
$$P(\textit{The nice woman})=0.35\times0.40=0.14$$But the sequence it skipped is far more likely:
$$P(\textit{The dog has})=0.30\times0.90=0.27$$Greedy commits to the best word at each step and can never go back. A slightly worse first word can open up a much better continuation.
Checking every possible sequence would fix this, but with vocabulary size $|\mathcal V|$ and length $L$ there are $|\mathcal V|^L$ sequences, which is far too many. Beam search is the practical middle ground. It keeps the $m$ best partial sequences (the beam, with beam size $m$) at every step:
- Expand: extend every sequence in the beam by every possible next word, and score each candidate by its total probability (in practice, the sum of log-probabilities).
- Prune: keep the $m$ highest-scoring candidates and discard the rest.
- Repeat until the sequences end with </s>, then return the most probable complete sequence.
With $m=2$, step 1 keeps nice (0.35) and dog (0.30). Step 2 expands both, giving ten candidates. Ranked by total probability, the top two are The dog has (0.27) and The nice woman (0.14). Dog was behind after step 1 but ends up ahead, which greedy decoding could never discover. With $m=1$ the beam holds only nice, and you get the greedy answer again.
7Beam size and length bias
Choosing the beam size
- Larger $m$ usually finds higher-probability outputs, because it explores more paths. It still isn't guaranteed to find the best sequence.
- Larger $m$ costs more time. Each step scores $m\times|\mathcal V|$ candidates.
- Larger $m$ gives less diverse outputs. Beam search is deterministic and tends to converge on safe, generic text. Typical beam sizes are small, around 4–10.
Length bias
Every extra word multiplies the score by another probability below 1, so longer sequences almost always get lower total probability. Beam search therefore favors short outputs: a hypothesis that ends with </s> early can beat a longer, better one simply because it has fewer factors.
| Hypothesis | length L | per-token probabilities | $\sum\log P$ | $\frac{1}{L}\sum\log P$ |
|---|---|---|---|---|
| A (short) | 4 | 0.4, 0.5, 0.5, 0.6 | −2.81 | −0.70 |
| B (long) | 8 | 0.65 each | −3.45 | −0.43 |
The usual fix is length normalization: rank complete hypotheses by the average log-probability per token instead of the total,
$$\text{score}(w_1,\dots,w_L)=\frac{1}{L}\sum_{i=1}^{L}\log P(w_i\mid w_{<i}).$$Many systems use a softer version, dividing by $L^\alpha$ or a similar length penalty with $0\le\alpha\le1$, so that $\alpha$ controls how strongly length is rewarded.
8Putting it together
| Strategy | Deterministic? | Main idea | Typical weakness |
|---|---|---|---|
| Random sampling | No | sample from $P$ as is | falls into the long tail |
| Greedy decoding | Yes | take the arg max each step | dull, repetitive; misses better sequences |
| Temperature | No | sharpen ($T<1$) or flatten ($T>1$) | doesn't remove the tail |
| Top-K | No | sample from the top $K$ words | fixed $K$ ignores the distribution's shape |
| Top-P (nucleus) | No | sample from the top mass $P$ | still random; needs a good $P$ |
| Beam search | Yes | keep the $m$ best partial sequences | slower; generic; length bias |
A rough rule of thumb: when there's essentially one right answer, such as translation or summarization, search-based methods (greedy or beam search) work well. For open-ended generation, such as stories or chat, sampling with a moderate temperature and top-P usually gives more natural and varied text.
9Check your understanding
Exercise 1: temperature by hand
A model gives $P=(0.5,\,0.3,\,0.2)$ over three words. Compute the distribution at $T=0.5$ and at $T=2$. (Use $z_i=\ln p_i$, so $P_T(w_i)\propto p_i^{1/T}$.)
$T=2$: take square roots: $0.707,\,0.548,\,0.447$, sum $1.702$, so $P_2\approx(0.415,\,0.322,\,0.263)$. Flatter.
In both cases the ranking is unchanged.
Exercise 2: top-K and top-P
The next-word distribution is a 0.40, b 0.25, c 0.15, d 0.10, e 0.06, f 0.04. (1) Which words does top-P keep with $P=0.8$, and what are their renormalized probabilities? (2) What does top-K with $K=2$ give?
(2) Keep {a, b}, total 0.65: $0.40/0.65\approx0.615$, $0.25/0.65\approx0.385$.
Exercise 3: beam search by hand
Run beam search with $m=3$ on the running example (Table 1, only nice, dog, car need expanding). Which three sequences survive step 2, and what's the output? Check your answer with Figure 5.
Quick quiz
10Further reading
- von Platen (2020). How to generate text: using different decoding methods for language generation with Transformers. Hugging Face blog. (A hands-on tour of greedy, beam, top-K, and top-P with code.)
- Holtzman, Buys, Du, Forbes & Choi (2020). The Curious Case of Neural Text Degeneration. ICLR. (Introduces top-P / nucleus sampling.)
- Fan, Lewis & Dauphin (2018). Hierarchical Neural Story Generation. ACL. (Popularized top-K sampling.)
- Wu et al. (2016). Google's Neural Machine Translation System. (Beam search with a length penalty.)
- Meister, Cotterell & Vieira (2020). If Beam Search is the Answer, What was the Question? EMNLP.