N-Gram Language Models

Read these sentences and decide how likely each one is to appear in ordinary English text:

The dog is barking at the stranger in the yard.
The sky colorful because painted an artist.
Plorp zix flanned the quibble through treemunk.

You can rank them instantly: the first is natural, the second uses real words in a broken order, and the third isn't English at all. A language model is a program that makes this judgment with numbers. It assigns a probability to every possible piece of text, so that natural text gets high probability and garbled text gets low probability.

This reading builds the simplest kind of language model, the n-gram language model, which estimates these probabilities by counting word sequences in a training corpus. We'll then see how to measure how good a language model is, using perplexity.

1What is a language model?

Let a piece of text be a sequence of words $x=[w_1,w_2,\dots,w_l]$, and let $\mathcal X$ be the set of all finite pieces of text. A language model learns a probability distribution over $\mathcal X$:

Every possible text, from “I love dogs.” to a full novel to random gibberish, gets a probability, and all of those probabilities add up to 1. Once we have such a distribution, we can use it in two ways:

• Score text. Compute $P(x)$ for a given text. P(The dog is barking at the stranger in the yard.) should be high; P(Cats upon they chairs sleeping their dreams fall.) should be low. Scoring is useful whenever a system must choose among candidate outputs, for example in speech recognition (recognize speech vs. wreck a nice beach) or machine translation.
• Generate text. Sample a new text from the distribution, $\tilde x\sim P(\mathcal X)$. This is what chatbots and text completion do.

2The chain rule: predicting one word at a time

There are infinitely many possible texts, so we can't simply store a probability for each one. Instead, we break the probability of a sequence into a product of simpler pieces using the chain rule of probability:

This is exact; no approximation has been made yet. It says that the probability of a sentence is the product of the probability of each word given all the words before it. For example,

A model built this way is called an auto-regressive language model. The whole problem has become one question asked over and over: given the context $w_1,\dots,w_{i-1}$, what is the probability of the next token $w_i$? This is next-token prediction, and it is exactly what modern large language models are trained to do too.

The challenge is estimating $P(w_i\mid w_1,\dots,w_{i-1})$. The context can be arbitrarily long, and most long contexts have never appeared anywhere in our training data, so we can't just look them up.

3The n-gram assumption

The n-gram model makes a simplifying assumption: the next word depends only on the previous $n-1$ words. Everything earlier is ignored.

Probabilities over such short contexts can be estimated by counting. Collect a training corpus, count how often each sequence of $n$ words (an n-gram) occurs, and divide:

In words: of all the times the context appeared, what fraction of the time was it followed by $w_i$? This ratio of counts is the maximum likelihood estimate: it is the choice of probabilities that makes the training corpus as likely as possible. The three most common cases have their own names:

The unigram model ignores context entirely: each word is drawn independently according to its overall frequency. It is the language-model version of a bag of words, so it assigns the same probability to a sentence and to any reordering of it. The bigram model looks one word back, and the trigram model two words back.

Sentence boundaries: <s> and </s>

Two special tokens make the model complete:

• <s> marks the start of a sentence. We pad each sentence with $n-1$ copies of it, so even the first word has a full context: $P(\textit{she}\mid\text{<s>})$ is the probability that a sentence starts with she. Since every sentence begins with <s>, it is never predicted and never counted as a word.
• </s> marks the end of a sentence, and it is predicted, just like a word. This lets the model decide when a sentence stops. Without it, the probabilities of all sentences of all lengths would not add up to 1.

4A worked example

Here is the tiny corpus used throughout the slides, together with two test sentences. The first test sentence appears in the training data; the second does not, although it is perfectly good English.

she likes to go hiking
he likes to go swimming
she likes to read books
she wants to read books

she likes to go hiking
he likes to read books

Use the calculator below to see each model's computation. Choose a model and a test sentence, or type your own sentence using words from the training corpus. Each factor shows the count of the n-gram over the count of its context.

The three models give these probabilities to the two test sentences:

Several lessons are packed into this table:

• Context helps a lot. The unigram probabilities are tiny because every word is treated as an independent draw from 11 possible tokens. Once the model can see the previous word, the next word becomes far more predictable.
• Unigrams ignore order. Try the “shuffled” sentence in Figure 3: the unigram model gives hiking go to likes she exactly the same probability as she likes to go hiking, while the bigram model gives it 0.
• Bigrams generalize. he likes to read books never appears in training, yet every one of its bigrams (he likes, likes to, to read, read books) does, so the bigram model gives it the same probability as the sentence it has seen.
• Trigrams lean toward memorization. The trigram model raises the probability of the seen sentence (0.125 → 0.167) but lowers the unseen one (0.125 → 0.083), because likes to was followed by read only once in training.

5Evaluating language models: perplexity

How do we decide which language model is better? The idea is to train on one corpus and test on a different, unseen one. If the training corpus is representative of good text, a good model should assign high probability to the test corpus: it should be “unsurprised” by real sentences it hasn't seen.

For a test corpus $\mathcal X$ with sentences $x_1,\dots,x_n$, we can measure this in several equivalent ways:

Each step fixes a practical problem. Products of many small probabilities underflow to 0 on a computer, so we take the log. Longer test corpora always have lower total log-likelihood, so we divide by the number of words $W$ to get an average per word. Finally, exponentiating the negative average turns it back into an interpretable number. Because $e^{-x}$ is decreasing, minimizing perplexity is the same as maximizing the probability of the test corpus.

Perplexity on the running example

Treat the two test sentences as a test corpus. Each has 5 words plus </s>, so $W=12$:

The bigram model wins. The trigram model is slightly worse, and the 4-gram model's perplexity is infinite: the 4-gram he likes to read never occurs in training, so the second test sentence gets probability 0, and $\ln 0=-\infty$. This is the central tension of n-gram models:

• Larger $n$ uses more context and can make sharper predictions.
• But the number of possible n-grams grows exponentially with $n$, so longer n-grams are seen less often, and most valid n-grams are never seen at all. Their estimated probability is 0, even though they are perfectly fine.

With a realistic corpus of millions of words, trigram and 4-gram models do outperform bigrams. But the zero-probability problem never fully goes away, because language always contains new combinations of words. Practical n-gram models therefore use smoothing techniques that move a little probability mass to unseen n-grams; those are beyond the scope of this reading.

Perplexity on real benchmarks

On the Penn Treebank, a standard benchmark of about one million words of news text, well-tuned n-gram models reach perplexities around 140. Neural language models brought this down steadily: LSTM-based models reached roughly 80 by 2014 and around 50 a few years later, and large pretrained Transformers went lower still (GPT-3 reported about 21 zero-shot in 2020). Comparing perplexities only makes sense on the same test set with the same vocabulary and tokenization, because those choices change what $W$ counts and which tokens must be predicted.

6Check your understanding

Exercise: a bigram model by hand

Train a bigram model (with <s> and </s>) on the corpus I am Sam / Sam I am / I do not like green eggs. Compute (1) $P(\textit{I}\mid\text{<s>})$, $P(\textit{am}\mid\textit{I})$, $P(\text{</s>}\mid\textit{Sam})$; (2) the probability of the sentence I am Sam; (3) its perplexity as a one-sentence test corpus.

Quick quiz

7Further reading

• Jurafsky & Martin, Speech and Language Processing (3rd ed. draft), Ch. 3, “N-gram Language Models.” The main reading for this topic.
• Shannon (1948). A Mathematical Theory of Communication. Bell System Technical Journal. (The first n-gram models of English, including generated samples.)
• Chen & Goodman (1998). An Empirical Study of Smoothing Techniques for Language Modeling. Harvard TR-10-98.
• Bengio, Ducharme, Vincent & Jauvin (2003). A Neural Probabilistic Language Model. JMLR.