CSCE 471 Natural Language Processing · Spring 2027

Tokenization

Kuan-Hao Huang · Course Website
Note: This page was created with the help of AI as a supplementary reading. It may contain errors or simplifications. The lecture slides are the primary reference for this course; please refer to them first.

Before a model can do anything with text, the text has to be cut into pieces, and each piece has to be mapped to an ID in a fixed vocabulary. Each piece then gets its own embedding vector. This first step is called tokenization, and its pieces are called tokens.

The obvious choice is to make every word a token. This reading explains why that choice runs into trouble, how an <UNK> token patches the problem, and how subword tokenization, in particular Byte-Pair Encoding (BPE), solves it more gracefully. BPE, or a close relative of it, is what nearly every modern language model uses.

After reading this, you should be able to explain the trade-offs in choosing a vocabulary size · describe how <UNK> handles rare and unseen words, and what it loses · run the BPE learning algorithm by hand · tokenize a new word by applying learned merges in order · name the main BPE variants used in practice.

1Words as tokens

The simplest tokenizer splits text on spaces and separates punctuation:

I like this movie so much.  →  Ilikethismoviesomuch.
Figure 1. Word-level tokenization: every word and every punctuation mark is one token.

Every distinct token needs its own embedding vector, so the vocabulary size $|V|$ determines how many embeddings we have to store and learn. How big should $|V|$ be? If we keep every word that appears in a large corpus, the vocabulary explodes. Word frequencies have a very long tail: a few words (the, of, and) appear constantly, while most distinct strings appear only once or twice. That tail contains typos (teh), random strings (zcvahu), symbols (#$^&*), rare names, and creative spellings (sooooo).

Keeping all of them is costly. The embedding table has $|V|\times d$ parameters, and a model that predicts the next word must compute a score for every vocabulary entry at every step. Worse, the embeddings of words seen only once or twice are barely trained, so they are of little use anyway. And no matter how large the vocabulary is, new text will always contain words that weren't in it.

2The unknown token

The standard fix is to keep only the most frequent words, for example every word that appears at least $k$ times, and replace everything else with a special unknown token, <UNK>. <UNK> gets its own embedding, which is trained on all the replaced words together:

I like this movie &*# so much.  →  Ilikethismovie<UNK>somuch.
I like this movie sooooo much.  →  Ilikethismovie<UNK>much.
Figure 2. Out-of-vocabulary words become <UNK>.

This keeps the vocabulary small and lets the model process any input. Try changing the frequency cutoff on a small corpus below:

Keep words that appear at least 2 times
Training corpus (vocabulary is built from this):
New sentence, tokenized with that vocabulary:
Figure 3 (interactive). A frequency cutoff trades vocabulary size against <UNK>s. Words shown faded in the corpus fall below the cutoff and would be trained as <UNK>.

But <UNK> throws information away. After tokenization, sooooo (very positive), &*# (noise), and Transformerify (a new verb) all look identical to the model. A human could guess what most of these mean, because words have internal structure:

Word-level tokenization can't use any of this structure. happiest and happier are as unrelated to the tokenizer as happiest and giraffe.

3Subword tokenization

Subword tokenization uses pieces of words as the basic units: happ + iest, driv + ing, grate + ful. Frequent words can still be single tokens, while rare words are split into frequent pieces. This gets the best of both worlds:

The remaining question is how to choose the pieces. We could hand-write a list of English prefixes and suffixes, but that wouldn't work for other languages, typos, code, or names. Instead, we let the data decide.

4Byte-Pair Encoding

Byte-Pair Encoding (BPE) was originally a data-compression algorithm (Gage, 1994): repeatedly replace the most common pair of adjacent bytes with a single new symbol. Sennrich, Haddow & Birch (2016) adapted it to build subword vocabularies. The idea is the same: if two symbols often appear next to each other, merge them into one.

Learning the merges

  1. Start by splitting every word in the training corpus into characters, and add a special end-of-word symbol </w> to the end of each word. The initial vocabulary is the set of all characters, plus </w>.
  2. Count every pair of adjacent symbols across the corpus (weighted by how often each word occurs), and find the most frequent pair $(x, y)$.
  3. Merge: add the new symbol $xy$ to the vocabulary, record the merge rule $x + y \Rightarrow xy$, and replace every occurrence of the pair in the corpus with $xy$.
  4. Repeat steps 2–3 until the vocabulary reaches the desired size.

The end-of-word symbol matters: it lets BPE tell a suffix from the middle of a word. est</w> (as in newest) is a different symbol from est (as in estimate). Step through the algorithm on the example from the slides:

Corpus:
Corpus
Most frequent pairs
Vocabulary
Merges (in order)
Figure 4 (interactive). The BPE learning algorithm, one merge per step. The highlighted pair is the most frequent one, which gets merged next. When several pairs tie, this demo picks the one that appears first in the corpus.
Ties. In the first step, three pairs tie at 9 occurrences: e s, s t, and t </w>. The algorithm doesn't say which to pick; implementations just use a fixed rule (for example, the first pair encountered). Different tie-breaking can produce different merge orders, but the final vocabularies are usually very similar.

Tokenizing new words

Once training is done, the output is the ordered list of merge rules. To tokenize a new word, split it into characters plus </w>, then apply the merges in the order they were learned, each to every place it matches. Characters that never appeared in training have no entry in the vocabulary and become <UNK>. Try it with the merges learned in Figure 4:

Word:
Figure 5 (interactive). Applying learned merges to a word, one rule at a time. Rules that don't match are skipped. Use the controls in Figure 4 to change how many merges have been learned.

After the nine merges shown in the slides (step 9 in Figure 4), lowest becomes low + est</w>: a word never seen in training, built from two meaningful pieces. powest becomes <UNK> o w est</w>, because p never appeared in the training corpus. In practice, the training corpus is huge, so every ordinary character is in the base vocabulary, and only truly exotic characters would become <UNK>.

5BPE in practice

Choosing the vocabulary size

The number of merges is a hyperparameter. With zero merges, every word is a sequence of characters: the vocabulary is tiny, but sequences are long and each token carries little meaning. With very many merges, almost every word becomes a single token, and we are back to the problems of word-level tokenization. Real systems sit in between: GPT-2's tokenizer has 50,257 tokens, and BERT's has about 30,000. With a vocabulary of that size, common words are single tokens and rare words split into a few pieces.

Variants you will meet

Why tokenization matters downstream

Tokenization decides what the model actually sees, and its quirks show up in surprising places. The same word can be split differently depending on capitalization or a leading space (Hello vs. ␣hello). Numbers are often split into arbitrary chunks, which makes arithmetic harder. And because tokenizers are usually trained mostly on English, text in other languages tends to need more tokens for the same content, which makes it slower and more expensive to process.

6Check your understanding

Exercise: BPE by hand

A corpus contains hug ×10, pug ×5, pun ×12, bun ×4, and hugs ×5. Using an end-of-word symbol </w>, find the first three merges. Break ties by picking the pair that appears first when reading the corpus in this order.

Merge 1. Pair counts: u g = 10 + 5 + 5 = 20; p u = 5 + 12 = 17; u n = n </w> = 16; h u = g </w> = 15; the rest are smaller. Merge u + g ⇒ ug.
Merge 2. Now u n = 16 and n </w> = 16 tie for the top (h ug and ug </w> are 15). u n appears first (in pun), so merge u + n ⇒ un.
Merge 3. un </w> = 12 + 4 = 16 is now the most frequent pair. Merge un + </w> ⇒ un</w>.
You can check this with the “hug / pug / pun / bun” corpus in Figure 4.

Quick quiz

7Further reading