Counting-Based Word Vectors

How can a computer tell that great and excellent mean nearly the same thing, while great and giraffe do not? If every word is treated as an unrelated symbol, it can't. A model that learned “this movie was great” is positive would have no idea that “this movie was excellent” should be positive too.

This reading is about word vectors: giving every word its own vector so that words with similar meanings get similar vectors. We'll build those vectors with nothing but counting and a bit of linear algebra, following four steps: count which words appear near each other, measure similarity with cosine, reweight the counts with PPMI, and compress them into short dense vectors with SVD.

1Why can't words just be symbols?

The simplest way to turn a word into a vector is a one-hot vector. Number the words in the vocabulary $\mathcal V$ from 1 to $|\mathcal V|$; word number $i$ becomes a vector of length $|\mathcal V|$ with a 1 in position $i$ and 0 everywhere else.

Imagine a search engine. A user types good restaurant, and we'd like to also return pages that say great restaurant. With one-hot vectors, the dot product of good and great is 0, exactly the same as the dot product of good and bad, or good and giraffe. Every pair of different words is orthogonal. One-hot vectors don't just encode similarity badly; they have no way to encode it.

A natural fix is to borrow knowledge from people. WordNet is a large hand-built thesaurus that lists synonyms, antonyms, and “is-a” relations (hypernyms) for English words. You can query it from Python:

from nltk.corpus import wordnet as wn
for s in wn.synsets("bad"):
    print(s.pos(), [l.name() for l in s.lemmas()])

# n ['bad', 'badness']
# s ['bad', 'spoiled', 'spoilt']
# a ['regretful', 'sorry', 'bad']
# s ['bad', 'risky', 'high-risk', 'speculative'] ...

If we add features for synonym sets, good and great now share some dimensions and finally look more similar than good and bad. But look at the output above: WordNet says sorry is a synonym of bad, which is true only in phrases like “I feel bad about it.” Resources like this miss nuance, reflect their builders' judgments, never keep up with new words (rizz, ghosting, LLM), and take years of expert work to build, which is why most of the world's languages don't have one. We'd rather have a method that learns word meaning from text itself.

2You shall know a word by the company it keeps

Try a small experiment. Suppose you've never seen the word ongchoi, and you come across these sentences:

This is the core idea behind almost all of modern NLP, and it has a name: the distributional hypothesis. Words that appear in similar contexts tend to have similar meanings. The linguist J.R. Firth put it memorably:

You shall know a word by the company it keeps.J.R. Firth, 1957

Here is the same idea as a table. Each column is a sentence with a blank, and each row records whether a word fits the blank:

That last sentence is the plan for the rest of this reading: represent each word by a row of context counts, then clean those counts up until similar rows really do mean similar words.

3Counting contexts

What exactly counts as a “context”? There are two classic answers.

In a term–document matrix, the context is the whole document. Each row is a word, each column is a document, and each cell counts how often the word appears in that document. This is the matrix behind classic information retrieval. In the matrix below, fool and wit have similar rows because both show up mostly in Shakespeare's comedies.

A document is a pretty big context, though. For word meaning, we usually want something more local. In a word–word co-occurrence matrix, the context of a word is the set of words within a window of $\pm k$ positions around it. We slide through the corpus, and every time target word $w$ appears within $k$ words of context word $c$, we add 1 to cell $(w, c)$. Try it below: click any word to see its window, and move the slider to change $k$.

After processing a real corpus, the matrix looks something like this. Each row is now a vector for one word, with one dimension per vocabulary word:

How big should the window be?

The window size is a real design decision, because it changes what kind of similarity the vectors capture. With a small window ($\pm1$ to $\pm3$), a word's contexts are mostly its grammatical neighbors, so words end up similar when they can substitute for each other. With a large window ($\pm5$ to $\pm10$), contexts start to include the whole topic of the passage, so words end up similar when they belong to the same topic. Levy and Goldberg (2014) show this nicely with the nearest neighbors of Hogwarts:

Neither answer is wrong. The context you choose defines the similarity you get.

4Comparing word vectors

Once each word is a vector, we need a way to measure how similar two vectors are. The raw dot product $\mathbf u\cdot\mathbf v$ is a tempting choice, but it has a bias: frequent words have more co-occurrences, so their vectors are longer and their dot products are large with everything. The standard fix is to divide by the vectors' lengths. This gives the cosine similarity, the cosine of the angle between the vectors:

Cosine ranges from $-1$ to $1$, and since counts are never negative, it stays between 0 and 1 for count vectors. Let's compute one by hand using the rows from Figure 6. With $\text{apple}=[0,0,8,0,2]$ and $\text{bread}=[0,0,9,0,1]$:

Try it yourself below. Edit either vector or pick a preset, and follow each step of the formula. The slider multiplies the second vector by a constant. Watch the dot product grow while the cosine stays exactly the same: cosine only cares about direction, not length.

So far, so good. Unfortunately, this tidy picture depends on the fact that we hid some columns.

5From raw counts to PPMI

The problem: frequent words dominate

Real text is full of function words like the, it, of, and and, which appear next to nearly everything. Add just two of them back to the matrix:

The cosine is now dominated by the two huge columns, and apple looks more similar to digital than to bread. The issue is that raw counts measure how frequent a context word is, not how informative it is. Seeing apple next to the tells us nothing; seeing it next to eat tells us a lot. We need a weighting that asks a sharper question: is this co-occurrence more common than we'd expect by chance?

Pointwise mutual information

That question is exactly what pointwise mutual information (PMI) measures. For two events $x$ and $y$:

The numerator is how often $x$ and $y$ actually occur together. The denominator is how often they would occur together if they were independent. If the two are equal, the ratio is 1 and PMI is 0. PMI is positive when the pair co-occurs more often than chance, and negative when it co-occurs less often.

Here's a concrete example. Suppose our corpus has $N=10^6$ (word, context) pairs, where apple appears 1,000 times, eat 5,000 times, and the 60,000 times. apple co-occurs with eat 100 times and with the 60 times. Then

Raw counts would treat 60 and 100 as roughly comparable. PMI doesn't: 60 is exactly what chance predicts for a word as common as the, so that pair gets zero weight, while apple appears with eat 20 times more often than chance, a strong signal.

PMI always compares two numbers: how often a pair was observed together, and how often it would be expected together by chance, $\text{count}(x)\cdot\text{count}(y)/N$. Plug in your own counts, or start from one of the presets:

Positive PMI

In practice we use positive PMI (PPMI), which replaces negative values with 0:

Why throw away negative values? A negative PMI claims that two words co-occur less than chance. For two words that each have probability around $10^{-6}$, you'd need far more than $10^{12}$ word pairs to reliably tell “less than chance” apart from “just didn't happen to see it.” Negative estimates are mostly noise. A nice side effect is that zeros stay zero, so the matrix stays sparse.

All of the probabilities come straight from the count matrix. If $f_{ij}$ is the count in row $i$ and column $j$, and $N=\sum_{i,j}f_{ij}$ is the total, then

In words, each PPMI entry needs only the cell, its row total, its column total, and the grand total. Use the buttons below to switch between raw counts and PPMI on a slightly larger toy matrix and watch what happens to the similarities:

A weak spot: rare words

PMI has one well-known flaw: it overrates rare events. Suppose a word and a context each appear exactly once in the corpus, and that one time they appear together. Then

With $N=10^6$, that's about 19.9, much larger than the 4.32 we computed for apple/eat, based on a single observation. A typo or an obscure name can end up with the strongest association in the whole matrix. Common fixes (Levy, Goldberg & Dagan, 2015) include:

• Context distribution smoothing: raise the context counts to a power $\alpha$ (usually $0.75$) before turning them into probabilities, $P_\alpha(c)=\text{count}(c)^\alpha/\sum_{c'}\text{count}(c')^\alpha$. This gives rare contexts a little more probability, which shrinks their PMI.
• Add-$k$ smoothing: add a small constant $k$ to every count before computing probabilities.
• Frequency cutoffs: simply drop words and contexts seen fewer than some number of times.

To see context smoothing at work, take one row of a toy matrix with $N=85{,}905$ (word, context) pairs. The word pie appears 1,000 times. Its contexts range from the very common the to quince, which appears only 5 times in the whole corpus and only once next to pie. Slide $\alpha$ from 1 (no smoothing) down to 0.5:

6From sparse to dense: SVD

PPMI vectors are a big improvement, but they are still long (one dimension per vocabulary word, often tens of thousands) and sparse (mostly zeros). We would much prefer short, dense vectors of maybe 50 to 300 real numbers. Short vectors are cheaper to use as features in a model. Dense vectors also tend to generalize better, and in practice they simply work better.

There's also a deeper reason. Suppose word $w_1$ often appears near car, and word $w_2$ often appears near automobile. Intuitively $w_1$ and $w_2$ should be similar. But in a sparse vector, car and automobile are two separate dimensions, so they contribute nothing to $\cos(w_1,w_2)$. Sparse vectors can't capture this kind of higher-order co-occurrence. What we want is a representation that notices car and automobile behave alike and merges them into a single “vehicle” direction.

That is what the singular value decomposition (SVD) does. Any matrix $X$ can be factored as $X=U\Sigma V^\top$, where $\Sigma$ is a diagonal matrix of non-negative singular values sorted from largest to smallest. The largest singular values correspond to the most important patterns in the data. If we keep only the top $k$ and discard the rest, we get the best possible rank-$k$ approximation of $X$:

Each row of $U_k\Sigma_k$ is a dense $k$-dimensional vector for one word. Because SVD has to explain as much of the matrix as possible with only $k$ directions, contexts that behave similarly (car and automobile) get folded into the same directions. When this technique is applied to a term–document matrix, it's known as Latent Semantic Analysis (LSA) (Deerwester et al., 1990), one of the earliest successful word embedding methods.

The figure below compresses each 7-dimensional row of our toy matrix into just 2 numbers so we can plot it. Compare SVD on raw counts with SVD on PPMI:

How many dimensions should we keep? The figure below rebuilds the PPMI matrix from only its top $k$ singular values, $X_k=U_k\Sigma_kV_k^\top$. With small $k$, the vectors are short but lose detail. As $k$ grows, $X_k$ gets closer to $X$, but we compress less. Real systems use $k$ between about 50 and 300 for vocabularies of many thousands of words.

7Putting it all together

The complete recipe has four steps: count co-occurrences in a $\pm k$ window, weight them with PPMI, reduce the dimension with truncated SVD, and use the resulting rows as word vectors. The whole thing fits in about twenty lines of NumPy:

import numpy as np

def cooccurrence(corpus, vocab, window=2):
    idx = {w: i for i, w in enumerate(vocab)}
    M = np.zeros((len(vocab), len(vocab)))
    for sent in corpus:                                # corpus: list of token lists
        for i, w in enumerate(sent):
            lo, hi = max(0, i - window), min(len(sent), i + window + 1)
            for j in range(lo, hi):
                if j != i and w in idx and sent[j] in idx:
                    M[idx[w], idx[sent[j]]] += 1
    return M

def ppmi(M):
    N = M.sum()
    expected = np.outer(M.sum(1), M.sum(0)) / N             # = N * P(w) * P(c)
    with np.errstate(divide="ignore", invalid="ignore"):
        pmi = np.log2(M / expected)                          # = log2 P(w,c) / (P(w) P(c))
    return np.nan_to_num(np.maximum(pmi, 0))             # -inf and nan become 0

X = ppmi(cooccurrence(corpus, vocab, window=4))
U, S, Vt = np.linalg.svd(X)
W = U[:, :100] * S[:100]                                   # |V| x 100 word vectors

This approach has real strengths. It's simple and transparent, involves no training loop, and makes direct use of statistics gathered from the whole corpus. It also has clear limitations. SVD becomes expensive for very large vocabularies, and adding a new word means recomputing everything. There are many hand-tuned choices (window size, weighting, $\alpha$, $k$). And every word gets exactly one vector, so bank as in “river bank” and bank as in “savings bank” share a single representation.

8Check your understanding

Exercise: PPMI by hand

Compute the PPMI matrix (using $\log_2$) for the following counts. The total is $N=10$.

Quick quiz

9Further reading

• Jurafsky & Martin, Speech and Language Processing (3rd ed. draft), Ch. 6, “Vector Semantics and Embeddings.” The main reading for this topic.
• Turney & Pantel (2010). From Frequency to Meaning: Vector Space Models of Semantics. JAIR.
• Deerwester et al. (1990). Indexing by Latent Semantic Analysis. JASIS.
• Levy & Goldberg (2014). Neural Word Embedding as Implicit Matrix Factorization. NeurIPS.
• Levy, Goldberg & Dagan (2015). Improving Distributional Similarity with Lessons Learned from Word Embeddings. TACL.