Learning-Based Word Vectors: Evaluation

Suppose you have trained a set of word vectors: one dense vector of, say, 300 numbers for every word in a vocabulary. Are they any good? Training gives us a loss value, but a low loss only says the vectors fit their training objective. It doesn't say whether great and excellent really ended up close together, or whether the vectors will help an actual application.

This reading covers the standard ways to answer that question: comparing vector similarity with human judgments, solving word analogies, and measuring performance on downstream tasks. It ends with a warning: the same tests that show what word vectors capture also reveal that they absorb social biases from their training text.

After reading this, you should be able to distinguish intrinsic from extrinsic evaluation · evaluate word similarity with Pearson and Spearman correlation · solve word analogies with vector arithmetic, and explain why the input words are excluded · use averaged word vectors as features for classification · explain how gender bias shows up in word vectors and how projection-based debiasing works.

1Two ways to evaluate

There are two broad families of evaluation:

Intrinsic evaluation tests the vectors directly on a small, specific task that probes a linguistic or semantic property: Are similar words close together? Do the vector differences encode consistent relationships?
Extrinsic evaluation plugs the vectors into a real application, such as sentiment classification, and measures how well that application performs.

The two answer different questions. Intrinsic tests are quick and tell us what the vectors capture. Extrinsic tests tell us whether the vectors are actually useful. We'll look at each in turn, then compare them in Section 5.

2Intrinsic evaluation: word similarity

The most direct test asks: does the cosine similarity between two word vectors agree with how similar people think the two words are? To find out, we need a dataset of word pairs with human similarity ratings. A classic example is WordSim-353 (Finkelstein et al., 2002): 353 word pairs, each rated on a scale from 0 (unrelated) to 10 (identical in meaning) by several people, with the ratings averaged.

Word 1	Word 2	Human (mean)
tiger	cat	7.35
tiger	tiger	10.00
book	paper	7.46
computer	internet	7.58
plane	car	5.77
professor	doctor	6.62
stock	phone	1.62
stock	CD	1.31
stock	jaguar	0.92

For each pair, the model's score is the cosine similarity $$\cos(\mathbf u_i,\mathbf u_j)=\frac{\mathbf u_i\cdot\mathbf u_j}{\|\mathbf u_i\|\,\|\mathbf u_j\|}$$ We then measure how well the model's scores agree with the human scores across all pairs.

Figure 1. A few pairs from WordSim-353. The model never sees the human scores; we only compare the two lists afterward.

Notice that the model's scores and the human scores live on completely different scales: cosine similarity lies between −1 and 1, while human ratings run from 0 to 10. So we don't compare the numbers directly. We ask whether the two lists go up and down together. That is what a correlation coefficient measures.

Pearson's correlation

Given paired values $(x_i, y_i)$ for $i=1,\dots,n$ (here, human score and model score for pair $i$), Pearson's correlation coefficient is

$$r=\frac{\sum_i (x_i-\bar x)(y_i-\bar y)}{\sqrt{\sum_i (x_i-\bar x)^2}\;\sqrt{\sum_i (y_i-\bar y)^2}}$$

where $\bar x$ and $\bar y$ are the means. It ranges from −1 to 1: $r=1$ means the points lie exactly on an upward-sloping line, $r=0$ means no linear relationship, and $r=-1$ means a perfect downward line. The key word is linear: Pearson measures how well a straight line fits.

Spearman's rank correlation

For word similarity, we don't really need the relationship to be a straight line. If the model always ranks more-similar pairs above less-similar ones, it has done its job, even if its cosine scores are bunched together or stretched out. Spearman's rank correlation $\rho$ captures exactly this: replace each list by its ranks, then compute Pearson's correlation on the ranks. When there are no ties, this simplifies to

$$\rho=1-\frac{6\sum_i d_i^2}{n(n^2-1)},\qquad d_i=\text{rank}(x_i)-\text{rank}(y_i)$$

Spearman measures whether the relationship is monotonic (always increasing), whether or not it is linear. It is the standard metric for word-similarity benchmarks. Try the example from the slides below, and edit the model's scores to see how both coefficients respond:

Figure 2 (interactive). Pearson vs. Spearman on five word pairs. Rank 1 is the most similar pair. The model scores are editable.

Why prefer ranks? The next figure shows three typical situations. When the relationship is monotonic but curved, Pearson is pulled below 1 while Spearman still reports perfect agreement. A single wild outlier can drag Pearson down a lot, but it moves only one rank, so Spearman barely changes.

Figure 3 (interactive). Pearson's $r$ and Spearman's $\rho$ on three synthetic datasets.

What the results look like

Here is a comparison from the GloVe paper (Pennington et al., 2014). Each number is a Spearman correlation (×100) between cosine similarity and human ratings on one benchmark.

Model	Corpus size	WS353	MC	RG	SCWS	RW
SVD	6B	35.3	35.1	42.5	38.3	25.6
SVD-S	6B	56.5	71.5	71.0	53.6	34.7
SVD-L	6B	65.7	72.7	75.1	56.5	37.0
CBOW†	6B	57.2	65.6	68.2	57.0	32.5
Skip-gram†	6B	62.8	65.2	69.7	58.1	37.2
GloVe	6B	65.8	72.7	77.8	53.9	38.1
SVD-L	42B	74.0	76.4	74.1	58.3	39.9
GloVe	42B	75.9	83.6	82.9	59.6	47.8
CBOW*	100B	68.4	79.6	75.4	59.4	45.5

Figure 4. Word-similarity results (Spearman ρ × 100) from Pennington et al. (2014). SVD factorizes the raw co-occurrence count matrix $X$; SVD-S uses $\sqrt{X}$ and SVD-L uses $\log(1+X)$. † marks models the GloVe authors trained with the word2vec tool. * marks Google's publicly released vectors trained on 100B words of Google News. Benchmarks: WordSim-353, Miller–Charles (MC), Rubenstein–Goodenough (RG), Stanford Contextual Word Similarity (SCWS), and Rare Words (RW).

Three things stand out. First, how the counts are transformed matters a great deal: plain SVD on raw counts is far worse than SVD on square-rooted or log-scaled counts. Second, more training text helps every method (compare 6B with 42B). Third, the Rare Words benchmark is the hardest for everyone, because rare words have few occurrences to learn from.

3Intrinsic evaluation: word analogy

Word similarity checks that related words are close. Word analogies check something more surprising: that relationships between words are encoded as consistent directions in the vector space. An analogy question has the form

man : woman ≈ king : ?

If the vector from man to woman represents “change the gender,” adding that same vector to king should land near queen. So we answer the question with vector arithmetic. For a general analogy $a : b \approx c : \;?$,

$$\hat w=\arg\max_{w\in V,\;w\notin\{a,b,c\}}\;\cos\!\left(\mathbf u_w,\;\mathbf u_b-\mathbf u_a+\mathbf u_c\right)$$

The restriction $w\notin\{a,b,c\}$ is important in practice. The target $\mathbf u_b-\mathbf u_a+\mathbf u_c$ is just $\mathbf u_c$ shifted by a small offset, so the closest word is very often $c$ itself (here, king). Without the restriction, many analogy questions would be “answered” by repeating one of the input words. The figure below lets you see this on a small set of toy vectors.

: ≈ : ? exclude a, b, c

Figure 5 (interactive). Solving analogies by vector arithmetic on 4-dimensional toy vectors. The plot shows two of the four dimensions (roughly “gender” and “royalty”); the cosine scores use all four. Because cosine compares directions in all four dimensions, the winning word is not always the point nearest the target in this 2-D picture. The dashed arrow is the offset $\mathbf u_b-\mathbf u_a$, copied to start at $\mathbf u_c$. Try man : woman ≈ king with the checkbox off.

Analogy benchmarks such as the Google analogy dataset (Mikolov et al., 2013) contain about 19,500 questions in two groups. Semantic questions cover relations like country–capital (Paris : France ≈ London : England) and family roles. Syntactic questions cover grammatical relations like comparative–superlative (bad : worst ≈ cool : coolest) and verb tense (walking : walked ≈ swimming : swam). The score is simply the fraction of questions answered exactly right.

Visualizations of trained vectors show why this works. Projected down to two dimensions, pairs such as man–woman, king–queen, and uncle–aunt are connected by nearly parallel offsets, and so are slow–slower–slowest and strong–stronger–strongest. See the GloVe project page and the TensorFlow word2vec tutorial for plots of real vectors.

A caveat. Analogies make memorable demos, but the parallelograms in real vectors are far from perfect. Much of the “correct” behavior comes from the target simply being close to $\mathbf u_c$, which is why excluding the input words changes results so much (Linzen, 2016). Treat analogy accuracy as one signal among several, not as proof that the vectors “understand” relations.

4Extrinsic evaluation: downstream performance

Ultimately, we want word vectors because they make applications better. Extrinsic evaluation measures that directly. Take a task such as sentiment classification, where the input is a text $x$ and the output is a label $y$. A classifier needs a fixed-length feature vector for the text, and word vectors give us an easy way to build one: look up the vector of every word in the text and average them.

Figure 6. Word vectors as features. The averaged vector $\mathbf w_{\text{avg}}$ feeds a classifier, which is trained on labeled examples. To compare two sets of word vectors, keep everything else fixed and compare the classifier's accuracy.

For a text with words $w_1,\dots,w_n$, the feature vector is

$$\mathbf w_{\text{avg}}=\frac1n\sum_{i=1}^{n}\mathbf w_{w_i}$$

This is a word-vector version of the bag-of-words representation: like bag-of-words, it ignores word order (Bob likes Alice and Alice likes Bob get the same vector). Unlike bag-of-words, similar words contribute similar features, so a classifier that learned “great is positive” automatically gives some credit to “excellent.”

The evaluation protocol is simple: train the same classifier, on the same training data, once with each set of word vectors, and compare accuracy on the same test set. Common tasks include sentiment analysis, named entity recognition, and question classification.

5Intrinsic vs. extrinsic

Intrinsic evaluation

Tests a specific property (similarity, analogy)
Fast: no extra model training
Easy to interpret what the vectors capture
But: a high score does not guarantee better downstream performance

Extrinsic evaluation

Tests performance on a real task
Measures what we actually care about
Slow: requires training a downstream model for each set of vectors
But: results depend on the downstream model and task

The warning in the intrinsic column is not hypothetical. Chiu, Korhonen & Pyysalo (2016) found that word-similarity scores often fail to predict which vectors do best on downstream tasks, and Faruqui et al. (2016) pointed out further problems with similarity benchmarks, such as subjective ratings and the blurry line between “similar” (cup–mug) and merely “related” (cup–coffee). In practice, people use intrinsic tests to quickly compare and debug vectors, and extrinsic tests to decide which vectors to actually use.

6Word vectors can encode bias

Word vectors learn from how people actually write, and human-written text reflects social stereotypes. The analogy test makes this visible. Bolukbasi et al. (2016) ran analogy queries of the form he : she ≈ x : ? on word2vec vectors trained on Google News and found answers like these:

Query	Answer
he : she ≈ brother : ?	sister	definitional, fine
he : she ≈ beer : ?	cocktail	stereotype
he : she ≈ physician : ?	registered nurse	stereotype
he : she ≈ professor : ?	associate professor	stereotype

Figure 7. Analogy answers reported by Bolukbasi et al. (2016), “Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings.”

The first answer is exactly what we'd want, since brother and sister differ by definition in gender. The others are a problem: nothing about physician is inherently male, yet the vectors treat it that way. A system that uses these vectors, say to rank résumés or complete sentences, can quietly inherit the stereotype.

Finding the gender direction

If gender is encoded as a direction in the space, we should be able to find it. Take pairs that differ only in gender, such as $(\textit{he},\textit{she})$, $(\textit{father},\textit{mother})$, $(\textit{king},\textit{queen})$, and compute their difference vectors $\mathbf u_{\text{he}}-\mathbf u_{\text{she}}$, and so on. Running principal component analysis (PCA) on these differences, Bolukbasi et al. found that a single component explains most of the variance; for differences between random pairs (dog–cat, house–building), the variance is spread over many components. That top component $\mathbf g$ (a unit vector) is the gender direction.

Projecting words onto $\mathbf g$ separates them into two kinds:

Definitional words, whose meaning includes gender: mother, king, sister. Having a gender component is correct for these.
Gender-neutral words that nonetheless sit far toward one side: homemaker, nurse, genius, architect. Their gender component is learned stereotype, not meaning.

Debiasing by projection

The “hard debiasing” method of Bolukbasi et al. removes the gender component from every gender-neutral word $w$:

$$\mathbf u_w\;\leftarrow\;\mathbf u_w-(\mathbf u_w\cdot\mathbf g)\,\mathbf g,\qquad\text{then renormalize to unit length}$$

After this neutralize step, every neutral word is exactly equidistant from he and she. A second equalize step adjusts definitional pairs such as grandmother–grandfather so that they are symmetric around the neutral words. Try the neutralize step below:

Figure 8 (interactive). Neutralizing gender-neutral words (toy 2-D vectors). The horizontal axis is the gender direction $\mathbf g$. Definitional words keep their position; gender-neutral words are projected onto the vertical axis. The table lists each neutral word's gender score $\mathbf u_w\cdot\mathbf g$ (unnormalized here, for readability).

Does it work? Only partly. Gonen & Goldberg (2019), in “Lipstick on a Pig,” showed that after hard debiasing, the gender component along $\mathbf g$ is gone, but the bias is still encoded in how words cluster: stereotypically female words remain close to each other and can still be recovered with high accuracy. Removing one direction hides the bias rather than eliminating it. Analogy-based demonstrations of bias also have to be read carefully. Because the analogy test excludes the input words, a query like he : she ≈ physician : ? is not allowed to answer physician, even when that is the closest word (Nissim et al., 2020).

7Check your understanding

Exercise: Spearman's ρ by hand

Five word pairs have human ratings $[2,\,7,\,5,\,9,\,4]$, and a model gives them cosine similarities $[0.31,\,0.62,\,0.70,\,0.85,\,0.20]$. Compute Spearman's $\rho$. (Rank 1 = highest.)

Human ranks: $[5,\,2,\,3,\,1,\,4]$. Model ranks: $[4,\,3,\,2,\,1,\,5]$.
Differences $d_i$: $[1,\,-1,\,1,\,0,\,-1]$, so $\sum d_i^2=4$.
$\rho=1-\dfrac{6\cdot4}{5\,(25-1)}=1-\dfrac{24}{120}=0.8$.
The model swaps two pairs of neighbors in the ranking, but otherwise agrees with the humans, so $\rho$ is high.

Quick quiz

8Further reading

Jurafsky & Martin, Speech and Language Processing (3rd ed. draft), Ch. 6, “Vector Semantics and Embeddings,” Sections 6.10–6.12.
Finkelstein et al. (2002). Placing Search in Context: The Concept Revisited. ACM TOIS. (WordSim-353.)
Mikolov, Yih & Zweig (2013). Linguistic Regularities in Continuous Space Word Representations. NAACL.
Pennington, Socher & Manning (2014). GloVe: Global Vectors for Word Representation. EMNLP.
Linzen (2016). Issues in Evaluating Semantic Spaces Using Word Analogies. RepEval Workshop.
Chiu, Korhonen & Pyysalo (2016). Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance. RepEval Workshop.
Faruqui, Tsvetkov, Rastogi & Dyer (2016). Problems With Evaluation of Word Embeddings Using Word Similarity Tasks. RepEval Workshop.
Bolukbasi, Chang, Zou, Saligrama & Kalai (2016). Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. NeurIPS.
Gonen & Goldberg (2019). Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them. NAACL.
Nissim, van Noord & van der Goot (2020). Fair is Better than Sensational: Man is to Doctor as Woman is to Doctor. Computational Linguistics.