Consider the two sentences “the movie is good” and “the movie is not good”. They share four of five words, yet they mean opposite things. A text classifier that only knows which words appear will struggle here. The deciding evidence is a short phrase, not good, where the order and adjacency of words matter.
A convolutional neural network (CNN) is built to catch exactly this kind of evidence. It slides small learned filters over the sentence, each one acting like a detector for a useful local pattern, roughly a soft, learned n-gram. It then keeps the strongest detection anywhere in the sentence. This reading builds a CNN for text classification step by step, using a short running example.
1Text classification with neural networks
In text classification, the input is a piece of text $x$ and the output is a label $y$ from a fixed set of $C$ classes: positive vs. negative sentiment, spam vs. not spam, sports vs. politics vs. business. Every approach has two parts:
- a representation that turns the text into a fixed-length vector $\mathbf x$, and
- a classifier (for example a feedforward neural network ending in a softmax) that maps $\mathbf x$ to a probability for each class, $P(y=c\mid\mathbf x)$.
The classifier is trained by minimizing the cross-entropy loss. If the true label is written as a one-hot vector $y$ (with $y_c=1$ for the correct class and 0 elsewhere), the loss is
$$\mathcal L_{CE}(y,\tilde y)=-\sum_{c=1}^{C}y_c\log P(y=c\mid\mathbf x),$$which simply equals $-\log$ of the probability assigned to the correct class. The hard part is the first step: how do we turn a sentence of any length into a good fixed-length vector? We'll represent each word by a $d$-dimensional embedding and look at what to do with a sequence of them.
2Two simple approaches and their problems
Here is the running example: Alice treats Bob well, with toy 4-dimensional word embeddings. Each column is one word, each row is one embedding dimension.
Averaging takes the mean of the word vectors, dimension by dimension. It gives a fixed-size vector for any sentence length, but it throws away word order: Alice treats Bob well and Bob treats Alice well get exactly the same vector, even though who treats whom has changed. So do good, not bad and bad, not good.
Concatenation stacks the word vectors into one long vector, so order is kept. But the length of the result depends on the sentence: 4 words give a 16-dimensional vector, 7 words give 28. A feedforward classifier has a fixed number of inputs, so it can't handle sentences of different lengths. It also learns separate weights for “word in position 1”, “word in position 2”, and so on, so a pattern like not good learned at one position doesn't transfer to another.
We want something in between: a fixed-size representation that still knows about word order, and that recognizes a useful pattern wherever it occurs.
3Capturing local order with n-grams
Full word order is hard to use, but local order is easy. Break the sentence into short overlapping windows, or n-grams:
| n | n-grams of Bob likes Alice very much |
|---|---|
| 1 (unigram) | Bob · likes · Alice · very · much |
| 2 (bigram) | Bob likes · likes Alice · Alice very · very much |
| 3 (trigram) | Bob likes Alice · likes Alice very · Alice very much |
| 4 | Bob likes Alice very · likes Alice very much |
Each bigram or trigram fixes the order of a few words: Bob likes Alice and Alice likes Bob are different trigrams. Because neighboring windows overlap, the windows together pin down much of the global order. Many decisions also depend mainly on short phrases: not good, highly recommend, waste of time. A CNN learns to score every window of the sentence with a small set of learned detectors.
4The convolution filter
Stack the embeddings of an $n$-word sentence as the columns of a $d\times n$ matrix, as in Figure 1. A filter (also called a kernel) of size $h$ is a learnable $d\times h$ weight matrix $\mathbf W$. It covers $h$ consecutive words and all $d$ dimensions at once.
To apply the filter at position $i$, line it up with the window of words $i,\dots,i+h-1$, multiply each weight by the embedding value under it, and add everything up:
$$v_i=f\!\left(\sum_{j=1}^{d}\sum_{k=1}^{h}w_{j,k}\,x_{j,\,i+k-1}+b\right)$$Here $x_{j,t}$ is dimension $j$ of word $t$'s embedding, $b$ is a learnable bias, and $f$ is a nonlinearity such as ReLU, $f(z)=\max(0,z)$. Equivalently, flatten the window and the filter into two $dh$-dimensional vectors and take their dot product. The result $v_i$ is one number: how strongly the window matches the filter's pattern.
Then slide the filter one word to the right and repeat. With $n$ words and filter size $h$, there are $n-h+1$ windows, so the filter produces a feature map $[v_1,\dots,v_{n-h+1}]$. The same weights are used at every position. This weight sharing is what lets a pattern learned in one place be recognized anywhere.
5Max pooling: one number per filter
The feature map still has variable length: $n-h+1$ values. Max pooling over time keeps only the largest one:
$$\hat v=\max_i v_i$$This solves the length problem. Whether the sentence has 4 words or 400, each filter contributes exactly one number, so the representation has a fixed size: one value per filter. Try switching Figure 2 between 4 and 7 words: the feature map grows from 2 to 5 values, but max pooling still outputs a single value.
Max pooling also has a natural meaning. A filter is a pattern detector, and $\hat v$ answers “how strongly does this pattern appear anywhere in the sentence?” It doesn't matter whether not good appears at the start or the end of a review. Other pooling choices exist, such as average pooling (the mean of the feature map) and k-max pooling (the $k$ largest values, in order), but max pooling is the standard choice for classification.
6Many filters: a bank of n-gram detectors
One filter detects one pattern. A real model uses many filters, often with several sizes (for example 100 filters each of sizes 3, 4, and 5). Each filter gives one number after max pooling, and the numbers are concatenated into the sentence vector. With $K$ filters in total, the sentence becomes a $K$-dimensional vector that goes into the classifier.
To see what filters can learn, here's a hand-built example. The toy embeddings have three interpretable dimensions: positive, negative, and negation (good = [1, 0, 0], bad = [0, 1, 0], not = [0, 0, 1], and other words are zero). Four size-2 filters, each followed by ReLU, are set by hand to detect four bigram patterns. Sentences are padded with one zero vector on each side.
In a trained CNN, nobody sets the filters by hand. They're learned by backpropagation from the cross-entropy loss, and the patterns they pick up are usually less clean than these. But inspecting which windows give each filter its maximum value often reveals phrase-like detectors of exactly this kind.
Parameters and weight sharing
A filter of size $h$ over $d$-dimensional embeddings has $d\times h$ weights plus one bias, and that count doesn't depend on the sentence length. Compare the two approaches for $d=300$-dimensional embeddings:
| Model | first layer | parameters | sentence length |
|---|---|---|---|
| Concatenation + MLP | 50 words × 300 dims → 100 hidden units | 1,500,100 | exactly 50 |
| CNN, one size | 100 filters of size 3 | 100 × (300·3 + 1) = 90,100 | any |
| CNN, three sizes | 100 filters each of sizes 3, 4, 5 | 360,300 | any |
7From one layer to multiple layers
Instead of pooling right away, we can feed the feature maps into a second convolution layer. If layer 1 has $K$ filters, then each window position $i$ now has a $K$-dimensional vector $[v_i^{(1)},\dots,v_i^{(K)}]$, one value per filter. These vectors play the role that word embeddings played in layer 1: layer 2's filters have size $K\times h$ and slide over them in the same way.
Each layer-2 output combines several layer-1 outputs, each of which already covered several words. So layer 2 sees a wider span of the sentence, called its receptive field. With filter size 3 in both layers, a layer-2 value covers 3 layer-1 windows, which together span 5 words. Each extra layer of size-$h$ filters adds $h-1$ more words.
Stacking layers lets the network build hierarchical features: layer 1 detects short phrases, and layer 2 detects combinations of phrases. For sentence classification, though, a single convolution layer with several filter sizes followed by max pooling is often enough. Deeper CNNs pay off more on long inputs, such as documents or character sequences.
8Putting it together
A standard CNN for sentence classification (Kim, 2014) puts all the pieces in a line:
- Embed: look up a $d$-dimensional embedding for each word (for example $d=300$), giving a $d\times n$ matrix.
- Convolve: apply filters of several sizes (Kim used 100 filters each of sizes 3, 4, and 5), each followed by ReLU.
- Pool: max pool each feature map over time, giving one number per filter: a 300-dimensional sentence vector.
- Classify: apply dropout, then a linear layer and a softmax over the $C$ classes.
- Train end to end with the cross-entropy loss, updating the filters (and optionally the embeddings) by backpropagation.
This simple one-layer model performed competitively with far more complex systems on sentiment and question classification benchmarks. It's still a strong, fast baseline.
CNNs also have clear limits. Each filter only sees a fixed-size window, so dependencies between distant words have to be built up through many layers. And the model doesn't process text the way a person reads, left to right, keeping a running memory of what came before. Models that read a sentence one word at a time are designed to address this.
9Check your understanding
Exercise 1: a filter by hand
A 3-word sentence has 2-dimensional embeddings $x_1=[1,\,2]$, $x_2=[0,\,1]$, $x_3=[2,\,-1]$. A size-2 filter has weights $w_{1,1}=1$, $w_{1,2}=0$ (dimension 1) and $w_{2,1}=-1$, $w_{2,2}=1$ (dimension 2), bias $b=1$, and ReLU. Compute the feature map and the max-pooled value.
$v_1$: dimension 1 gives $1\cdot1+0\cdot0=1$; dimension 2 gives $-1\cdot2+1\cdot1=-1$; plus bias: $1-1+1=1$; ReLU: $1$.
$v_2$: dimension 1 gives $1\cdot0+0\cdot2=0$; dimension 2 gives $-1\cdot1+1\cdot(-1)=-2$; plus bias: $0-2+1=-1$; ReLU: $0$.
Feature map $[1,\,0]$, max pooled value $\hat v=1$.
Exercise 2: sizes and parameters
Sentences have $n=20$ words with $d=50$-dimensional embeddings. A convolution layer has 64 filters, all of size $h=4$, with no padding, followed by max pooling. (1) How long is each feature map? (2) What is the size of the pooled sentence vector? (3) How many parameters does the layer have? (4) Which of these answers change if $n=35$?
Exercise 3: receptive field
Three convolution layers are stacked, each with filter size 3. How many input words does one value in layer 3 depend on?
Quick quiz
10Further reading
- Kim (2014). Convolutional Neural Networks for Sentence Classification. EMNLP. (The one-layer CNN described in section 8.)
- Zhang & Wallace (2017). A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification. IJCNLP. (How filter sizes, number of filters, and pooling affect results.)
- Kalchbrenner, Grefenstette & Blunsom (2014). A Convolutional Neural Network for Modelling Sentences. ACL. (Multi-layer CNN with k-max pooling.)
- Collobert, Weston, Bottou, Karlen, Kavukcuoglu & Kuksa (2011). Natural Language Processing (Almost) from Scratch. JMLR. (Early convolutional models for NLP.)