RNN with Attention

When a human translator renders “I like cats a lot” as 我很喜歡貓, they don't memorize the English sentence and then write Chinese from memory. They glance back at the source as they write: at I while writing 我, at a lot while writing 很, at like while writing 喜歡, at cats while writing 貓.

A basic sequence-to-sequence (seq2seq) model can't glance back. Attention gives it that ability: at every decoding step, the decoder looks over all the encoder's hidden states and decides which ones matter right now. This reading builds attention on top of a recurrent encoder–decoder, step by step, with numbers you can follow.

After reading this, you should be able to explain the bottleneck problem of basic seq2seq models · compute attention scores, attention weights, and the context vector for one decoding step · describe how the context vector is combined with the decoder state to predict a word · read an attention map as a soft alignment · compare dot-product, multiplicative, and additive attention · explain why attention helps with both the bottleneck and vanishing gradients.

1The seq2seq bottleneck

In a basic seq2seq model, an encoder RNN reads the source words $x_1,\dots,x_n$ and produces hidden states $h_1,\dots,h_n$. A decoder RNN then starts from the final state $h_n$ and generates the target one token at a time, with decoder states $s_1,s_2,\dots$.

Two problems follow from this design:

The bottleneck. A single vector, $h_n$, must capture all the information about the source sentence. Everything the decoder will ever know about the input has to fit through it. That's manageable for five words, much less so for fifty.
Long paths. Information about the first source word must survive the entire encoder and then the decoder before it can affect a late output word. Over long sequences, gradients along these paths can still vanish, even with LSTMs.

Yet each output word needs only part of the source: 我 needs I, 很 needs a lot, 貓 needs cats. Different steps need different parts.

2The attention mechanism

Key idea: keep all the encoder states $h_1,\dots,h_n$, and at each decoding step $t$, compute a fresh summary of them that focuses on the parts relevant to the current decoder state $s_t$. One decoding step with attention has five parts:

Attention scores. Compare the decoder state with each encoder state, here with a dot product: $$\alpha_{t,i}=h_i^\top s_t\qquad(i=1,\dots,n)$$ A large score means encoder position $i$ is relevant to this step.
Normalized attention scores (attention weights). Turn the scores into a probability distribution over source positions with a softmax: $$\hat\alpha_{t,i}=\frac{\exp(\alpha_{t,i})}{\sum_{k=1}^{n}\exp(\alpha_{t,k})}$$
Weighted sum (context vector). Average the encoder states using those weights: $$a_t=\sum_{i=1}^{n}\hat\alpha_{t,i}\,h_i$$ This is a summary of the source tailored to step $t$.
Attention output. Combine the context with the decoder state, by concatenating them and applying a learned layer: $$\tilde o_t=\tanh\!\left(\mathbf W\,[a_t;\,s_t]\right)$$
Prediction. As in any decoder, a softmax over the vocabulary gives the next-word distribution: $$P(y_t\mid\cdot)=\text{softmax}(V\,\tilde o_t)$$

Step through the running example. The vectors below are small, hand-made 4-dimensional states, designed so that you can see attention “look” at the right source word.

Figure 1 (interactive). One decoding step of attention. Pick the output token being generated. The bars show the normalized attention weights $\hat\alpha_{t,i}$ over the five source words; below are the scores, the weights, and the resulting context vector $a_t$. (Toy vectors made up for this page; a trained model's states are high-dimensional, but the computation is the same.)

Notice what the context vector does. At the step that produces 我, $\hat\alpha$ puts most of its weight on I, so $a_t$ is nearly $h_{\text{I}}$. At the step that produces 很, the weight shifts to a lot. The decoder no longer depends on one fixed summary of the source: it builds a new one every step.

3Attention as soft alignment

Collect the attention weights from every decoding step into a matrix, with one row per output token and one column per source word, and you get an attention map. It shows which source words the model looked at when producing each output word.

Figure 2 (interactive). The attention map for the toy example: darker cells mean larger weights. Click a cell to see its weight. Each row sums to 1.

The map is a soft alignment between the two sentences: 我↔I, 很↔a lot, 喜歡↔like, 貓↔cats. Two things about it are worth noting:

It's soft: one output word can spread its attention over several source words (很 uses both a and lot), and two output words can attend to the same source word (喜 and 歡 both look at like).
It's learned without supervision: nobody tells the model which words correspond. The alignment emerges because attending to the right words helps predict the next target word. Real translation models produce maps like this, with crossings where the two languages order words differently.

4Why attention helps

It removes the bottleneck. The decoder can read directly from every encoder state, so the source no longer has to fit into one vector.
It shortens gradient paths. Every encoder state connects to every decoder step through the weighted sum. Gradients can flow along these short connections instead of only through long recurrent chains, which helps with vanishing gradients.
It's interpretable. The attention weights show, at least roughly, what the model was looking at, as in Figure 2.
It costs extra computation. Each decoding step computes $n$ scores, so a target of length $m$ needs $n\times m$ scores. For sentence-length inputs this is cheap.

Attention in general. Strip away the RNNs and attention is a general operation: given a query vector (here $s_t$) and a set of vectors to look at (here $h_1,\dots,h_n$), score each vector against the query, normalize the scores with a softmax, and return the weighted sum. The output is a summary of the set that depends on the query. This operation is useful far beyond translation.

5Different types of attention

The only design choice in the scoring step is how to compare $h_i$ with $s_t$. Three common choices:

Type	Score $\alpha_{t,i}$	Notes
Dot-product	$h_i^\top s_t$	No parameters. Requires $h_i$ and $s_t$ to have the same dimension.
Multiplicative	$h_i^\top W s_t$	A learned matrix $W$ lets the dimensions differ and learns which features to compare. Also called Luong (“general”) attention.
Additive	$v^\top\tanh(W_1h_i+W_2s_t)$	A small one-hidden-layer network computes the score. Also called Bahdanau attention.

Table 1. Common attention scoring functions. $W$, $W_1$, $W_2$, and $v$ are learned. The rest of the computation (softmax, weighted sum) is the same for all three.

6Attention beyond translation

Any time a generator needs to consult a large input while producing its output step by step, attention fits naturally:

Machine translation: attention maps between source and target words reveal the learned alignment, including reorderings.
Speech recognition: the “Listen, Attend and Spell” model attends over audio frames while spelling out characters; its attention map moves steadily forward through the audio as the transcript is written.
Image captioning: “Show, Attend and Tell” attends over regions of an image while generating a caption. When it writes bird, the attention concentrates on the bird.

7Check your understanding

Exercise 1: attention by hand

Three encoder states are $h_1=[1,0]$, $h_2=[0,1]$, $h_3=[1,1]$, and the decoder state is $s_t=[2,0]$. Using dot-product attention, compute the scores, the attention weights, and the context vector $a_t$.

Scores: $h_1^\top s_t=2$, $h_2^\top s_t=0$, $h_3^\top s_t=2$.
Weights: $e^2\approx7.389$, $e^0=1$, $e^2\approx7.389$; the sum is $15.778$, so $\hat\alpha\approx(0.468,\,0.063,\,0.468)$.
Context: $a_t=0.468[1,0]+0.063[0,1]+0.468[1,1]\approx[0.937,\,0.532]$.

Exercise 2: cost

The source has 20 words and the model generates a 25-token translation. How many attention scores are computed in total? What is the size of each step's attention weight vector?

$20\times25=500$ scores. Each step's weight vector has 20 entries (one per source word), and they sum to 1.

Exercise 3: scoring functions

The encoder states have dimension 512 and the decoder states have dimension 256. Which scoring functions in Table 1 can be used directly? What is the shape of $W$ in multiplicative attention?

Dot-product attention can't be used, because $h_i^\top s_t$ needs equal dimensions. Multiplicative and additive attention both work. In $h_i^\top W s_t$, $W$ must be $512\times256$.

Quick quiz

8Further reading

Bahdanau, Cho & Bengio (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR.
Luong, Pham & Manning (2015). Effective Approaches to Attention-based Neural Machine Translation. EMNLP.
Olah & Carter (2016). Attention and Augmented Recurrent Neural Networks. Distill.
Chan, Jaitly, Le & Vinyals (2016). Listen, Attend and Spell. ICASSP.
Xu et al. (2015). Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. ICML.