Long Short-Term Memory

Read this sentence and predict the missing word: “The cats that the dog chased ___ cute.” You'd write are, not is, because the subject is cats, even though the singular dog came in between. To get this right, a model has to store one fact (“the subject is plural”) for several words, protect it from the distracting words in the middle, and use it at exactly the right moment.

A plain recurrent neural network (RNN) is bad at this. It rewrites its whole memory at every step, so old information fades quickly, and its training signal from distant words vanishes. The Long Short-Term Memory (LSTM) network fixes this by adding a second, protected memory and a set of gates that decide what to erase, what to write, and what to read out. This reading builds the LSTM one piece at a time and then looks at a simpler variant, the GRU.

After reading this, you should be able to explain why a plain RNN struggles with long-range information · describe the roles of the cell state and hidden state · write the forget, input, and output gate equations and the cell and hidden state updates · compute one LSTM step by hand · explain why the additive cell update helps gradients flow · write the GRU equations and compare RNN, LSTM, and GRU.

1Why a plain RNN forgets

A plain RNN updates its hidden state with

$$h_t=\tanh\!\left(W h_{t-1}+U x_t+b\right).$$

There's only one memory, $h_t$, and it's completely recomputed at every step. Information from early words survives only by being passed through $W$ and the tanh again and again. During training, the gradient that tells the model how an early word affected a later prediction is a product of many step-to-step factors, one per step. When those factors are below 1, the product shrinks exponentially: the vanishing gradient problem. In practice, a plain RNN learns to use only the last several words.

The LSTM's answer has two parts:

A separate long-term memory, updated mostly by addition instead of being recomputed through a multiplication and a squashing function at every step.
Gates that control, at every step, how much of that memory to erase, how much new information to add, and how much to expose.

2Two memories and gates

An LSTM carries two vectors from step to step, both of size $d_h$:

the cell state $C_t$, the long-term memory. It runs along the top of the standard LSTM diagram like a conveyor belt, with only small, controlled changes at each step;
the hidden state $h_t$, the short-term memory. It's what the LSTM outputs at step $t$ and what the gates look at to make their decisions.

A gate is a vector of numbers between 0 and 1, computed with a sigmoid $\sigma(z)=1/(1+e^{-z})$. It's applied by element-wise multiplication, written $*$ (multiply matching entries; it is not a matrix product). An entry of 1 lets the corresponding value through unchanged, 0 blocks it completely, and anything in between lets part of it through. Each gate is computed from the previous hidden state and the current input, with its own parameters:

$$\text{gate}_t=\sigma\!\left(W^{(\cdot)}h_{t-1}+U^{(\cdot)}x_t+b^{(\cdot)}\right).$$

3The LSTM, step by step

At each step, the LSTM reads $x_t$ and $h_{t-1}$, updates the cell state, and then produces the new hidden state. There are four computations.

1. Forget gate: how much to erase

$$f_t=\sigma\!\left(W^{(f)}h_{t-1}+U^{(f)}x_t+b^{(f)}\right)$$

For each entry of the cell state, $f_t$ says how much of the old value to keep: 1 means keep it all, 0 means erase it.

2. Input gate and new information: what and how much to write

$$\tilde C_t=\tanh\!\left(W^{(C)}h_{t-1}+U^{(C)}x_t+b^{(C)}\right),\qquad i_t=\sigma\!\left(W^{(i)}h_{t-1}+U^{(i)}x_t+b^{(i)}\right)$$

The candidate $\tilde C_t$ (values between −1 and 1) is the new information that could be written. The input gate $i_t$ says how much of it to actually write.

3. Update the cell state

$$C_t=f_t*C_{t-1}+i_t*\tilde C_t$$

Erase part of the old memory, then add part of the new information. This is the heart of the LSTM, and it's an addition: the new long-term memory is the (partly kept) old one plus something new.

4. Output gate: what to read out

$$o_t=\sigma\!\left(W^{(o)}h_{t-1}+U^{(o)}x_t+b^{(o)}\right),\qquad h_t=o_t*\tanh(C_t)$$

The cell state may hold information that isn't relevant right now. The output gate $o_t$ decides which parts of the (squashed) memory to expose as the hidden state, which is used for predictions at this step and for computing the gates at the next step.

Try the update with a 1-dimensional LSTM, where every vector is a single number.

Presets:

Figure 1 (interactive). One LSTM step with scalar values. Set the gates directly (in a real LSTM they are computed from $h_{t-1}$ and $x_t$) and watch how the old memory, the new information, and the output combine.

4A worked example: remembering the subject

Back to “The cats that the dog chased are cute.” Imagine one entry of the cell state that tracks “the subject is plural” (+1) or “singular” (−1). The gate values below are chosen by hand to show the kind of behavior a trained LSTM can learn; they aren't computed from real weights.

Figure 2 (interactive). One cell-state entry over the sentence (red), compared with a simplified plain-RNN memory $m_t=0.5\,m_{t-1}+s_t$ (grey dashed), where $s_t$ is +0.9 for cats, −0.9 for dog, and 0 otherwise. The LSTM writes “plural” at cats, refuses to overwrite it at dog, holds it through the relative clause, and opens the output gate at chased, just before the verb must be predicted.

Notice the division of labor. The forget gate stays near 1 to hold the fact, the input gate stays near 0 so that dog doesn't overwrite it, and the output gate keeps the fact hidden until it's needed. A plain RNN has none of these controls: its memory of cats is diluted at every step and is mostly gone by the time the verb arrives.

5Why the addition helps gradients

Follow the cell state backward from $C_t$ to $C_{t-1}$. Looking only at the direct path through $C_t=f_t*C_{t-1}+i_t*\tilde C_t$,

$$\frac{\partial C_t}{\partial C_{t-1}}=f_t\quad\text{(element-wise)}.$$

There's no weight matrix and no tanh derivative on this path, just the forget gate. Over many steps, the gradient along the cell state is the product of the forget gates. If the network learns to keep $f_t$ close to 1 for information it needs, that product stays close to 1 and the gradient passes through many steps almost unchanged. This is the uninterrupted gradient flow along the top of the LSTM diagram. In a plain RNN, each step instead multiplies by $\text{diag}(1-h_t^2)\,W$, which tends to shrink the signal.

LSTM forget gate f0.95 RNN step factor0.50

Figure 3 (interactive). How much gradient signal survives after $k$ steps, on a log scale: $f^k$ along the LSTM cell state (red) versus (RNN factor)$^k$ (grey).

Not a guarantee. The LSTM doesn't make vanishing gradients impossible: if the forget gate is often small, the product still shrinks, and other paths through the gates still involve repeated multiplications. What it provides is an easy way for the network to preserve information and gradients over long distances, which a plain RNN would have to learn with much more difficulty. A common practical trick is to initialize the forget-gate bias $b^{(f)}$ to a positive value such as 1, so that $f_t$ starts near 1 and the LSTM remembers by default.

6Gated Recurrent Units (GRU)

The GRU is a simpler gated RNN. It has two gates instead of three and no separate cell state: the hidden state $h_t$ does both jobs.

$$\begin{aligned} r_t&=\sigma\!\left(W^{(r)}h_{t-1}+U^{(r)}x_t+b^{(r)}\right) &&\text{reset gate}\\ z_t&=\sigma\!\left(W^{(z)}h_{t-1}+U^{(z)}x_t+b^{(z)}\right) &&\text{update gate}\\ \tilde h_t&=\tanh\!\left(W(r_t*h_{t-1})+Ux_t+b\right) &&\text{candidate}\\ h_t&=(1-z_t)*h_{t-1}+z_t*\tilde h_t &&\text{new hidden state} \end{aligned}$$

The update gate $z_t$ merges the LSTM's forget and input gates into one: the new state is a weighted average of the old state and the candidate. When $z_t=0$, the old state is copied forward unchanged, which plays the same role as the LSTM's “keep” behavior and lets gradients flow over long distances. When $z_t=1$, the state is replaced by the candidate.
The reset gate $r_t$ decides how much of the previous state to use when computing the candidate. With $r_t\approx0$, the candidate depends almost only on the current word, letting the GRU “start fresh.”

Watch the convention. Here (as on the slides) $z_t$ is the weight on the new candidate. The original GRU paper (Cho et al., 2014) and some libraries use the opposite convention, $h_t=z_t*h_{t-1}+(1-z_t)*\tilde h_t$. The model is the same; only the meaning of $z_t$ flips.

7RNN vs. LSTM vs. GRU

	RNN	LSTM	GRU
Memory	$h_t$	$h_t$ and $C_t$	$h_t$
Gates	none	forget, input, output	reset, update
Weight sets $(W,U,b)$	1	4	3
Parameters ($d_x=300$, $d_h=128$)	54,912	219,648	164,736
Long-range dependencies	poor	good	good

Table 1. Each weight set has $d_h(d_h+d_x+1)$ parameters. The LSTM has four (three gates plus the candidate); the GRU has three.

In practice, LSTMs and GRUs usually perform similarly; neither wins across all tasks. The GRU has fewer parameters and is a bit faster to train, while the LSTM is the more widely used default. Both can be stacked into multiple layers and run bidirectionally, exactly like plain RNNs, and for several years bidirectional LSTMs were the standard building block for most NLP tasks.

A bit of history. The LSTM was introduced by Hochreiter and Schmidhuber in 1997, but the original version had no forget gate: the cell state could only accumulate. The forget gate was added by Gers, Schmidhuber, and Cummins in 2000, giving the LSTM used today.

8Check your understanding

Exercise 1: one LSTM step

For a 1-dimensional LSTM, suppose $C_{t-1}=1.0$, and at step $t$ the gates and candidate come out to $f_t=0.9$, $i_t=0.5$, $\tilde C_t=0.8$, $o_t=0.6$. Compute $C_t$ and $h_t$.

$C_t=0.9\times1.0+0.5\times0.8=0.9+0.4=1.3$.
$h_t=0.6\times\tanh(1.3)\approx0.6\times0.862\approx0.517$.
You can check this in Figure 1.

Exercise 2: counting parameters

With $d_x=100$ and $d_h=50$, how many parameters does an RNN cell, an LSTM cell, and a GRU cell have?

One weight set: $50\times(50+100+1)=7{,}550$. RNN: $7{,}550$. LSTM: $4\times7{,}550=30{,}200$. GRU: $3\times7{,}550=22{,}650$.

Exercise 3: gradients along the cell state

If the forget gate stays at 0.95 for 20 steps, what fraction of the gradient survives along the cell-state path? Compare with a plain RNN whose step factor is 0.5.

LSTM: $0.95^{20}\approx0.36$, so about a third survives. RNN: $0.5^{20}\approx9.5\times10^{-7}$, essentially nothing.

Quick quiz

9Further reading

Olah (2015). Understanding LSTM Networks. Blog post. (The source of the diagrams used on the slides.)
Hochreiter & Schmidhuber (1997). Long Short-Term Memory. Neural Computation.
Gers, Schmidhuber & Cummins (2000). Learning to Forget: Continual Prediction with LSTM. Neural Computation. (Adds the forget gate.)
Cho et al. (2014). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. EMNLP. (Introduces the GRU.)
Chung, Gulcehre, Cho & Bengio (2014). Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. (LSTM vs. GRU.)