CSCE 471 Natural Language Processing · Spring 2027

Recurrent Neural Networks

Kuan-Hao Huang · Course Website
Note: This page was created with the help of AI as a supplementary reading. It may contain errors or simplifications. The lecture slides are the primary reference for this course; please refer to them first.

When you read “All the cats are cute,” you don't look at the whole sentence at once. You read one word at a time, and as you go you keep a running mental summary: all of something, the cats, they are something, cute. By the last word, that summary is your understanding of the sentence.

A recurrent neural network (RNN) processes text the same way. It keeps a vector called the hidden state, its memory, and updates it once per word: read the next word, update the memory. Because the same update rule is applied at every step, one small network can handle sentences of any length, and in principle the final memory can depend on every word that came before.

After reading this, you should be able to write the RNN update rule and compute hidden states by hand · explain why an RNN's size doesn't depend on the input length · use an RNN for text classification and for language modeling · describe stacked and bidirectional RNNs · compare the strengths and weaknesses of RNNs · explain where the vanishing and exploding gradient problems come from.

1The recurrent unit

Let $x_1,\dots,x_n$ be the word embeddings of a sentence. An RNN computes a sequence of hidden states $h_1,\dots,h_n$, starting from an initial state $h_0$ (usually the zero vector):

$$h_t=\sigma\!\left(W h_{t-1}+U x_t+b\right)$$

The learnable parameters are $W$ (size $d_h\times d_h$), $U$ (size $d_h\times d_x$), and $b$ (size $d_h$), where $d_x$ is the embedding size and $d_h$ is the hidden size. The key point is that the same $W$, $U$, and $b$ are used at every time step. Whether a sentence has 5 words or 500, the RNN has the same number of parameters: it just applies the same update more times.

Step through the running example with a tiny RNN that has 2-dimensional embeddings and a 2-dimensional hidden state, so you can follow every number.

Figure 1 (interactive). A 2-dimensional RNN with $W=\begin{bmatrix}0.5&-0.3\\0.2&0.6\end{bmatrix}$, $U=I$, $b=0$, $\sigma=\tanh$, $h_0=0$. The plot shows the path of the hidden state as each word is read. Reversing the word order gives a different final state: unlike averaging embeddings, an RNN is sensitive to order. (Embeddings are made up for this page.)

2Unrolling the RNN for classification

Drawing one copy of the recurrent unit per time step is called unrolling the RNN. The unrolled picture is a chain: $h_0\to h_1\to\dots\to h_n$, with word $x_t$ feeding into step $t$. It looks like a deep network with $n$ layers, except that every layer shares the same weights.

For text classification, the final hidden state $h_n$ has read the whole sentence, so it serves as the sentence representation. Feed it into a classifier (for example a feedforward network with a softmax) and train everything end to end with the cross-entropy loss over $C$ classes:

$$\mathcal L_{CE}(y,\tilde y)=-\sum_{c=1}^{C}y_c\log P(y=c\mid h_n)$$

Using only $h_n$ puts the whole sentence through a single vector, and the last few words tend to dominate it. A common alternative is to pool over all hidden states, using the elementwise mean or max of $h_1,\dots,h_n$, so that every position contributes directly.

3RNN language models

An RNN can also be a language model: a model of the probability of the next word given all previous words. At every step, add an output layer that turns the hidden state into a distribution over the vocabulary $\mathcal V$:

$$\hat y_t=\text{softmax}(V h_t),\qquad \hat y_t[w]=P(x_{t+1}=w\mid x_1,\dots,x_t)$$

Here $V$ is a $|\mathcal V|\times d_h$ output matrix (plus a bias, omitted). Unlike an n-gram model, which only sees a fixed window of the last few words, $h_t$ summarizes the entire prefix, so the prediction can in principle use words from far back. And the model has no counts to store: similar words have similar embeddings, so it can generalize to word sequences it never saw.

Training. Run the RNN over a training sentence. At each position, feed in the true current word (this is called teacher forcing) and score the probability assigned to the true next word. The loss is the sum of the per-step cross-entropies:

$$\mathcal L=-\sum_{t}\log P(x_{t+1}\mid x_1,\dots,x_t)$$
input so fartarget$P(\text{target})$$-\ln P$
<s>All0.052.996
<s> Allthe0.400.916
<s> All thecats0.102.303
<s> All the catsare0.500.693
<s> All the cats arecute0.201.609
<s> All the cats are cute</s>0.600.511
total9.028
Table 1. The loss for one training sentence, with made-up model probabilities. The average per-token loss is $9.028/6=1.505$, so the perplexity on this sentence is $e^{1.505}\approx4.50$.

The average per-token cross-entropy is exactly the log of the perplexity, so minimizing this loss directly lowers perplexity. To generate text, run the same model one step at a time: pick the next word from $\hat y_t$, feed it back in as $x_{t+1}$, and repeat until </s>.

4Stacked and bidirectional RNNs

Stacked (multi-layer) RNNs

The hidden states $h_1^{(1)},\dots,h_n^{(1)}$ of one RNN form a new sequence, so a second RNN can read them:

$$h_t^{(2)}=\sigma\!\left(W^{(2)}h_{t-1}^{(2)}+U^{(2)}h_t^{(1)}+b^{(2)}\right)$$

Each layer has its own parameters. As with stacked convolution layers, higher layers can build more abstract features. In practice, 2 to 4 layers are common.

Bidirectional RNNs

A left-to-right RNN's state $h_t$ only knows about words $1,\dots,t$. But for understanding a word, what comes after can matter as much as what comes before: in “the bank of the river,” the word river tells you which bank is meant. A bidirectional RNN runs two separate RNNs, one forward (left to right) and one backward (right to left), and concatenates their states at each position:

$$h_t=\big[\,\overrightarrow{h}_t\,;\,\overleftarrow{h}_t\,\big]$$
Figure 2 (interactive). Click a word to see which words its hidden state can depend on.

Bidirectional RNNs are a natural choice when the whole input is available up front: classifying a sentence, or labeling each word in it. They can't be used as language models or for left-to-right generation, because the backward RNN would need to see the future words the model is supposed to predict.

5Strengths and weaknesses

AdvantagesDisadvantages
Can process input of any lengthRecurrent computation is slow: $h_t$ needs $h_{t-1}$, so steps can't run in parallel
Model size doesn't grow with the input length (weights are shared)In practice, information from many steps back is hard to use
Step $t$ can, in theory, use information from all earlier stepsTraining suffers from vanishing (and exploding) gradients
Table 2. RNN strengths and weaknesses. The second and third weaknesses are closely related, as the next section shows.

Compared with a CNN, which looks at local windows of words in parallel, an RNN reads sequentially and carries a memory forward. That makes long-range information available in principle, at the price of sequential computation.

6Training and the vanishing gradient problem

An unrolled RNN is just a deep network with shared weights, so it's trained with ordinary back-propagation, applied through the unrolled chain. This is called back-propagation through time (BPTT). Because $W$ is used at every step, its gradient collects one contribution from each step. For a loss computed at step 5, the chain rule gives

$$\frac{\partial\mathcal L}{\partial W}=\sum_{t=1}^{5}\frac{\partial\mathcal L}{\partial h_5}\left(\prod_{k=t+1}^{5}\frac{\partial h_k}{\partial h_{k-1}}\right)\frac{\partial h_t}{\partial W},$$

where $\frac{\partial h_t}{\partial W}$ means the direct effect of $W$ on $h_t$ through step $t$'s own update, holding $h_{t-1}$ fixed. The term for step $t$ multiplies together $5-t$ step-to-step derivatives. For $h_k=\tanh(Wh_{k-1}+Ux_k+b)$, each one is

$$\frac{\partial h_k}{\partial h_{k-1}}=\text{diag}\!\left(1-h_k^2\right)W.$$

The contribution from $t$ steps back is a product of many such factors. If they are typically smaller than 1 (the derivative of tanh is at most 1, and the sigmoid's is at most 0.25), the product shrinks exponentially. A factor of 0.5 over 20 steps gives $0.5^{20}\approx10^{-6}$. This is the vanishing gradient problem: the gradient signal from distant words fades away, so the weights are updated mostly according to short-term effects, and the RNN struggles to learn long-range dependencies. If the factors are larger than 1, the product instead grows exponentially: the exploding gradient problem.

Recurrent weight w 0.80
Figure 3 (interactive). A 1-dimensional RNN $h_t=\sigma(w\,h_{t-1}+x_t)$ over 30 random inputs. Each bar shows how much a change in $h_t$ affects the last state, $|\partial h_{30}/\partial h_t|$, on a log scale. Bars far below 1 mean the gradient has vanished; bars far above mean it exploded. With tanh, saturation keeps the factors small, so this 1-dimensional example mostly vanishes; switch to “no activation” and set w > 1 to see it explode.
Remedies. Exploding gradients are usually handled with gradient clipping: if the gradient's norm exceeds a threshold, scale it down before the update. Vanishing gradients are harder. The standard solution is to change the recurrent unit itself, using gated units such as the LSTM and GRU that can pass information across many steps with little decay. Those architectures are beyond the scope of this reading.

7Check your understanding

Exercise 1: an RNN by hand

A 1-dimensional RNN has $h_t=\tanh(0.5\,h_{t-1}+x_t)$ and $h_0=0$. The inputs are $x_1=1$ and $x_2=-1$. Compute $h_1$ and $h_2$.

$h_1=\tanh(0.5\cdot0+1)=\tanh(1)\approx0.762$.
$h_2=\tanh(0.5\cdot0.762-1)=\tanh(-0.619)\approx-0.551$.

Exercise 2: counting parameters

An RNN classifier uses 300-dimensional embeddings and a 128-dimensional hidden state. (1) How many parameters are in $W$, $U$, and $b$? (2) How does this change for sentences that are twice as long? (3) If the same RNN becomes a language model with a vocabulary of 10,000 words, how many parameters does the output layer $V$ (with its bias) add?

(1) $W$: $128\times128=16{,}384$. $U$: $128\times300=38{,}400$. $b$: $128$. Total $54{,}912$.
(2) It doesn't change: the same weights are reused at every step.
(3) $10{,}000\times128+10{,}000=1{,}290{,}000$. The output layer is much larger than the recurrent part.

Exercise 3: vanishing gradients

Suppose every step-to-step derivative $\partial h_k/\partial h_{k-1}$ is about 0.5. Roughly how much weaker is the gradient signal from 20 steps back than from 1 step back? What if every factor were 1.5?

The signal from 20 steps back has 19 more factors: $0.5^{19}\approx1.9\times10^{-6}$ times weaker, effectively zero. With 1.5, it's $1.5^{19}\approx2.2\times10^{3}$ times stronger: the gradient explodes.

Quick quiz

8Further reading