Sequence-to-Sequence Models

Translate “I like cats a lot” into Chinese: 我很喜歡貓. The input is a sequence, the output is a sequence, and the two have different lengths, different words, and different word orders. Classifying the whole input (one label) or tagging each word (one label per input word) can't produce this. We need a model that first understands the input and then generates a new sequence of its own length.

A sequence-to-sequence (seq2seq) model does exactly that by joining two recurrent neural networks: an encoder that reads the input, and a decoder that writes the output one word at a time. This reading builds the decoder first, puts the two together, and then looks closely at how the model is trained, using a technique called teacher forcing, and how training differs from generation.

After reading this, you should be able to write the chain-rule factorization behind text generation · explain the roles of <bos> and <eos> · describe how a seq2seq model connects an encoder and a decoder · compute the training loss of a seq2seq model · explain teacher forcing and how training differs from inference · describe exposure bias and the information bottleneck · contrast seq2seq models with decoder-only models.

1Generation as a sequence of classifications

By the chain rule of probability, the probability of any word sequence can be written exactly as a product of next-word probabilities:

$$\begin{aligned} P(w_1,\dots,w_l)&=P(w_1)\,P(w_2,\dots,w_l\mid w_1)\\ &=P(w_1)\,P(w_2\mid w_1)\,P(w_3,\dots,w_l\mid w_1,w_2)\\ &=\prod_{i=1}^{l}P(w_i\mid w_1,\dots,w_{i-1}). \end{aligned}$$

So generating text reduces to repeatedly answering one question: given the words so far, what's the next word? Each answer is a classification over the whole vocabulary. A neural network with a fixed context window can only condition on the last few words. An RNN can condition on all of them, because its hidden state $h_t$ summarizes the entire prefix.

An RNN used this way is a decoder (or generator). Two special tokens make it work:

<bos> (begin of sentence) is fed in at the first step, so the model has an input before any real word exists.
<eos> (end of sentence) is a word the model can predict. Generating it means “stop,” which lets the model decide the output length.

input at step t<bos>Allthecatsarecute

target at step tAllthecatsarecute<eos>

Figure 1. Training an RNN decoder on one sentence. The input sequence is the target sequence shifted right by one position, with <bos> in front. Every target position (underlined) contributes a cross-entropy loss.

The training loss averages the cross-entropy over the $N$ target positions:

$$\mathcal L=\frac{1}{N}\sum_{i=1}^{N}\mathcal L_{CE}^{(i)},\qquad \mathcal L_{CE}^{(i)}=-\log P(\text{target}_i\mid \text{inputs}_{1..i}).$$

2Encoders and decoders

The same RNN machinery serves two different purposes:

An encoder focuses on representation and understanding. It reads a given input and produces vectors: one per word (for tagging) or one for the whole sequence (for classification). It may read in both directions, because the whole input is available.
A decoder focuses on generation. Its output at each step becomes the next step's input, so it must run left to right: it can't look at words it hasn't produced yet.

3Putting them together: seq2seq

When a task needs understanding and generation at the same time, connect an encoder to a decoder:

The encoder RNN reads the source $x_1,\dots,x_n$ and produces hidden states $h_1,\dots,h_n$.
Its final state $h_n$ becomes the initial state of the decoder. This single vector is the decoder's entire view of the input.
The decoder RNN, with its own separate parameters, generates the target $y_1,\dots,y_m$ one token at a time, starting from <bos> and stopping at <eos>.

The model defines a conditional language model over the output given the input:

$$P(y_1,\dots,y_m\mid x_1,\dots,x_n)=\prod_{t=1}^{m}P(y_t\mid y_1,\dots,y_{t-1},\,x_1,\dots,x_n).$$

Many tasks have this shape: machine translation (English → Chinese), grammatical error correction (a flawed sentence → a corrected one), summarization (a long document → a short summary), dialogue response generation, and more. For the running example, the source is I like cats a lot and the target is 我很喜歡貓, split into characters. That's a tokenization choice: 喜歡 (“like”) is one word but two tokens here.

A note on notation. On the slides, $y_t$ labels the decoder's input at step $t$: $y_1=$ <bos>, $y_2=$ 我, and so on, which is the target sequence shifted right by one. The decoder's output at step $t$ is a distribution over the next target token. Keep this one-position shift in mind; it's the core of teacher forcing.

4Training with teacher forcing

At each decoder step, the model needs an input token. During generation, the only option is the token the model itself just produced. During training, though, we know the correct target sentence, and we have a choice. Teacher forcing means: always feed the correct previous target token as the decoder's next input, regardless of what the model predicted.

With teacher forcing, the decoder inputs for 我很喜歡貓 are <bos>, 我, 很, 喜, 歡, 貓, and the targets are 我, 很, 喜, 歡, 貓, <eos>. The loss is the average of the six per-step losses $-\log P(\text{correct token})$, exactly like a language model's loss, except that every step is also conditioned on the source sentence.

Figure 2 (interactive). The decoder of a toy translation model for I like cats a lot. Above each decoder step are the model's top predictions. In training, the next input is always the gold token (green), and each step's loss is $-\ln P(\text{gold})$. In inference, the next input is the model's own prediction (dashed arrow). The third mode forces a mistake at step 1 to show how errors carry forward. (For simplicity, this toy model's predictions depend only on the source and the current input token.)

Why teacher forcing?

Every step learns from the right context. If the decoder were fed its own early, mostly wrong predictions, later steps would be trained on garbage prefixes, and their targets would no longer line up with what came before. Teacher forcing gives every position a meaningful training signal from the very start.
Training is stable and fast. Early in training the model's guesses are nearly random; without teacher forcing, learning would be much slower and noisier.
All inputs are known in advance. The decoder inputs are simply the target shifted right, so no sampling is needed during training. (For RNNs the steps still run in order; for architectures without recurrence, this allows all positions to be trained in parallel.)

The catch: exposure bias

Teacher forcing creates a mismatch. During training, the decoder only ever sees correct prefixes. During inference, it sees its own predictions, including its mistakes. When it makes one, it lands in a situation it never practiced, and the error can compound: in Figure 2, a wrong first word (你, “you”) leads the model to fluently produce 你喜歡貓 (“you like cats”), a different meaning. This mismatch is called exposure bias. A common remedy is scheduled sampling: during training, occasionally feed the model's own prediction instead of the gold token, and increase that probability over time, so the model gradually learns to recover from its own errors.

5Inference: generating the output

At test time there's no gold target. The decoder starts from the encoder's final state with input <bos>, and at each step:

computes a distribution over the vocabulary for the next token,
chooses a token (for example the most probable one, or with a search over several candidates such as beam search),
feeds the chosen token back in as the next input,

until it produces <eos> (or reaches a maximum length). Because each choice is fed back, generation is inherently sequential, and an early wrong choice affects everything after it.

6Limitations and practical tricks

The information bottleneck. Everything the decoder knows about the source must pass through one fixed-size vector, the encoder's final state $h_n$. For a five-word sentence that's fine, but for a 50-word sentence it's a lot to compress, and early source words are the most likely to be lost. Translation quality of basic RNN seq2seq models drops noticeably as sentences get longer. Letting the decoder look back at all the encoder states, not just the last one, is the natural fix.

Practical tricks.

Bidirectional encoder. The encoder sees the whole source up front, so it can be bidirectional. The decoder can't, because it generates left to right.
Stacked LSTMs. Strong early seq2seq systems used several LSTM layers in both the encoder and the decoder.
Reversing the source. Sutskever et al. (2014) found that feeding the source sentence in reverse order improved translation, because it puts the first source words close to the first target words, making early dependencies short.

7Decoder-only models vs. seq2seq models

A seq2seq model isn't the only way to map an input to an output. A decoder-only model is a single language model that simply continues writing: put the input in as a prompt, followed by a separator, and let the model generate the rest.

Seq2seq: two networks

encoder readsIlikecatsalot

decoder writes<bos>我很喜歡貓<eos>

Decoder-only: one network, one sequence

one streamIlikecatsalot<sep>我很喜歡貓<eos>

Figure 3. The same translation in the two styles. Underlined tokens are the ones whose prediction is trained (for a decoder-only model doing this task, the loss is often computed only on the output part).

Seq2seq: encode first, then generate. The input is processed by a dedicated encoder (which can be bidirectional), and the decoder generates conditioned on it.
Decoder-only: continue writing. Input and output live in one sequence and are processed by the same left-to-right model. The input is just the beginning of the text.

Both are trained with teacher forcing and generate one token at a time. With RNNs the two styles behave fairly similarly; the differences become more important in more recent architectures.

8Check your understanding

Exercise 1: inputs and targets

A seq2seq model is trained on the pair Good morning → 早安 (two target tokens). With teacher forcing, list the decoder's inputs and targets at each step. How many terms are in the loss?

Inputs: <bos>, 早, 安. Targets: 早, 安, <eos>. Three steps, so three cross-entropy terms, averaged: $\mathcal L=\frac13\big[-\log P(\text{早})-\log P(\text{安})-\log P(\text{<eos>})\big]$.

Exercise 2: computing the loss

Under teacher forcing, a decoder assigns probabilities 0.5, 0.8, and 0.9 to the three correct target tokens. What is the training loss (natural log)?

$-\ln0.5=0.693$, $-\ln0.8=0.223$, $-\ln0.9=0.105$. Average: $(0.693+0.223+0.105)/3\approx0.340$.

Exercise 3: training vs. inference

During training with teacher forcing, the model predicts 你 instead of 我 at step 1. What is the decoder's input at step 2? What would it be at inference time?

In training, the input at step 2 is the gold token 我; the wrong prediction only increases the step-1 loss. At inference, the input would be the model's own prediction, 你, and the rest of the output would be generated from that wrong prefix.

Quick quiz

9Further reading

Sutskever, Vinyals & Le (2014). Sequence to Sequence Learning with Neural Networks. NeurIPS.
Cho et al. (2014). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. EMNLP.
Williams & Zipser (1989). A Learning Algorithm for Continually Running Fully Recurrent Neural Networks. Neural Computation. (Origin of teacher forcing.)
Bengio, Vinyals, Jaitly & Shazeer (2015). Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks. NeurIPS.