Sequential Labeling

Some NLP tasks need one answer per sentence: is this review positive or negative? Many others need an answer for every word. Is close a verb or an adjective here? Which words are names of people or places? Which span of this paragraph answers the question? This reading shows how a recurrent neural network (RNN) can produce one label per word, and how a surprising range of tasks, even building parse trees, can be rewritten as “one label per word” problems.

This family of tasks is called sequential labeling (or sequence labeling, or tagging): given words $w_1,\dots,w_n$, predict labels $y_1,\dots,y_n$, one per word.

1From sentence-level to token-level encoders

An RNN reads word embeddings $x_1,\dots,x_n$ and produces hidden states $h_1,\dots,h_n$. There are two ways to use them:

• Sentence-level encoder. Keep only the final state $h_n$ as a sentence embedding, and feed it to a classifier: one label for the whole sentence.
• Token-level encoder. Keep every $h_t$ as a representation of word $t$, and attach a classifier to each one: one label per word.

The token-level representations $h_t$ are contextualized: unlike a word embedding $x_t$, which is the same for a word wherever it appears, $h_t$ depends on the words around it. That's exactly what tagging needs, because a word's label often depends on its context.

A forward RNN's $h_t$ only sees words $1,\dots,t$. That's enough to tell can close from is close, but not enough to tell Washington signed the bill (a person) from Washington is a state (a place): at the first word, the forward RNN hasn't seen anything that disambiguates. For tagging, the whole sentence is available up front, so in practice we use a bidirectional RNN (usually a BiLSTM), whose state at each position combines a left-to-right and a right-to-left pass and therefore sees the whole sentence.

2Part-of-speech tagging

Part-of-speech (POS) tagging assigns each word its grammatical category. A common tag set is the Penn Treebank's, with about 45 tags (the cross-lingual Universal POS set has 17):

Tagging each word from its word embedding alone fails here: both occurrences of close have the same embedding, so they must get the same tag. A token-level RNN fixes this, because $h_t$ for close differs between the two sentences.

Tagging is a structured prediction problem. With a tag set $T$ and $n$ words there are $|T|^n$ possible outputs, and the labels constrain each other: a determiner is usually followed by an adjective or a noun, not by another determiner. The simplest neural approach still predicts each label separately from its contextualized $h_t$, and relies on the RNN to carry the context.

3Named entity recognition and BIO tags

Named entity recognition (NER) finds spans of text that name things, such as people (PER), locations (LOC), and organizations (ORG). The output is a set of spans, which may be several words long: New York City, Kuan-Hao Huang. To turn spans into one label per word, we use BIO tags:

• B-X: the beginning of a span of type X,
• I-X: inside a span of type X (any word after the first),
• O: outside any span.

The B tag is what lets us separate two adjacent entities of the same type. The slides use a single generic type (B-Entity, I-Entity); real datasets distinguish types, as below. A word whose label is I-X must come right after a B-X or another I-X, so a sequence like O, I-LOC is invalid.

Notice the difference between the two metrics. Token accuracy counts correct labels. Span-level F1, the standard NER metric, counts an entity as correct only if both its boundaries and its type match exactly. In the default prediction, a single wrong label (City tagged O) costs only 10% of token accuracy but turns a correct entity into a wrong one, which hurts both precision and recall.

4Extractive question answering

In extractive question answering, the input is a question and a passage, and the answer is a span copied from the passage. For example, given a passage about Super Bowl 50 and the question “Which NFL team represented the AFC at Super Bowl 50?”, the answer is the span Denver Broncos.

This is span finding again, so BIO tags work: label the answer's first word B-Ans, the rest of it I-Ans, and every other passage word O.

One detail matters: the labels depend on the question, so the model must see it. A common setup feeds the question and the passage together as one sequence, question [SEP] passage, and only the passage words are labeled. The same passage paired with a different question (“Who did the Broncos defeat?”) gets a different labeling.

5Dependency parsing as sequential labeling

A dependency parse connects each word to its head, the word it modifies or depends on, with a labeled arc describing the relation. Exactly one word attaches to a special <ROOT> token, and the arcs form a tree.

A tree isn't a sequence of labels, but it can be encoded as one: since every word has exactly one head, give each word $w_i$ the label $(p_i,\,l_i)$, where

• $p_i$ is the relative offset of the head: head position minus word position (with <ROOT> at position 0), and
• $l_i$ is the dependency relation (nsubj, dobj, amod, …).

Decoding is just as easy: from each word's offset, recover its head and draw the arc. A tagger trained to predict $(p_i,l_i)$ is therefore a dependency parser. A predicted label sequence isn't guaranteed to form a valid tree (it might contain a cycle or two roots), so practical systems add a light repair step.

6Constituency parsing as sequential labeling

A constituency parse breaks a sentence into nested phrases (constituents): noun phrases (NP), verb phrases (VP), prepositional phrases (PP), up to the whole sentence (S). The words are the leaves, each with its POS tag.

To encode this tree as one label per word, look at each pair of neighboring words $w_i$ and $w_{i+1}$ and record $(n_i,\,c_i)$:

• $n_i$: the number of common ancestors (phrase nodes above both words), and
• $c_i$: the nonterminal at their lowest common ancestor.

The labels for the whole sentence are 2NP, 1S, 2VP, 4NP, 4NP, 3NP, 4PP, 5NP, 1S. The depth of the shared ancestors rises and falls as the phrases open and close, which is enough information to rebuild the tree. As with dependencies, a tagger that predicts these labels (typically together with POS tags) is a constituency parser.

7Training a sequence labeler

Every task above has the same model: a (usually bidirectional) RNN produces $h_1,\dots,h_N$ for an $N$-word input, and a shared classifier predicts a label distribution from each $h_i$. The training loss is the average of the per-word cross-entropy losses:

The individual classifications are connected only through the shared RNN states: each $y_i$ is predicted from its own $h_i$. The model never directly sees its other predicted labels, so it can output an invalid sequence like O, I-PER. A common extension adds a conditional random field (CRF) layer on top of the BiLSTM, which scores whole label sequences and learns which label transitions are allowed. That extension is beyond the scope of this reading.

8Check your understanding

Exercise 1: BIO tagging

Tag Barack Obama visited Paris and London yesterday with BIO tags, using PER and LOC.

Exercise 2: span-level F1

A sentence has 4 gold entities. A tagger predicts 5 spans, of which 3 exactly match gold spans (same boundaries and type). Compute precision, recall, and F1.

Exercise 3: dependency labels

In She reads books, reads is the root, She is its nsubj, and books is its dobj. With <ROOT> at position 0, write the $(p_i,l_i)$ label of each word.

Quick quiz

9Further reading

• Huang, Xu & Yu (2015). Bidirectional LSTM-CRF Models for Sequence Tagging.
• Lample, Ballesteros, Subramanian, Kawakami & Dyer (2016). Neural Architectures for Named Entity Recognition. NAACL.
• Rajpurkar, Zhang, Lopyrev & Liang (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. EMNLP. (The source of the Super Bowl 50 example.)
• Strzyz, Vilares & Gómez-Rodríguez (2019). Viable Dependency Parsing as Sequence Labeling. NAACL.
• Gómez-Rodríguez & Vilares (2018). Constituent Parsing as Sequence Labeling. EMNLP.