Text Classification: Formulation

Is this product review happy or angry? Is this news article about health or politics? Is this post spreading false information? Should this chatbot request go to the math engine or the search engine? Each question takes a piece of text and asks for one answer from a short, fixed list. That's text classification, one of the most common tasks in natural language processing.

This reading defines the task precisely, frames it as supervised learning, explains how to split data into training, validation, and test sets, discusses how to measure performance, and lays out the general two-part framework that almost every text classifier follows.

After reading this, you should be able to recognize text classification tasks and their label sets · distinguish binary, multi-class, and multi-label classification · explain why generation can be viewed as repeated classification · state the supervised learning setup for classification · explain the roles of the training, validation (dev), and test sets · compute accuracy, precision, recall, and F1, and explain why accuracy can mislead · describe the feature + classifier framework.

1Text classification is everywhere

Task	Input text	Label set
Sentiment analysis	a product review	{Positive, Negative}
Topic classification	a news article	{Technology, Business, Economy, Health, Politics, Education, Sports}
Misinformation detection	a social media post	{Suspicious, Normal}
Request routing in LLM systems	a user's prompt	{normal, math mode, code mode, …}, {use search, use calculator, …}
Safety filtering	a user's prompt	{safe, harmful, …}

Table 1. Examples of text classification. Large language model (LLM) products quietly run many classifiers behind the scenes, for example to decide which mode or tool to use, or whether a request raises ethical issues.

Generation is a sequence of classifications

Even text generation is built from classification. To continue “This morning, I woke …”, a language model picks the next word from all possible words: up. Then it picks the next one, at, and so on. Each step is a classification problem whose label set is the entire vocabulary. Understanding classification well is therefore the foundation for almost everything else in NLP.

2The task, formally

In text classification, the input is a piece of text $x$. It can be a phrase, a sentence, a paragraph, or a whole document. The output is a category (or class) from a fixed, finite label set

$$C=\{c_1,c_2,\dots,c_k\},$$

with $k=|C|$ classes. For the news article “A small study found that ChatGPT outdid human physicians when assessing medical case histories…”, the label set is the seven news topics and a good answer is Health.

Tasks differ in how many classes there are and how many labels each text can have:

Binary classification: exactly two classes ($k=2$), such as Positive vs. Negative.
Multi-class classification: more than two classes, and each text gets exactly one of them, such as one news topic per article.
Multi-label classification: each text can get any number of classes. The ChatGPT article above is arguably both Health and Technology. A multi-label task is often handled as $k$ separate yes/no decisions, one per class.

3Supervised learning

We don't write classification rules by hand. Instead, we learn them from examples whose correct labels are known. This is supervised learning, and it has two stages.

Training stage. We are given labeled training data

$$\mathcal D_{\text{train}}=\{(x_1,y_1),(x_2,y_2),\dots,(x_m,y_m)\},$$

where each example $x_i\in\mathcal X$ (the space of all possible texts) comes with its label $y_i\in C$. We use it to train a classifier (or model), a function

$$f:\mathcal X\to C.$$

Testing stage. We then measure how well $f$ works on new labeled data $\mathcal D_{\text{test}}=\{(x_1,y_1),\dots,(x_n,y_n)\}$ that was not used for training. The model makes predictions $\tilde y_i=f(x_i)$, and we compare them with the true labels using a scoring function $S$:

$$\text{Performance}=\frac1n\sum_{i=1}^{n}S(y_i,\tilde y_i).$$

The simplest choice is $S(y,\tilde y)=1$ if $y=\tilde y$ and $0$ otherwise. Then the average is the accuracy: the fraction of test examples classified correctly.

Why a separate test set? The goal is generalization: doing well on texts the model has never seen. A model can score perfectly on its training data simply by memorizing it, so training performance says little about real-world performance. The test set must be kept separate from the training data and used only for the final evaluation.

4Training, validation, and test sets

Building a classifier involves many choices besides the learned parameters: which features to use, how large a model to train, how long to train it, what learning rate to use. These choices are called hyperparameters, and we pick them by trying several options and keeping the best one.

But best according to what? If we choose the option with the highest test score, the test set has influenced the model, and its score is no longer an honest estimate of performance on new data. The standard fix is a third split:

Training set: used to learn the model's parameters.
Validation set (also called the development or dev set): used to compare hyperparameter settings and make design decisions, such as when to stop training.
Test set: used once, at the very end, to report the final performance of the chosen model.

Model complexity setting2

Figure 1 (interactive). Model selection with a validation set (illustrative numbers). As the model gets more complex, training accuracy keeps rising, but dev accuracy peaks and then falls as the model starts memorizing the training data (overfitting). We choose the setting with the best dev accuracy, and only then look at the test set, once.

Typical splits are around 80/10/10 or 70/15/15 of the labeled data, though many benchmark datasets come with fixed official splits so that results are comparable across papers. When labeled data is scarce, k-fold cross-validation can replace a single dev set: split the training data into $k$ parts, train $k$ times, each time validating on a different part, and average the results.

5Measuring performance: beyond accuracy

Accuracy is easy to understand but can be badly misleading when classes are imbalanced. Suppose 20 of 1,000 posts contain misinformation. A “classifier” that labels everything Normal gets 98% accuracy, and it is completely useless.

For a class we care about (the positive class, here Suspicious), count:

TP (true positives): suspicious posts predicted suspicious,
FP (false positives): normal posts predicted suspicious,
FN (false negatives): suspicious posts predicted normal,
TN (true negatives): normal posts predicted normal.

Then

$$\text{Precision}=\frac{TP}{TP+FP},\qquad \text{Recall}=\frac{TP}{TP+FN},\qquad F_1=\frac{2\cdot\text{Precision}\cdot\text{Recall}}{\text{Precision}+\text{Recall}}.$$

Precision asks: of the posts we flagged, how many really were suspicious? Recall asks: of the suspicious posts, how many did we catch? F1 is their harmonic mean, which is high only when both are high.

suspicious posts caught (TP)15 normal posts flagged (FP)25

Figure 2 (interactive). A confusion matrix for misinformation detection on 1,000 posts, 20 of which are suspicious. Move the sliders or pick a preset and compare accuracy with precision, recall, and F1.

For multi-class problems, precision, recall, and F1 are computed for each class and then averaged: a macro average treats every class equally, while a micro average pools all the counts first. Which metric matters depends on the application: a spam filter should have high precision (don't lose real email), while a medical screening tool should have high recall (don't miss a case).

6A general framework: features and classifiers

Knowing what to learn still leaves two questions: how does the model understand the example $x$, and how does it make a prediction $y$? Almost every text classifier answers them with two components:

$$\text{Text } x\;\longrightarrow\;\boxed{\text{Feature (Representation)}}\;\longrightarrow\;\boxed{\text{Classifier (Model)}}\;\longrightarrow\;\text{Label } y$$

Feature (representation): teaches the model how to understand $x$ by converting the text into a mathematical form, usually a vector of numbers, that captures its essential characteristics. Examples include bag-of-words counts, n-gram features, and word embeddings.
Classifier (model): teaches the model how to make a prediction $y$ from that representation. Examples include logistic regression, feedforward neural networks, CNNs, RNNs, LSTMs, and Transformers.

Both components can range from very simple to very powerful. A simple representation might only record which words appear in the text; a powerful model might read the words in order and take their context into account. Whatever the choices, the two-part structure stays the same, and the classifier's parameters are learned from $\mathcal D_{\text{train}}$.

7Check your understanding

Exercise 1: task types

Classify each task as binary, multi-class, or multi-label: (a) deciding whether an email is spam; (b) assigning a movie review a star rating from 1 to 5; (c) tagging a research paper with all relevant subject areas.

(a) Binary (spam / not spam). (b) Multi-class: five classes, exactly one per review. (c) Multi-label: a paper can belong to several areas at once.

Exercise 2: precision and recall

A spam filter is evaluated on 500 emails, 50 of which are spam. It flags 40 emails as spam, and 30 of those really are spam. Compute accuracy, precision, recall, and F1 for the spam class.

TP $=30$, FP $=40-30=10$, FN $=50-30=20$, TN $=500-30-10-20=440$.
Accuracy $=(30+440)/500=0.94$. Precision $=30/40=0.75$. Recall $=30/50=0.60$. $F_1=\frac{2\cdot0.75\cdot0.6}{0.75+0.6}=\frac{0.9}{1.35}\approx0.667$.

Exercise 3: using the splits

A student trains five models with different learning rates, reports the test accuracy of all five, and claims the best one as the final result. What's wrong, and how should they fix it?

Choosing the best of five by test accuracy uses the test set for model selection, so the reported number is optimistically biased. They should hold out a validation (dev) set, pick the learning rate by dev accuracy, and then evaluate only the chosen model on the test set, once.

Quick quiz

8Further reading

Voita. NLP Course: Text Classification. (An illustrated introduction to the feature + classifier view.)
Pang, Lee & Vaithyanathan (2002). Thumbs up? Sentiment Classification using Machine Learning Techniques. EMNLP.
scikit-learn documentation. Cross-validation: evaluating estimator performance. (Train/validation/test splits and k-fold cross-validation.)