Bag-of-Words Features

A classifier is a mathematical function: it takes numbers in and gives a label out. Text, though, is a string of characters. Before any classifier can work on “This restaurant is the best one in College Station,” we must turn the sentence into a vector of numbers that keeps the information the classifier needs.

The simplest way to do this, and still a surprisingly strong one, is to ignore grammar and word order entirely and just record which words appear. This reading builds that representation, called bag-of-words, shows where it fails, extends it with n-grams, and then compares the different ways of filling in the vector, from simple 0/1 values to TF-IDF weights.

1Text needs a mathematical form

Most text classifiers have two parts: a feature (representation) step that turns text $x$ into a vector, and a classifier (model) that turns the vector into a label $y$:

This reading is only about the first box: how to turn text into a vector. The features here are fixed rules that need no training at all. You define them once, and every text gets a vector.

2The bag-of-words representation

Imagine cutting a sentence into words and dropping them into a bag. The bag remembers which words went in, but not their order. That's a bag-of-words (BoW).

To turn the bag into a vector:

1. Build a vocabulary $V$: the list of all distinct words in the training data, usually after simple preprocessing such as lowercasing. Each word gets a fixed position (dimension).
2. Represent a text by a vector $\mathbf x$ of length $|V|$, the vocabulary size. In the binary version, $x_j=1$ if word $j$ appears in the text and $0$ otherwise.

Strictly speaking, a binary vector records a set of words. When we also keep counts (section 5), the representation is a true bag (a multiset), where repeated words count more than once.

Advantages

• Simple and fast: no training required, easy to compute for any text.
• Fixed length: every text, from a phrase to a book, becomes a vector of the same length $|V|$, which is exactly what a standard classifier needs.
• Interpretable: each dimension is a word, so you can read off exactly what the representation contains.

Disadvantages

• No word order: “Bob likes Alice very much” and “Alice likes Bob very much” have exactly the same BoW vector. BoW fails to capture sentence structure, including negation (not good).
• High-dimensional and sparse: $|V|$ is often tens of thousands, but a sentence uses only a handful of words, so almost every entry is 0.
• No notion of similarity: good and great are just two unrelated dimensions, so evidence learned for one doesn't transfer to the other.

3Capturing local order with n-grams

An n-gram is a sequence of $n$ consecutive words. A sentence with $L$ words has $L-n+1$ of them:

A bag-of-n-grams representation adds one dimension for each n-gram in the vocabulary, usually on top of the unigrams. The unigram part of Bob likes Alice and Alice likes Bob is still identical, but the bigram part differs (Bob likes vs. Alice likes), so the vectors are no longer the same. N-gram features capture more sentence structure, including short phrases such as not good and very much.

The price is size. The number of possible bigrams is $|V|^2$ and of trigrams $|V|^3$. Only the n-grams that actually occur in the training data get a dimension, and in practice we usually keep only those that occur at least a few times.

4Feature variants: what goes in each dimension?

The binary vector is only one option. The same dimensions can hold other numbers:

The four variants keep different amounts of information. Compare them on the same document below, and try repeating the document twice, which is what happens when a review says the same thing at twice the length.

• Binary → count: counts let repeated words matter more. A review that says great twice is probably more positive than one that says it once.
• Count → frequency: raw counts grow with document length, so a long review looks “more” of everything. Dividing by the length removes this effect: the doubled document has exactly the same frequencies.
• Frequency → TF-IDF: frequencies still give the most weight to the most common words, which are usually function words like the and is. Multiplying by IDF pushes those toward zero and highlights words that are distinctive for the document.

TF-IDF goes one step further. Frequent words aren't necessarily informative: the, is, and and appear in almost every document and say little about its content. TF-IDF multiplies two factors:

• Term frequency (TF) $f_t$: how often term $t$ appears in this document (here, its word frequency).
• Inverse document frequency (IDF) $\log\frac{N}{n_t}$, where $N$ is the total number of documents in the collection and $n_t$ is the number of documents that contain $t$. A term found in every document gets $\log 1=0$; a term found in few documents gets a large IDF.

So TF-IDF is high for words that are frequent in this document but rare in the collection, which are exactly the words that characterize it.

Comparing the variants

No variant is best for every task. Binary features are often enough for short texts, where most words appear once anyway. Counts and frequencies matter more for longer documents. TF-IDF is the standard choice when common words would otherwise dominate, for example in document retrieval or topic classification of long articles.

5Check your understanding

Exercise 1: binary BoW

The vocabulary is [a, cat, dog, the, sat, on, mat]. Write the binary BoW vectors of the cat sat on the mat and a dog sat.

Exercise 2: n-grams

List the bigrams of I do not like it. Which bigram captures meaning that the unigrams miss?

Exercise 3: TF-IDF by hand

A collection has $N=10$ documents. In a 20-word document, the word pasta appears 2 times and occurs in 2 of the 10 documents; the word the appears 4 times and occurs in all 10. Compute both TF-IDF values (natural log).

Quick quiz

6Further reading

• scikit-learn documentation. Text feature extraction. (CountVectorizer, n-grams, and TfidfVectorizer in practice.)
• Lavin (2019). Analyzing Documents with TF-IDF. Programming Historian. (A gentle walkthrough of TF-IDF on real documents.)
• Voita. NLP Course: Text Classification. (Bag-of-words features in the context of text classification.)