CSCE 471 Natural Language Processing · Spring 2027

Logistic Regression

Kuan-Hao Huang · Course Website
Note: This page was created with the help of AI as a supplementary reading. It may contain errors or simplifications. The lecture slides are the primary reference for this course; please refer to them first.

Once a text has been turned into a feature vector $\mathbf x$, we need a classifier: a rule that maps $\mathbf x$ to a label $y$. Suppose $\mathbf x$ records two numbers about a product review, how many positive words it contains and how many negative words. Intuitively, positive words should push the prediction toward Positive and negative words toward Negative, each by some amount. Choosing “some amount” automatically, from labeled data, is exactly what logistic regression does.

Logistic regression is the simplest learned classifier, and nearly every idea in it (a score, a probability, a loss, gradients, gradient descent) carries over directly to neural networks. This reading builds it step by step, with interactive figures you can use to check each step.

After reading this, you should be able to compute a logistic regression prediction from weights, bias, and features · explain the sigmoid function and the linear decision boundary · compute the cross-entropy loss and explain why it is a good measure · state the gradient of the loss and explain it intuitively · carry out a step of gradient descent by hand · extend the model to more than two classes with softmax · explain the main limitation of a linear classifier.

1The model: a weighted score turned into a probability

Start with binary classification. The input is a feature vector $\mathbf x=[x_1,x_2,\dots,x_d]$ and the label is $y\in\{0,1\}$ (for example, 1 = Positive). Logistic regression has learnable parameters: a weight vector $\mathbf w=[w_1,\dots,w_d]$, one weight per feature, and a bias $b$. It makes a prediction in three steps.

  1. Score. Compute a weighted sum of the features: $$z=\mathbf w\cdot\mathbf x+b=w_1x_1+w_2x_2+\dots+w_dx_d+b.$$ A positive weight means “this feature is evidence for class 1”; a negative weight means evidence for class 0.
  2. Probability. Squash the score into the range (0, 1) with the sigmoid function: $$\tilde y=P(y=1\mid\mathbf x)=\sigma(z),\qquad \sigma(t)=\frac{1}{1+e^{-t}}.$$ Large positive $z$ gives a probability near 1, large negative $z$ near 0, and $z=0$ gives exactly 0.5.
  3. Prediction. Predict 1 if $\tilde y\ge0.5$ and 0 otherwise.

Since $\sigma(z)\ge0.5$ exactly when $z\ge0$, the two classes are separated where $\mathbf w\cdot\mathbf x+b=0$. This set is the decision boundary: a straight line when there are two features, a plane with three, and a flat hyperplane in general. That's why logistic regression is called a linear classifier.

w₁ (positive words)0.5 w₂ (negative words)−0.5 bias b0.0
Figure 1 (interactive). Logistic regression on 17 toy reviews with two features: $x_1$ = number of positive words, $x_2$ = number of negative words. Red points are positive reviews ($y=1$), blue points negative ($y=0$). The shading shows $P(y=1\mid\mathbf x)$ and the dark line is the decision boundary $\mathbf w\cdot\mathbf x+b=0$. Click a point to see its score on the sigmoid curve.

2Measuring goodness: the cross-entropy loss

Different weights give different predictions. To find the best ones automatically, we need a number that says how bad a set of parameters is: a loss function. For one example with true label $y$ and predicted probability $\tilde y$, logistic regression uses the cross-entropy loss (log is the natural logarithm):

$$\mathcal L_{CE}(y,\tilde y)=-\big[y\log\tilde y+(1-y)\log(1-\tilde y)\big].$$

Only one of the two terms is ever active. If $y=1$ the loss is $-\log\tilde y$; if $y=0$ it is $-\log(1-\tilde y)$. In words: the loss is $-\log$ of the probability the model gave to the correct answer.

predicted ỹ = P(y = 1 | x)0.90
Figure 2 (interactive). Cross-entropy loss as a function of the predicted probability, for a positive example ($y=1$, red) and a negative example ($y=0$, blue). The loss is near 0 when the model is confidently right and grows without bound when it is confidently wrong.
true ypredicted ỹlossverdict
10.9−log 0.9 ≈ 0.105confident and right: small loss
10.1−log 0.1 ≈ 2.303confident and wrong: large loss
00.9−log 0.1 ≈ 2.303confident and wrong: large loss
00.1−log 0.9 ≈ 0.105confident and right: small loss
Table 1. The lower the loss, the more accurate the output probability.

For the whole training set $\mathcal D_{\text{train}}=\{(\mathbf x_1,y_1),\dots,(\mathbf x_m,y_m)\}$, average over the examples:

$$\mathcal L_{\text{total}}=\frac1m\sum_{i=1}^{m}\mathcal L_{CE}(y_i,\tilde y_i)=-\frac1m\sum_{i=1}^{m}\big[y_i\log\tilde y_i+(1-y_i)\log(1-\tilde y_i)\big].$$

Training means solving the optimization objective

$$[\mathbf w^*;b^*]=\theta^*=\arg\min_\theta\mathcal L_{\text{total}},$$

where $\theta$ collects all the parameters. Try it in Figure 1: the readout shows $\mathcal L_{\text{total}}$, and you can lower it by hand. The rest of this reading shows how to do this automatically.

3Which way is downhill? The gradient

Training uses an iterative optimization loop: randomly initialize the parameters, evaluate how good they are (the loss), find a good direction to change them, update, and repeat.

The gradient $\nabla_\theta\mathcal L_{\text{total}}$ is the vector of partial derivatives of the loss with respect to each parameter. It points in the direction in which the loss increases fastest, so the negative gradient $-\nabla_\theta\mathcal L_{\text{total}}$ is a good direction to reduce it. For logistic regression the gradient has a remarkably simple form:

$$\frac{\partial\mathcal L_{\text{total}}}{\partial\mathbf w}=\frac1m\sum_{i=1}^{m}(\tilde y_i-y_i)\,\mathbf x_i,\qquad \frac{\partial\mathcal L_{\text{total}}}{\partial b}=\frac1m\sum_{i=1}^{m}(\tilde y_i-y_i).$$
Intuition: error × input. For each example, $\tilde y_i-y_i$ is the prediction error. If a positive example ($y_i=1$) gets $\tilde y_i=0.3$, the error is $-0.7$, so the gradient points toward decreasing $\mathbf w\cdot\mathbf x_i$, and stepping in the negative-gradient direction increases the weights of the features that example has. A correctly and confidently classified example has error near 0 and barely affects the update. Each weight $w_j$ is moved in proportion to how much its feature $x_{i,j}$ was present when the model made errors.
Optional: deriving the gradient

Write $\tilde y_i=\sigma(z_i)$ with $z_i=\mathbf w\cdot\mathbf x_i+b$, so $\partial z_i/\partial w_j=x_{i,j}$. The key fact about the sigmoid is

$$\sigma'(z)=\sigma(z)\big(1-\sigma(z)\big).$$

(Proof: $\sigma(z)=(1+e^{-z})^{-1}$, so $\sigma'(z)=\frac{e^{-z}}{(1+e^{-z})^2}=\frac{1}{1+e^{-z}}\cdot\frac{e^{-z}}{1+e^{-z}}=\sigma(z)(1-\sigma(z))$.)

Using the chain rule on each log term:

$$\frac{\partial\log\sigma(z_i)}{\partial w_j}=\frac{1}{\sigma(z_i)}\cdot\sigma(z_i)\big(1-\sigma(z_i)\big)\cdot x_{i,j}=\big(1-\sigma(z_i)\big)x_{i,j},$$ $$\frac{\partial\log\big(1-\sigma(z_i)\big)}{\partial w_j}=\frac{1}{1-\sigma(z_i)}\cdot\big(-\sigma(z_i)(1-\sigma(z_i))\big)\cdot x_{i,j}=-\sigma(z_i)\,x_{i,j}.$$

Plug both into the loss:

$$\frac{\partial\mathcal L_{\text{total}}}{\partial w_j}=-\frac1m\sum_i\Big[y_i\big(1-\sigma(z_i)\big)x_{i,j}+(1-y_i)\big(-\sigma(z_i)\big)x_{i,j}\Big]=-\frac1m\sum_i\big(y_i-\sigma(z_i)\big)x_{i,j}=\frac1m\sum_i(\tilde y_i-y_i)\,x_{i,j}.$$

The derivation for $b$ is identical with $x_{i,j}$ replaced by 1.

4Gradient descent

Gradient descent repeatedly takes a small step in the negative-gradient direction:

$$\mathbf w^{(t+1)}=\mathbf w^{(t)}-\eta\,\nabla_{\mathbf w}\mathcal L_{\text{total}},\qquad b^{(t+1)}=b^{(t)}-\eta\,\nabla_b\mathcal L_{\text{total}}.$$

The step size (or learning rate) $\eta$ controls how far each step goes. Too small, and training crawls; too large, and the updates overshoot and can bounce around or diverge. The cross-entropy loss of logistic regression is convex (bowl-shaped), so with a suitable $\eta$ gradient descent reaches the best parameters from any starting point.

Learning rate η:
Figure 3 (interactive). Gradient descent on the data of Figure 1, starting from $\mathbf w=\mathbf 0$, $b=0$. Left: the current decision boundary. Right: $\mathcal L_{\text{total}}$ after each step. Compare learning rates: 0.05 is slow, 0.5 converges smoothly, and 5 takes large, jumpy steps.

One step by hand

Take two training examples, $\mathbf x_1=[2,0]$ with $y_1=1$ and $\mathbf x_2=[0,2]$ with $y_2=0$, start from $\mathbf w=[0,0]$, $b=0$, and use $\eta=0.5$.

  1. Both scores are 0, so $\tilde y_1=\tilde y_2=\sigma(0)=0.5$, and the loss is $-\log0.5\approx0.693$.
  2. Errors: $\tilde y_1-y_1=-0.5$ and $\tilde y_2-y_2=+0.5$.
  3. Gradient: $\nabla_{\mathbf w}=\frac12\big[(-0.5)[2,0]+(0.5)[0,2]\big]=[-0.5,\,0.5]$ and $\nabla_b=\frac12(-0.5+0.5)=0$.
  4. Update: $\mathbf w=[0,0]-0.5\,[-0.5,0.5]=[0.25,\,-0.25]$, $b=0$.
  5. Now $\tilde y_1=\sigma(0.5)\approx0.622$ and $\tilde y_2=\sigma(-0.5)\approx0.378$, and the loss has dropped to about 0.474.
In practice: mini-batches. Computing the gradient over all $m$ examples at every step is expensive for large datasets. Stochastic or mini-batch gradient descent estimates the gradient from a small random subset of examples at each step. The steps are noisier but much cheaper, and this is how almost all models are trained.

5From binary to multiclass: softmax

With $C$ classes, $y\in\{1,\dots,C\}$, give each class $c$ its own weight vector $\mathbf w_c$ and bias $b_c$, and compute one score per class:

$$z_c=\mathbf w_c\cdot\mathbf x+b_c.$$

The softmax function turns the $C$ scores into a probability distribution, which is positive and sums to 1:

$$P(y=c\mid\mathbf x)=\text{softmax}(z_c)=\frac{e^{z_c}}{\sum_{c'=1}^{C}e^{z_{c'}}},\qquad \text{Prediction}=\arg\max_c P(y=c\mid\mathbf x).$$

The loss is the multiclass cross-entropy. Writing the true label as a one-hot vector ($y_c=1$ for the correct class and 0 for the others),

$$\mathcal L_{CE}(y,\tilde y)=-\sum_{c=1}^{C}y_c\log P(y=c\mid\mathbf x),$$

which again is just $-\log$ of the probability assigned to the correct class. The binary case is the special case $C=2$, and the gradient keeps the same “error × input” form: $\nabla_{\mathbf w_c}=\frac1m\sum_i\big(P(y_i=c\mid\mathbf x_i)-y_{i,c}\big)\mathbf x_i$.

True class:
Figure 4 (interactive). Softmax over four class scores. Drag a score $z_c$ to see how all four probabilities change together, and how the cross-entropy loss depends only on the probability of the true class.

6The limit of linear weights

Logistic regression finds linear weights that map $\mathbf x$ to $y$. Its decision boundary is always a straight line (or a flat hyperplane). That's often enough for text with rich features: with tens of thousands of bag-of-words dimensions, a hyperplane can separate a lot. But when the classes are arranged in a curved pattern, such as two interleaved spirals or one class surrounding the other, no straight line can separate them, however the weights are chosen.

There are two ways out: design better features, so that the data becomes linearly separable in the new feature space, or use a more powerful model that learns nonlinear decision boundaries by itself. Much of modern NLP takes the second route, while keeping the same ingredients you've seen here: a score, a softmax, a cross-entropy loss, and gradient descent.

7Check your understanding

Exercise 1: a prediction

A model has $\mathbf w=[1,-2]$ and $b=0.5$. For $\mathbf x=[2,1]$, compute $z$, $\tilde y$, the prediction, and the loss if the true label is $y=1$ and if it is $y=0$.

$z=1\cdot2+(-2)\cdot1+0.5=0.5$, $\tilde y=\sigma(0.5)\approx0.622$, so the prediction is 1.
If $y=1$: loss $=-\log0.622\approx0.474$. If $y=0$: loss $=-\log(1-0.622)=-\log0.378\approx0.974$.

Exercise 2: one gradient descent step

With a single training example $\mathbf x=[1,2]$, $y=1$, starting from $\mathbf w=[0,0]$, $b=0$, and $\eta=1$, compute the gradient and the updated parameters.

$\tilde y=\sigma(0)=0.5$, error $=0.5-1=-0.5$. Gradient: $\nabla_{\mathbf w}=-0.5\,[1,2]=[-0.5,-1]$, $\nabla_b=-0.5$.
Update: $\mathbf w=[0.5,\,1]$, $b=0.5$. Now $z=0.5+2+0.5=3$ and $\tilde y=\sigma(3)\approx0.953$: much closer to the correct label.

Exercise 3: softmax

Three classes have scores $z=[2,1,0]$ and the true class is the first. Compute the probabilities and the loss.

$e^2\approx7.389$, $e^1\approx2.718$, $e^0=1$; the sum is $11.107$. Probabilities $\approx[0.665,\,0.245,\,0.090]$. Loss $=-\log0.665\approx0.408$.

Quick quiz

8Further reading