Neural Networks

Logistic regression finds linear weights that map a feature vector $\mathbf x$ to a label $y$. Its decision boundary is always a straight line, or a flat hyperplane in higher dimensions. That works when the two classes sit on opposite sides of some line. But what if one class forms a ring around the other, or the two classes spiral into each other? No choice of weights can draw a straight line between them.

A neural network fixes this by stacking many simple units, each one a lot like logistic regression, with a nonlinear function between them. The result can learn nonlinear decision boundaries of almost any shape. This reading builds a neural network from a single neuron up, shows why the nonlinearity is essential, and explains how the whole network is trained with back-propagation.

1From biological to artificial neurons

Neural networks borrow a loose idea from biology. A biological neuron receives signals from other neurons through its dendrites and becomes active, sending a signal of its own along its axon, when the total input it receives is strong enough.

An artificial neuron mimics this in three steps. It takes inputs $\mathbf x=[x_1,\dots,x_d]$, multiplies each by a weight $w_i$, adds a bias $b$, and passes the sum through an activation function $\varphi$:

Compare this with logistic regression, $\tilde y=\sigma\big(\sum_i w_ix_i+b\big)$. They're the same formula. Logistic regression is a single neuron whose activation function is the sigmoid. A neural network is what you get when you connect many neurons together.

2The multilayer perceptron

Put several neurons side by side and they form a layer. Each neuron in the first layer looks at the whole input $\mathbf x$ with its own weights and bias. For example, the first two neurons of layer 1 compute

where $w_{j,i}^{(1)}$ is the weight from input $x_i$ to neuron $j$, and the superscript $(1)$ marks the layer. Stacking the weight vectors as the rows of a matrix $\mathbf W^{(1)}$ and the biases into a vector $\mathbf b^{(1)}$, the whole layer is one line:

with $\varphi$ applied to each entry. The outputs $\mathbf h^{(1)}$, called hidden units, become the input of the next layer:

Finally, an output layer turns the last hidden layer into a prediction. For binary classification it is a single sigmoid neuron:

This architecture is the multilayer perceptron (MLP). You'll also see it called a feedforward neural network (information flows in one direction, from input to output, with no loops), a fully connected network, or a dense network (every neuron connects to every neuron in the previous layer). The last hidden layer is effectively a learned feature representation, and the output layer is just logistic regression on top of it.

3Why the nonlinearity is essential

What makes a neural network more powerful than logistic regression? Not the number of layers alone. Suppose we removed the activation function, so each layer just computed $\mathbf W\mathbf x+\mathbf b$. Then two layers would give

which is again a single linear layer. Any stack of linear layers collapses to one, and the network would be no better than logistic regression. The activation function introduces nonlinearity: each layer applies a nonlinear transform, so the network can bend, fold, and stretch the input space until the classes become separable by the final linear output layer.

Which activation should you use? For hidden layers, ReLU, $\max(0,z)$, is the most common default: it's cheap and its gradient doesn't shrink for positive inputs. Sigmoid and tanh saturate (their derivatives approach 0 for large $|z|$), which slows the training of deep networks. For the output layer, use sigmoid for binary classification and softmax for multiclass classification, so the outputs are probabilities.

4Training: loss, gradients, and back-propagation

Training a neural network uses the same recipe as logistic regression. The parameters are all the weights and biases,

and we minimize the average cross-entropy loss over the training set,

with gradient descent: randomly initialize, compute the loss, compute the gradient, and update every parameter, for example $\mathbf W^{(1)}\leftarrow\mathbf W^{(1)}-\eta\,\partial\mathcal L/\partial\mathbf W^{(1)}$, and likewise for every other $\mathbf W^{(l)}$ and $\mathbf b^{(l)}$.

The new question is how to compute the gradients for the weights deep inside the network. The answer is back-propagation, which is simply the chain rule applied from the output backward:

1. Forward pass: compute $\mathbf h^{(1)},\mathbf h^{(2)},\mathbf h^{(3)},\tilde y$ and the loss, storing every intermediate value.
2. Backward pass: start from $\partial\mathcal L/\partial\tilde y$ at the output. Then, layer by layer going backward, use the chain rule: $$\frac{\partial\mathcal L}{\partial\mathbf h^{(3)}}=\frac{\partial\mathcal L}{\partial\tilde y}\cdot\frac{\partial\tilde y}{\partial\mathbf h^{(3)}},\qquad \frac{\partial\mathcal L}{\partial\mathbf h^{(2)}}=\frac{\partial\mathcal L}{\partial\mathbf h^{(3)}}\cdot\frac{\partial\mathbf h^{(3)}}{\partial\mathbf h^{(2)}},\ \dots$$ At each layer, the gradient for that layer's parameters is read off the same way, e.g. $\frac{\partial\mathcal L}{\partial\mathbf W^{(3)}}=\frac{\partial\mathcal L}{\partial\mathbf h^{(3)}}\cdot\frac{\partial\mathbf h^{(3)}}{\partial\mathbf W^{(3)}}$.

The key efficiency is reuse: the gradient with respect to $\mathbf h^{(3)}$ is computed once and reused for $\mathbf W^{(3)}$, $\mathbf b^{(3)}$, and $\mathbf h^{(2)}$, and so on down the network. In practice, deep learning libraries such as PyTorch do all of this automatically (“autograd”); you write the forward pass and the library computes every gradient.

The matrix version can look intimidating, but the idea is the same with plain numbers. Here's a tiny network with one input, one hidden neuron, and one output:

5From binary to multiclass classification

For $C$ classes, give the output layer $C$ units and apply the softmax function to turn their scores into a probability distribution:

and train with the multiclass cross-entropy loss, where $y_c=1$ for the correct class and 0 otherwise:

Everything else, from the hidden layers to back-propagation, stays the same.

6Neural networks for text classification

In the text classification framework, the neural network replaces logistic regression as the classifier, on top of any feature representation:

With bag-of-words input, $\mathbf x$ has one dimension per vocabulary word. Each hidden unit computes a weighted combination of words and then applies a nonlinearity, so it can respond to combinations that no single word weight can express, for example firing when not and good appear together. Two practical points:

• Size. With a 10,000-word vocabulary and 100 hidden units, the first layer alone has about a million parameters (Figure 2). Bigger networks can fit more complex patterns but are also more prone to overfitting, so the validation set becomes essential for choosing the depth, width, and learning rate.
• Hyperparameters. The number of layers, the number of hidden units per layer, the activation function, and the learning rate are all design choices made before training.

7Check your understanding

Exercise 1: one neuron

A neuron has weights $\mathbf w=[1,-1,2]$, bias $b=-1$, and ReLU activation. Compute its output for $\mathbf x=[2,3,1]$ and for $\mathbf x=[3,0,1]$.

Exercise 2: counting parameters

An MLP has 3 inputs, hidden layers of 4, 3, and 4 units, and 1 output (the network on the slides). How many parameters does it have?

Exercise 3: back-propagation

In the tiny network of Figure 4 with ReLU, let $x=1$, $w_1=0.5$, $b_1=0$, $w_2=2$, $b_2=-0.5$, and $y=1$. Compute $\tilde y$, then $\partial\mathcal L/\partial w_2$ and $\partial\mathcal L/\partial w_1$.

Quick quiz

8Further reading

• Jurafsky & Martin. Speech and Language Processing (3rd ed. draft), Chapter 6: Neural Networks.
• 3Blue1Brown. Neural Networks. (An excellent visual series, including what back-propagation is really doing.)
• Smilkov & Carter. TensorFlow Playground. (Train small networks in the browser and watch the decision boundary change.)