ROB 498 / 599 · Fall 2026 · Companion to L05 and L06

Neural networks, from one line up

L05 built a network from one neuron and trained it by hand, and the first half of L06 turned it into a convolutional network. This tutorial covers the same ground slowly. Every example is worked in numbers, the drawings are the ones from the slides, and four figures move when you drag them. Nothing here assumes you have trained a network before.

Covers L05, and the convolution half of L06 Read with module notes B1–B4 Worked every example in numbers

How to read this

Nine steps, in order. Each one adds a single idea to the network built in the step before, and each is worked in numbers you can check on paper. Together they cover L05 and the convolution half of L06.

If you have trained a network before, read the straight answers, then step 3 (backpropagation worked by hand) and step 7 (why training goes unstable). Those are the two most people can use rather than recite.

The drawings are the ones from the slides. Four of them move: drag a slider and the numbers beside it change. The module notes B1 to B4 are the same material in a third order, with the self-checks.

Step 1–3
one neuron, a loss, and the gradient that trains it

Step 4–5
layers, the bend that makes them worth stacking, and classification

Step 6–7
how training is actually run, and how it fails

Step 8–9
convolution, and the shape of a depth network

Straight answers

The questions that came up in class and after it, answered in a line or two each. Every one is unpacked later.

Is a neural network just curve fitting?

Yes. It is a function with a very large number of unknown numbers, fitted to examples by making a loss small. What makes it interesting is that the same fitting procedure works whether the function has two unknowns or two billion, and that nobody has to design the features by hand.

Where do the weights come from before training?

They are drawn at random, scaled to the width of the layer (step 7). Starting at zero would make every neuron in a layer compute the same thing and receive the same gradient, so they would never become different from one another.

What is a gradient, concretely?

One number per weight: how much the loss would change if that weight went up by a tiny amount. Stack those numbers into a vector and you have the gradient. Training subtracts a small multiple of it from the weights.

Why not solve for the best weights directly?

For a linear model with squared error you can, and that is least squares (L04). Once activations bend the function, there is no formula for the minimum, and there are many minima. Gradient descent is what is left.

Why does a network need an activation function?

Without one, a stack of layers collapses into a single linear layer: the composition of two matrix multiplications is a matrix multiplication (step 4). Every bend in the fitted function comes from an activation.

Step, batch, epoch: which is which?

A step is one update of the weights, computed from one mini-batch. An epoch is one pass through the whole training set, so 60,000 examples in batches of 100 is 600 steps per epoch.

Why keep validation and test separate?

You choose the learning rate, the model size and the stopping point by looking at validation loss, so those choices are fitted to it. The test set answers a different question: how well does the finished model do on data that took part in no decision at all.

Do I need to be able to derive backpropagation?

You need to know what it computes and why it is cheap: one backward sweep gives the derivative of the loss with respect to every weight, at about the cost of one forward pass. Step 3 derives it once on a two-layer network so that the framework's loss.backward() is not a black box.

Why convolution instead of a fully connected layer on an image?

Parameters and structure. One fully connected layer from a $224 \times 224$ colour image to 1000 units holds 150,528,000 weights; a $3 \times 3$ convolution with 64 output channels holds 1,792 and can be applied anywhere in the image (step 8).

Batch norm or layer norm?

Batch norm averages over the examples in the batch, so it degrades when the batch is small, which is the normal case in 3D work. Layer norm and group norm average within one example and behave the same at batch size 1 (step 7).

1 One neuron is a line

Start with one input, one output and two unknown numbers:

\[ \hat{y} = wx + b \]

$w$ is the weight and $b$ the bias. Computing $\hat{y}$ from $x$ with the current $w$ and $b$ is the forward pass. That is the whole of a neuron. Everything that follows repeats it, bends it, and stacks it.

One neuron: an input, a weight, a bias, and the sum they produce. Rosenblatt's 1958 perceptron was this, followed by a threshold.

With several inputs the weights become a vector and the neuron computes a weighted sum:

\[ \hat{y} = w_1x_1 + w_2x_2 + w_3x_3 + b = \mathbf{w}^\top\mathbf{x} + b \]

One weight per input, one bias per neuron. A $28 \times 28$ image flattened into a vector is 784 inputs, so a single neuron reading it holds 784 weights and one bias.

The name is borrowed from biology, where a cell sums what arrives at its dendrites and fires if the total is large enough. The resemblance ends about there, and nothing below depends on it.

2 A loss, and the way downhill

Training needs a single number that says how wrong the current weights are. For a quantity such as depth, use the mean squared error over the $N$ training pairs:

\[ L(w, b) = \frac{1}{N}\sum_{i=1}^{N}\big(\hat{y}_i - y_i\big)^2 = \frac{1}{N}\sum_{i=1}^{N}\big(wx_i + b - y_i\big)^2 \]

Squaring does two things: it makes a miss count the same whether the prediction is above or below, and it makes one large miss cost more than several small ones. The data are fixed, so $L$ is a function of $w$ and $b$ alone. Training means making it small.

Six points and one line. The dashed segments are the residuals, and the loss is the mean of their squares.

Drag it yourself

The line below starts at $w = 0.8$, $b = 0.4$, where the mean squared error is 0.61. Move the sliders and watch it fall. The least-squares answer for these six points is $w = 0.63$, $b = 0.92$, with a loss of 0.54: you will not get below that by hand, and neither will gradient descent.

Try these. $\eta = 0.05$, 20 steps: the line settles onto the least-squares answer and stops moving, because the gradient there is zero. $\eta = 0.3$: each step overshoots and the loss climbs instead of falling. The largest stable learning rate depends on the data, not on the network, which is why it has to be tuned.

Gradient descent

The derivative of the loss with respect to a weight says which way that weight should move. Positive slope means the loss grows as the weight grows, so decrease it. The learning rate $\eta$ sets how far to go:

\[ w \leftarrow w - \eta\,\frac{\partial L}{\partial w}, \qquad b \leftarrow b - \eta\,\frac{\partial L}{\partial b} \]

The derivative at the current weight says which way is uphill and how steep it is. The step goes the other way, scaled by $\eta$.

With two parameters the loss is a surface and the pair of derivatives points along its steepest slope. With ten million parameters nothing changes conceptually: each weight still moves against its own partial derivative.

The same bowl and the same starting point, at three learning rates. Too small wastes time; too large bounces; above a threshold it diverges.

One training step, in numbers

One example, $x = 2.1$ with target $y = 4$. Start from $w = 1$, $b = 0$ and use $\eta = 0.01$. Forward pass first:

\[ \hat{y} = (1)(2.1) + 0 = 2.1, \qquad L = (\hat{y} - y)^2 = (2.1 - 4)^2 = 3.61 \]

Now the two derivatives. The loss depends on $w$ only through $\hat{y}$, so the chain rule applies:

\[ \frac{\partial L}{\partial \hat{y}} = 2(\hat{y} - y) = -3.8, \qquad \frac{\partial \hat{y}}{\partial w} = x = 2.1, \qquad \frac{\partial \hat{y}}{\partial b} = 1 \]

\[ \frac{\partial L}{\partial w} = \frac{\partial L}{\partial \hat{y}}\,\frac{\partial \hat{y}}{\partial w} = -7.98, \qquad \frac{\partial L}{\partial b} = -3.8 \]

Both are negative, which says the prediction is too small and both parameters should grow. Take the step:

\[ w' = 1 - (0.01)(-7.98) = 1.0798, \qquad b' = 0 - (0.01)(-3.8) = 0.038 \]

\[ \hat{y}' = 2.306, \qquad L' = 2.87 \quad (\text{was } 3.61) \]

One more step of the same size gives $w = 1.151$, $b = 0.072$, $\hat{y} = 2.489$ and a loss of 2.28. The steps shrink as the loss flattens out, because the gradient is shrinking too. That is the whole of training, repeated for every example and every weight.

Check yourself: what happens with $\eta = 0.5$ here?

$w' = 1 - 0.5(-7.98) = 4.99$, so $\hat{y}' = 10.5$ against a target of 4, and the loss goes from 3.61 to about 42. One step has overshot the minimum and landed further away than it started. Halve it again and it diverges completely.

3 Backpropagation is the chain rule, organised

With two parameters you can write the derivatives out by hand. With two million you cannot, and you do not have to: the same chain rule, applied on a graph, gives every derivative in one sweep.

Draw the forward computation as a graph. Each node takes its inputs, computes a value, and keeps it. Then walk backwards from the loss, multiplying by each node's local derivative as you pass through it. The numbers on the graph below are the ones from the step above.

The worked step as a graph. Forward values sit above each node, gradients below. The multiply node sends back the incoming gradient times the other input, which is where $x$ enters $\partial L/\partial w$.

Two properties make this work at scale. A node needs only the values at its own inputs, so nothing has to be recomputed. And the backward sweep costs about what the forward pass costs, whatever the number of parameters. Storing the forward values is the price: during training it is the activations, not the weights, that fill GPU memory.

Worked: a two-layer network, forward and backward

This is the example to do on paper once. Two inputs, two hidden units with ReLU, one output, squared error against a single target. The numbers are chosen so that one hidden unit is active and the other is not.

\[ \mathbf{x} = \begin{bmatrix}1\\ 1\end{bmatrix}, \quad \mathbf{W}_1 = \begin{bmatrix}0.5 & -0.3\\ -0.8 & 0.2\end{bmatrix}, \quad \mathbf{b}_1 = \begin{bmatrix}0.1\\ -0.4\end{bmatrix}, \quad \mathbf{w}_2 = \begin{bmatrix}1.2\\ -0.7\end{bmatrix}, \quad b_2 = 0.3, \quad y = 1 \]

Forward. The pre-activations, then the ReLU, then the output:

\[ \mathbf{z}_1 = \mathbf{W}_1\mathbf{x} + \mathbf{b}_1 = \begin{bmatrix}0.3\\ -1\end{bmatrix}, \qquad \mathbf{h} = \max(0, \mathbf{z}_1) = \begin{bmatrix}0.3\\ 0\end{bmatrix}, \qquad \hat{y} = \mathbf{w}_2^\top\mathbf{h} + b_2 = 0.66 \]

\[ L = (\hat{y} - y)^2 = 0.12 \]

Backward. Start at the loss and work left. Each line is one node's local derivative times what arrived from the right:

Node	Local derivative	Gradient
loss	$\partial L/\partial \hat{y} = 2(\hat{y} - y)$	-0.68
output weights	$\partial \hat{y}/\partial \mathbf{w}_2 = \mathbf{h}$	-0.2, 0
output bias	$\partial \hat{y}/\partial b_2 = 1$	-0.68
hidden activations	$\partial \hat{y}/\partial \mathbf{h} = \mathbf{w}_2$	-0.82, 0.48
through the ReLU	$1$ where $z > 0$, else $0$	-0.82, 0
first-layer weights	$\partial z_{1,i}/\partial W_{1,ij} = x_j$	-0.82, -0.82; 0, 0

Read the last two rows together. The second hidden unit has $z = -1$, so the ReLU passed a zero backwards and its whole row of first-layer weights gets a gradient of zero. It learns nothing from this example. A unit whose sum is negative for every example never learns again, which is the dead ReLU of step 7.

Every framework does exactly this. When you call loss.backward(), PyTorch walks the graph it recorded during the forward pass and fills in the same numbers.

4 Layers, and the bend that makes them worth stacking

Put two neurons in a row, with nothing between them:

\[ w_2(w_1x + b_1) + b_2 = (w_2w_1)\,x + (w_2b_1 + b_2) = w'x + b' \]

Still one weight and one bias. The same algebra holds with matrices, so a hundred stacked linear layers compute exactly what one linear layer computes. Depth buys nothing until something nonlinear sits between the layers.

An activation function $\sigma$ is applied to every neuron's sum:

\[ \hat{y} = \sigma\big(\mathbf{w}^\top\mathbf{x} + b\big) \]

A neuron with three inputs and an activation. Without the activation, stacking these buys nothing.

Each ReLU unit contributes one bend, at the input where its sum crosses zero. Add units and the fitted function becomes a piecewise-linear curve with as many bends as you paid for:

Three ReLU units and their sum. Each unit contributes one bend, at the input where its own sum crosses zero.

With enough hidden units, a single hidden layer can approximate any continuous function on a bounded region as closely as you like. That is the universal approximation result, and it is worth knowing exactly what it does not say: it does not say gradient descent will find those weights, and it does not say how many units you need. In practice a deeper network needs far fewer units than a single wide layer for the same accuracy.

Three activations, and how to pick

ReLU, tanh and sigmoid, with their derivatives dashed. The derivative is what backpropagation multiplies by on the way back.

Name	Formula	Range	Largest slope	Use it for
ReLU	$\max(0, x)$	$[0, \infty)$	1	hidden layers, by default
tanh	$(e^{x} - e^{-x})/(e^{x} + e^{-x})$	$(-1, 1)$	1	hidden layers in older networks
sigmoid	$1/(1 + e^{-x})$	$(0, 1)$	0.25	a single yes-or-no output

The dashed curves in the figure are the derivatives, which is what backpropagation multiplies by on the way back. The sigmoid's never exceeds 0.25, so ten sigmoid layers multiply the gradient by at most $0.25^{10} \approx 9.5 \times 10^{-7}$ before it reaches the first layer. That is the vanishing gradient of step 7, and it is why ReLU replaced sigmoid in hidden layers.

Four questions decide the choice: does the output need to be bounded, is the derivative well behaved, how much does it cost to evaluate, and what does the task need at the output. Regression wants no activation at the output, a yes-or-no question wants a sigmoid, and one of many classes wants a softmax, which is step 5.

A network, and its size

A fully connected network: every neuron reads every output of the layer before it. Hidden layers are the ones between input and output.

Each layer is a weight matrix, a bias vector and an activation:

\[ \mathbf{h}_1 = \sigma(\mathbf{W}_1\mathbf{x} + \mathbf{b}_1), \qquad \mathbf{h}_2 = \sigma(\mathbf{W}_2\mathbf{h}_1 + \mathbf{b}_2), \qquad \hat{\mathbf{y}} = \mathbf{W}_3\mathbf{h}_2 + \mathbf{b}_3 \]

Row $i$ of $\mathbf{W}_1$ is the weight vector of neuron $i$ in the first hidden layer; the matrix is bookkeeping for many neurons at once. A layer with $m$ inputs and $n$ outputs holds $mn + n$ numbers, so the digit classifier from the lecture, $784 \to 16 \to 16 \to 10$, holds $12{,}560 + 272 + 170 = 13{,}002$. Depth Anything's large model holds about 335 million.

5 Classification: scores to probabilities

For a category rather than a quantity, the last layer gives one score $z_k$ per class. Scores can be any real number. Softmax turns them into probabilities by exponentiating and dividing by the total:

\[ p_k = \frac{e^{z_k}}{\sum_j e^{z_j}} \]

Exponentiating makes every entry positive; dividing makes them sum to one. The order is preserved, so the largest score stays the most likely class. With scores $2$, $1$ and $-1$ for flower, person and tree:

\[ e^{z} = 7.39,\ 2.72,\ 0.37 \quad\text{summing to } 10.48, \qquad p = 0.705,\ 0.259,\ 0.035 \]

Scores in, probabilities out. Softmax keeps the order and makes the three numbers sum to one.

Cross-entropy

The loss looks only at the probability given to the correct class:

\[ L = -\log p_{\text{correct}} \]

If the picture is a flower, $L = -\log 0.705 = 0.35$. If the same scores were produced for a person, $L = -\log 0.259 = 1.35$, and for a tree, $3.35$. Confident and right costs almost nothing; confident and wrong costs a great deal, which is what pushes the network away from stating nonsense firmly.

$-\log p$ against $p$. A probability near 1 costs almost nothing; a probability near 0 costs a great deal.

Why the two go together

Softmax and cross-entropy are always paired, and the reason is the gradient. Write $y_k$ for the one-hot target, 1 on the correct class and 0 elsewhere. Then the derivative of the loss with respect to the scores is

\[ \frac{\partial L}{\partial z_k} = p_k - y_k \]

No exponentials, no division, nothing that can overflow: just the predicted probability minus the target. For the numbers above, with flower as the correct answer, the gradient on the scores is $0.705 - 1 = -0.295$, $0.259$, $0.035$. The correct class is pushed up, every other class is pushed down in proportion to how much it claimed, and the size of the push is exactly how wrong the probability was. Frameworks fuse the two into one operation (CrossEntropyLoss takes scores, not probabilities) both for this stability and for the speed.

Drag the scores

Try these. All three equal: every probability is $1/3$ and the loss is $-\log(1/3) = 1.10$, which is what an untrained network gives. Raise the correct score: the loss falls towards zero and so does the gradient, so a confident correct answer stops pulling. Raise a wrong score to 6: the loss grows without bound while the gradient on that class approaches 1, its largest possible value.

Task	Robot example	Output layer	Loss
Regression	depth at every pixel	none, or ReLU to keep it positive	squared or absolute error
Yes or no	is this pixel free space?	sigmoid	binary cross-entropy
One of many	which of 20 object classes?	softmax	cross-entropy

Semantic segmentation is the third row applied at every pixel, and a depth network is the first row applied at every pixel. The hidden layers can be identical; the head and the loss are what differ.

6 Training in practice

Computing the gradient over the whole training set for every step is too slow, and it is not needed. Average it over a random mini-batch instead. That is stochastic gradient descent: each step points roughly downhill rather than exactly downhill, and the noise costs little.

The training set split into mini-batches. One pass through all of them is an epoch; each batch gives one step.

One pass through the training data is an epoch. Sixty thousand images in batches of 100 is 600 steps per epoch. Batch size is capped by memory, and 3D data fills memory quickly: batches of 2 to 8 scenes are normal, which matters in step 7.

Three datasets, three jobs

Training, validation and test. Only the first updates weights, and only the last is untouched by every choice you made.

Training data updates the weights. Validation data never updates the weights, but you look at it during training to choose the learning rate, the model size and when to stop. Test data is looked at once, after every choice has been made.

A test set you consulted while tuning has quietly become a validation set, and the number it gives you is optimistic. Say that sentence twice; it is the most common way an honest evaluation goes wrong.

Split by scene or by drive, never by frame. Neighbouring frames of a video are nearly the same image, so a random frame split puts near-copies of the test set into training and the reported error will be far too good.

Reading the curves

Three pairs of curves. The third is overfitting, and the marked epoch is the checkpoint worth keeping.

Three cases, and what to do about each. Training loss falls, validation loss does not move: the model is memorising the training set and learning nothing that carries over. Both fall: it is learning something real, so keep going. Training falls while validation rises: overfitting. Stop, and keep the checkpoint from the epoch with the lowest validation loss, which is early stopping.

Eight points and a degree-7 polynomial. Eight coefficients for eight points drives the training error to zero and fits the noise exactly.

Overfitting, with a slider

The points below come from a smooth curve plus noise. Raise the degree of the fitted polynomial and watch the two errors part company: the training error keeps falling while the error on held-out points turns around and climbs.

Try these. Degree 9 with 10 points: the curve passes through almost every training point, the training error is near zero, and the held-out error is enormous. Same degree with 40 points: the wiggles flatten out and the held-out error drops, which is the first fix on the list below. A model with enough parameters can fit the noise, and noise does not repeat.

The fixes, roughly in the order worth trying: more data; data augmentation, which manufactures more data by flipping, cropping and recolouring; a smaller model; dropout, which zeroes a random half of the units on each step so no unit can rely on a particular partner; weight decay, a penalty on large weights; and early stopping. All of them trade a little training accuracy for less gap between training and validation.

7 Keeping training stable

The gradient reaching an early layer is a product of one factor for every layer after it. Products of many numbers behave badly: below one they collapse, above one they explode.

The gradient reaching each layer, drawn to scale. Every layer multiplies by one more factor, so by layer 1 there is nothing left to see.

Vanishing gradients. Ten sigmoid layers multiply by at most $0.25$ each, so the gradient reaching the first layer is at most $9.5 \times 10^{-7}$ of what left the loss. The early layers stop learning while the late ones train normally, and the symptom is a loss that falls a little and then sticks.

Exploding gradients. Factors above one, often helped along by a large learning rate, send the loss to NaN in a few steps. Clipping the gradient norm is the usual patch.

Dead ReLU. A unit whose sum is negative for every input outputs zero with slope zero, so it receives no gradient and never recovers. You saw one in the worked example in step 3. Leaky ReLU keeps a small slope below zero so its units cannot die.

Four settings that interact

The activation: ReLU or a variant in hidden layers. The learning rate: large steps are what push units dead and gradients up. The initial weights: random, and scaled to the width of the layer, so that the variance of the signal neither grows nor shrinks as it passes through. He initialisation draws from a normal with variance $2/n_{\text{in}}$ for ReLU layers; Xavier uses $1/n_{\text{in}}$ for tanh. Normalisation: of the inputs, and of the activations inside the network.

Change one and the best setting of the others moves. That is why recipes in papers list all four together, and why copying a learning rate without the initialisation it was tuned with rarely works.

Normalisation, worked

Batch norm averages down the batch; layer norm averages across the features of one example. Same formula, different axis.

Normalising means rescaling numbers so that their mean and spread are controlled. For one feature over a batch of $B$ examples:

\[ \mu = \frac{1}{B}\sum_j z_j, \qquad \sigma^2 = \frac{1}{B}\sum_j (z_j - \mu)^2, \qquad \hat{z}_j = \frac{z_j - \mu}{\sqrt{\sigma^2 + \epsilon}}, \qquad \text{out}_j = \gamma\,\hat{z}_j + \beta \]

Take four values, $2, 4, 4, 6$. The mean is 4 and the variance is 2, so the standard deviation is 1.414 and the normalised values are -1.41, 0, 0, 1.41. They now have mean 0 and spread 1, and $\gamma$ and $\beta$, both learned, let the network scale and shift them back if that is what it wants. $\epsilon$ is tiny and only stops a division by zero.

Batch norm averages each feature over the batch. At test time there may be a single image, so it uses running averages saved during training; forgetting model.eval() is the classic bug, and the symptom is a model that scores well in training and badly in deployment. It also degrades when the batch is small.

Layer norm averages over the features of one example, so it behaves identically at batch size 1 and is what transformers use. Group norm splits the channels into groups and normalises within each, and is the usual compromise in 3D networks that train on two or four scenes at a time.

8 Convolution

Feed an image to a fully connected layer and two things go wrong. The parameter count: $224 \times 224 \times 3 = 150,528$ inputs into 1000 units is 150,528,000 weights for one layer. And the structure: such a layer has no idea which pixels are neighbours. Shuffle every image's pixels the same way and it learns exactly as well, which tells you it was never using the fact that edges are local.

Edges and corners live in patches of pixels. A single pixel is only a brightness, and a fully connected layer cannot tell which pixels are neighbours.

What an image needs is a small set of weights, applied to a patch, and reused at every position. That is a convolution:

\[ (I \ast K)(r, c) = \sum_m \sum_n I(r + m,\ c + n)\,K(m, n) \]

One output value is the filter laid on one patch, multiplied entry by entry and summed. This filter computes the left column minus the right.

Lay the filter on a patch, multiply entry by entry, add. Slide one pixel and repeat. The same nine weights are used everywhere, which is weight sharing: an edge detector that works in one corner works in the other.

Slide it yourself

The shaded patch and the shaded output cell are the same computation, written out in the panel. Switch the filter and watch which patches respond: the vertical edge filter gives zero wherever the left and right columns match, and its largest values sit where the image changes from side to side.

Stride, padding, and the size of the output

Stride is how far the filter moves; padding is a border of zeros that controls the output size.

Stride is how far the filter moves each time; padding is a border of zeros added around the input. Together with the filter size they fix the output size:

\[ \text{out} = \left\lfloor \frac{W - K + 2P}{S} \right\rfloor + 1 \]

With $W = 7$, $K = 3$, $P = 0$: stride 1 gives 5, stride 2 gives 3. With $W = 224$, $K = 3$, $P = 1$: stride 1 gives 224, stride 2 gives 112. Setting $P = (K-1)/2$ with $S = 1$ keeps the size unchanged, which is why $3 \times 3$ convolutions with padding 1 are everywhere. Height and width are computed separately by the same formula.

Channels

A filter spans every input channel, and each filter produces one output channel, which is one feature map.

A colour image has three channels, and a filter spans all of them: a $3 \times 3$ filter on a 3-channel input holds $3 \times 3 \times 3$ weights and one bias. Use $C_{\text{out}}$ such filters and the output has $C_{\text{out}}$ channels, one feature map each. For 3 channels in and 64 out that is 1,792 parameters, against 151 million for the fully connected layer, and the convolution's count does not change with the size of the image.

Receptive field

Three $3 \times 3$ layers see a $7 \times 7$ patch of the input. Depth is what lets a local operation eventually see a large region.

The receptive field of a unit is the patch of input it can see. Each $3 \times 3$ layer widens it by 2, so three stacked layers see $7 \times 7$; stride and pooling widen it faster. Local connections keep the parameter count small, and depth is what lets a local operation eventually see a large region. How large it needs to be is a property of the task: fine geometry needs small neighbourhoods, and judging the layout of a room needs large ones.

Pooling and dropout

Max pooling with a $2 \times 2$ window and stride 2: it halves height and width and has nothing to learn.

Max pooling keeps the largest value in each window, average pooling the mean. A $2 \times 2$ window with stride 2 halves height and width and has no weights to learn. It reduces computation later and widens the receptive field. Most modern networks downsample with stride-2 convolutions instead, which does the same job with weights that can be trained.

Dropout acts on the units rather than the data: during training each unit's output is zeroed with probability $p$, redrawn every step, so no unit can depend on one particular partner. At test time every unit is used, with outputs scaled to match what the next layer saw during training.

9 From a classifier to a depth network

LeNet's shape: convolution and pooling shrink height and width and add channels, then fully connected layers finish the job.

A convolutional classifier alternates convolution and pooling, shrinking height and width while growing the number of channels, then flattens what is left and finishes with fully connected layers and a softmax. That is LeNet (LeCun et al., 1998), which read handwritten digits, and AlexNet (2012) is the same recipe made far larger and run on GPUs.

Trace the shapes once, with the formula from step 8:

Layer	Output	Parameters
input, one channel	$32 \times 32 \times 1$	0
$5 \times 5$ conv, 6 filters, $P = 0$	$28 \times 28 \times 6$	156
$2 \times 2$ max pool, $S = 2$	$14 \times 14 \times 6$	0
$5 \times 5$ conv, 16 filters, $P = 0$	$10 \times 10 \times 16$	2,416
$2 \times 2$ max pool, $S = 2$	$5 \times 5 \times 16$	0
flatten	400	0
fully connected, 120	120	48,120
fully connected, 10	10	1,210

Two details worth noticing. The convolutions hold a few thousand parameters and the fully connected layers hold fifty thousand, which is the usual split. And the first convolution's 156 parameters are $5 \times 5 \times 1 \times 6 + 6$: the filters are the cheap part.

What the layers learn

Visualise what excites each channel of a trained network and a progression appears: early filters respond to edges and colour, later ones to textures, then parts, then whole objects. The network has learned its own image pyramid and its own descriptors, tuned to the task, in place of the hand-designed ones from L04. That is also why a network trained on one large image task transfers to another with a little fine-tuning, which is what Unit 2 relies on.

The shape of a depth network

The shape of a depth network. The encoder shrinks, the decoder grows back to one output per pixel, and skip connections carry the fine detail.

Depth is a number per pixel, not a class per image, so drop the fully connected layers. An encoder shrinks the image into features, a decoder grows them back to full resolution, and skip connections carry fine detail from encoder to decoder at matching sizes. That is the U-Net shape, and it is what almost every dense prediction network looks like.

Train it with a loss at every pixel against measured depth, and the choice of loss decides what the network can even represent. MiDaS is trained with a loss that is invariant to scale and shift, so its output is relative depth: correct up to $aZ + b$ with $a$ and $b$ unknown for each image. No amount of training data changes that, because the loss was written not to care. A metric depth network uses a loss that does care, and takes the scale from the intrinsics or from training on one camera.

That closes the loop with the first three slides of L05. The geometry of Unit 0 gives you depth up to a scale from two views; a network gives you a dense guess from one view; and which of the two is metric depends on where the metres came from, never on the network's confidence.

Cheat sheet

Thing	Formula	Where
Neuron	$\hat{y} = \sigma(\mathbf{w}^\top\mathbf{x} + b)$	step 1
Squared error	$L = \frac{1}{N}\sum_i (\hat{y}_i - y_i)^2$	step 2
Gradient step	$w \leftarrow w - \eta\,\partial L/\partial w$	step 2
Chain rule on a graph	multiply local derivatives from the loss backwards	step 3
ReLU and its derivative	$\max(0, x)$; $1$ if $x > 0$, else $0$	step 4
Layer size	$mn + n$ parameters for $m$ inputs, $n$ outputs	step 4
Softmax	$p_k = e^{z_k}/\sum_j e^{z_j}$	step 5
Cross-entropy, and its gradient	$L = -\log p_{\text{correct}}$; $\partial L/\partial z_k = p_k - y_k$	step 5
Epoch	$\lceil N/B \rceil$ steps for $N$ examples in batches of $B$	step 6
Normalisation	$\hat{z} = (z - \mu)/\sqrt{\sigma^2 + \epsilon}$, then $\gamma\hat{z} + \beta$	step 7
Convolution output size	$\lfloor (W - K + 2P)/S \rfloor + 1$	step 8
Convolution parameters	$K^2 C_{\text{in}} C_{\text{out}} + C_{\text{out}}$	step 8
Receptive field	$+ (K - 1)$ per layer at stride 1	step 8

Reference code

Two versions of the same two-layer network, learning $f(x_1, x_2) = x_1^2 + 0.5x_2$ from eight points. The first writes the backward pass out with no autograd anywhere, and is the code behind step 3. The data and the initial weights are written down rather than drawn at random, so the printed losses are exactly 0.8701 at step 0, 0.0041 at step 500, and 0.0005 from step 1000 on.

import numpy as np

X = np.array([[-1.0, -0.6], [-0.6, 0.8], [-0.2, -0.2], [0.2, 0.6],
              [ 0.6, -1.0], [ 1.0, 0.2], [-0.8, 0.4], [0.8, -0.4]])
y = (X[:, 0] ** 2 + 0.5 * X[:, 1]).reshape(-1, 1)      # what the network has to learn

W1 = np.array([[0.9, -0.4, 0.3, 0.7], [-0.5, 0.8, -0.9, 0.2]])   # 2 inputs -> 4 hidden
b1 = np.zeros(4)
W2 = np.array([[0.6], [-0.8], [0.5], [0.3]])                     # 4 hidden -> 1 output
b2 = np.zeros(1)
eta = 0.1

for step in range(2001):
    # forward
    z1 = X @ W1 + b1                      # (8, 4) pre-activations
    h = np.maximum(0.0, z1)               # ReLU
    yh = h @ W2 + b2                      # (8, 1) predictions
    loss = np.mean((yh - y) ** 2)

    # backward: one line per node, right to left
    dyh = 2.0 * (yh - y) / len(X)         # dL/dyhat, averaged over the batch
    dW2 = h.T @ dyh                       # (4, 1)
    db2 = dyh.sum(axis=0)
    dh = dyh @ W2.T                       # (8, 4)
    dz1 = dh * (z1 > 0)                   # the ReLU gate: 1 where the unit was active
    dW1 = X.T @ dz1                       # (2, 4)
    db1 = dz1.sum(axis=0)

    for p, gp in ((W1, dW1), (b1, db1), (W2, dW2), (b2, db2)):
        p -= eta * gp                     # one gradient step, in place

    if step % 500 == 0:
        print(step, round(float(loss), 4))

Three things are worth noticing in that code. The backward block is the table from step 3, line for line. dz1 = dh * (z1 > 0) is the whole of the ReLU's derivative: a gate that passes the gradient where the unit was active and blocks it where it was not. And dividing by len(X) once, in dyh, is what makes this the gradient of the mean squared error rather than the sum.

The same network in PyTorch. The forward pass is a list of layers and the backward pass is one call.

import torch, torch.nn as nn

model = nn.Sequential(nn.Linear(2, 4), nn.ReLU(), nn.Linear(4, 1))
opt = torch.optim.SGD(model.parameters(), lr=0.1)
loss_fn = nn.MSELoss()

for step in range(2001):
    loss = loss_fn(model(X), y)   # forward pass and loss
    opt.zero_grad()               # clear the previous step's gradients
    loss.backward()               # the whole backward pass
    opt.step()                    # w <- w - lr * gradient

zero_grad is there because PyTorch adds new gradients to whatever is already stored. That is useful when one step accumulates several batches, and a bug the rest of the time: leave it out and the gradients from every previous step pile up.

The same ideas, in other people's notation

Here	Elsewhere	Note
$\hat{y}$	$a$, $f(x)$, $\hat{f}$	the network's output
$\mathbf{z}$	pre-activation, logits, scores	"logits" means the scores before a softmax
$\sigma$	$\phi$, $g$, ReLU written as $(\cdot)^+$	$\sigma$ sometimes means the sigmoid specifically
$\eta$	$\alpha$, lr	the learning rate
$L$	$J$, $\mathcal{L}$, cost, objective, criterion	"criterion" is the PyTorch word
$\theta$	$\mathbf{w}$, parameters, weights	$\theta$ usually means all of them at once
Channels	feature maps, filters, kernels, depth	"depth" here means channel count, not distance

Where to read more

3Blue1Brown, Neural networks, chapters 1 to 4. The best visual account of gradient descent and backpropagation. Watch chapters 3 and 4 after step 3 here.
Dive into Deep Learning (d2l.ai), chapters 3 to 5 and 7. Free, and every chapter is runnable code beside the maths. Chapter 7 is convolution.
Stanford CS231n notes. The convolutional-network notes are still the clearest short treatment of receptive fields and layer arithmetic.
Module notes B1 to B4. The same material in course notation, with self-checks.

The next things you will meet in this course are not in any of those: attention and transformers in L10, networks that read point clouds in L06, and sparse convolution for 3D grids. Each arrives with the paper that introduced it.