A convolutional neural network learns small filters and applies each filter across an image. Each output is a weighted sum of a local patch. Reusing the weights gives the network a useful assumption: a pattern worth detecting in one place may also be worth detecting elsewhere.
I want to keep the arithmetic visible here. One small image, one filter, every output, then a backward pass that changes the filter. The larger architecture grows from those operations.
[!NOTE] The prerequisites are a dot product and the chain rule. The earlier posts on linear algebra and backpropagation cover those. A dot product multiplies matching entries and adds them; the chain rule multiplies derivatives along a path and adds contributions from different paths.
What does a CNN do to an image?
Here is a small classifier I will use for the shape calculations. Its widths are chosen for this example; they are not measurements from a trained model. Both convolutions use stride one, dilation one, and a learned bias per filter.
flowchart TD
X["RGB image: 3 × 32 × 32"] --> C1["3 × 3 convolution: 8 filters, padding 1"]
C1 --> A1["ReLU: 8 × 32 × 32"]
A1 --> P["2 × 2 max pooling, stride 2: 8 × 16 × 16"]
P --> C2["3 × 3 convolution: 16 filters, padding 1"]
C2 --> A2["ReLU: 16 × 16 × 16"]
A2 --> G["Global average pooling: 16 numbers"]
G --> F["Linear classifier: 10 logits"]
F --> L["Softmax probabilities and cross-entropy loss"]
An image is a tensor, an array with several axes. I use channel-first notation: channels, height, width. A batch adds an axis in front.
| Symbol | Meaning | Shape or range |
|---|---|---|
| $N$ | Images in a batch | Positive integer |
| $C_{\mathrm{in}}, C_{\mathrm{out}}$ | Input and output channels | Positive integers |
| $H,W$ | Input height and width | Positive integers |
| $k_h,k_w$ | Filter height and width | Positive integers |
| $X$ | Input batch | $N\times C_{\mathrm{in}}\times H\times W$ |
| $K$ | Learned filter weights | $C_{\mathrm{out}}\times C_{\mathrm{in}}\times k_h\times k_w$ |
| $b$ | One learned bias per output channel | $C_{\mathrm{out}}$ |
| $Z,A$ | Values before and after activation | $N\times C_{\mathrm{out}}\times H_{\mathrm{out}}\times W_{\mathrm{out}}$ |
RGB provides three input channels. Eight filters produce eight output channels, also called feature maps. Each filter spans all three RGB channels. The network sums those channel contributions to produce one value at each output position.1
Pooling reduces spatial size. ReLU changes values without changing shape. The last linear layer produces one raw score, or logit, per class.
How does a convolution work?
Start with one grayscale image, one filter, stride one, and no padding. Omit the batch and channel axes. The output at row $i$, column $j$ is
The indices $u,v$ select positions inside the filter. The indices $i,j$ select its placement on the image. Move the filter, multiply matching entries, sum, and add the same bias.
One complete numerical convolution
Given a $3\times3$ input, a $2\times2$ filter, and scalar bias $b=0$:
Find the $2\times2$ output. For the top-left patch, the four products are
Add them, then add the bias:
The remaining placements use the same four weights:
| Output | Matching products | Sum, then bias | Result |
|---|---|---|---|
| $Z_{0,0}$ | $1\cdot1+0\cdot2+0\cdot0-1\cdot1$ | $1+0+0-1+0$ | 0 |
| $Z_{0,1}$ | $1\cdot2+0\cdot0+0\cdot1-1\cdot3$ | $2+0+0-3+0$ | -1 |
| $Z_{1,0}$ | $1\cdot0+0\cdot1+0\cdot2-1\cdot1$ | $0+0+0-1+0$ | -1 |
| $Z_{1,1}$ | $1\cdot1+0\cdot3+0\cdot1-1\cdot0$ | $1+0+0+0+0$ | 1 |
Check: a two-cell-wide filter fits at two positions along each three-cell axis. The output has four values. This particular filter measures the top-left value minus the bottom-right value of each patch.
Is this convolution or cross-correlation?
The operation above is cross-correlation. Mathematical convolution flips the kernel along both spatial axes. CNN libraries commonly keep the unflipped operation and call the layer convolution.2
For the numerical kernel, a flip gives
On the top-right patch, that produces $-2+3=1$, while the unflipped filter produced $-1$. The convention changes a calculation with fixed weights. With unrestricted learned weights, either orientation can represent the same filters. I use the unflipped convention throughout.
What do stride, padding, and dilation change?
Stride $s$ is the step between filter placements. Padding $p$ adds cells on each side; here those cells are zero. Dilation $d$ is the spacing between sampled filter entries. All three affect the output size.
For one spatial axis, let $n$ be the input length and $k$ the number of filter entries. The effective filter span is
The padded input has length $n+2p$. A filter whose starting coordinate is $t$ fits if
Starts occur at $0,s,2s,\ldots$. Count those positions, including the first:
Apply this separately to height and width. This formula assumes symmetric padding, positive integer stride and dilation, and an effective kernel that fits the padded input.1
For $n=32$, $k=3$, $p=1$, $d=1$, and $s=2$:
The final unused position is allowed. There is no requirement that the division produce an integer.
| Input axis | Kernel | Padding per side | Stride | Dilation | Effective span | Output axis |
|---|---|---|---|---|---|---|
| 32 | 3 | 0 | 1 | 1 | 3 | 30 |
| 32 | 3 | 1 | 1 | 1 | 3 | 32 |
| 32 | 3 | 1 | 2 | 1 | 3 | 16 |
| 32 | 3 | 2 | 1 | 2 | 5 | 32 |
The dilated filter still has three entries per axis. It samples offsets $0,2,4$ across a five-cell span. A two-dimensional $3\times3$ dilated filter has nine spatial weights, not twenty-five.
For completeness, the channel-aware forward equation for one image is
Here $o$ selects the output channel, $c$ selects the input channel, and $\widetilde X$ is the already padded image. Subscripts $h,w$ allow separate vertical and horizontal settings. This is ordinary convolution with all input channels connected to every output channel, also called groups=1.
Why does weight sharing save parameters?
One filter has $C_{\mathrm{in}}k_hk_w$ weights and one bias. There are $C_{\mathrm{out}}$ filters, so
The image height and width are absent. A larger image uses the weights more times; it does not add weights to this layer.
For the first layer of our classifier:
It produces $8\cdot32\cdot32=8{,}192$ output values from $3\cdot32\cdot32=3{,}072$ input values. Compare layers producing that same number of outputs:
| Connection pattern | Parameters with bias | Computed total |
|---|---|---|
| Dense: every output sees every input | $8{,}192(3{,}072+1)$ | 25,174,016 |
| Local: a separate 27-weight filter at each output | $8{,}192(27+1)$ | 229,376 |
| Convolution: eight filters reused at every location | $8(27+1)$ | 224 |
The local row counts all patch slots, including slots touching padded zeros. The dense layer can represent functions this convolution cannot. The saving comes from restricting the connections and requiring weights at different positions to match.
There is a linear-algebra view of the same restriction. Flatten a one-dimensional input $x=[x_0,x_1,x_2]^\top$ and apply weights $a,b$ with no bias:
The $2\times3$ matrix contains repeated weights and zeros. With $a=1$, $b=-1$, and $x=[1,2,4]^\top$, its rows give $y_0=1-2=-1$ and $y_1=2-4=-2$. A spatial convolution is a structured linear map. Adding a bias makes it affine.
Why add ReLU and pooling?
ReLU, the rectified linear unit, acts on each scalar:
Our numerical convolution becomes
Without nonlinear operations, a stack of affine layers remains one affine map. If $W_1,W_2$ are compatible matrices and $b_1,b_2$ their biases, expand two layers:
ReLU makes which entries pass through depend on the input. The resulting network is piecewise affine rather than one affine function everywhere.
A $2\times2$ max pool with stride two selects the largest value in each non-overlapping window. Average pooling takes the mean. Applied to the activated patch above:
Both turn this $2\times2$ map into $1\times1$ and add no learned parameters. They discard different information. Max pooling keeps the largest response; averaging keeps its contribution relative to the whole window.
Global average pooling takes the mean of an entire channel. For a map of height $h$ and width $w$, its channel-$c$ output is
In the classifier, each of the sixteen $16\times16$ maps becomes one number. The final dense layer therefore needs $10(16+1)=170$ parameters. Flattening those maps first would give $16\cdot16\cdot16=4{,}096$ inputs and $10(4{,}096+1)=40{,}970$ parameters. Averaging saves parameters but removes explicit spatial layout from the classifier input.
How does a small filter see a larger region?
A receptive field is the region of the original input that can affect a particular feature. Later filters read earlier features, so their input coverage grows with depth.3
For a sequential stack, track $r_\ell$, the receptive-field width after layer $\ell$, and $j_\ell$, the distance in original-input pixels between neighboring features. Start with $r_0=j_0=1$. With effective kernel span $k_{\mathrm{eff},\ell}$ and stride $s_\ell$:
A filter adds one previous jump for each extra sampled position when dilation is one. Dilation stretches that span. Apply the equations per axis to the example architecture:
| Layer | Receptive-field calculation | Width $r$ | Jump $j$ |
|---|---|---|---|
| Input | Initial pixel | 1 | 1 |
| $3\times3$ convolution, stride 1 | $1+(3-1)\cdot1$ | 3 | 1 |
| ReLU | Pointwise; no added coverage | 3 | 1 |
| $2\times2$ pool, stride 2 | $3+(2-1)\cdot1$ | 4 | 2 |
| $3\times3$ convolution, stride 1 | $4+(3-1)\cdot2$ | 8 | 2 |
Each feature before global pooling can depend on an $8\times8$ region. At image edges, part of that region may be padding. This is theoretical coverage; learned weights and activation gates determine which inputs influence a particular prediction.
Are CNNs translation invariant?
Translation equivariance means shifting the input shifts the feature map by the corresponding amount. Translation invariance means the output stays the same. These are different properties.
The weight-sharing equation explains equivariance. Use one dimension, stride one, and an infinite grid to avoid boundary issues. Define a shift by integer $a$ as $(T_a x)[i]=x[i-a]$. Let $F$ be cross-correlation with fixed kernel $k[u]$:
For example, with $k=[1,-1]$, a value $5$ at input position 2 and zeros elsewhere produces $-5$ at output position 1 and $5$ at position 2. Move that input value to position 3: the outputs move to positions 2 and 3. Their values do not change.
Finite-image boundaries and subsampling need more care. Here is a counterexample to the claim that max pooling guarantees invariance. Use width-two max pooling with stride two, and circularly shift a four-entry input one place right:
The pooled values changed. The shift changed which values shared a window. Zhang’s work on antialiased CNNs studies this sensitivity to downsampling.4
Global averaging is invariant to a permutation of an unchanged set of feature values. An image shift that introduces padding, crops content, or changes downsampled features can change that set. A CNN classifier therefore has no blanket guarantee of translation invariance.
What objective trains a CNN classifier?
For our ten-class model, global pooling produces $g\in\mathbb R^{16}$. The classifier has $V\in\mathbb R^{10\times16}$ and bias $a\in\mathbb R^{10}$:
The logit vector $\ell$ has ten entries. For $M$ mutually exclusive classes, softmax converts logits to probabilities $p_j$. If the correct class is $t$, cross-entropy loss is
Use two logits $[\log2,0]$ and correct class $t=1$ to make one prediction by hand:
The probabilities sum to one. The loss is positive because the correct class has probability below one; the decimal is rounded to six places.
To differentiate with respect to logit $\ell_j$, expand the log probability first:
The indicator $\mathbf1[j=t]$ equals one for the correct class and zero otherwise. The two-logit example gives gradients $[-1/3,1/3]$. Gradient descent raises the correct logit and lowers the other one. The cross-entropy post follows this objective in more detail.
How does backpropagation update a shared filter?
Let $\delta_{o,i,j}=\partial L/\partial Z_{o,i,j}$ be the gradient arriving at a convolution’s output before activation. For valid, stride-one cross-correlation on one image, differentiate with respect to one weight:
The same weight helped produce many output values, so it receives a contribution from each use. Biases obey the same rule:5
For a batch, also sum over images. If the objective averages examples, that normalization is already present in $\delta$.
The input gradient adds contributions from every output window that used the input pixel. For input row $r$, column $s$:
where out-of-range $\delta$ entries are zero. The subtraction in the indices comes from solving $r=i+u$ and $s=j+v$ for the output position.
ReLU supplies a gate before these sums. If $G=\partial L/\partial A$ is the arriving gradient, then
At zero, ReLU has no unique derivative; taking the gradient as zero is a common convention. A max-pooling backward pass routes the gradient to the selected maximum. Average pooling spreads it equally across the window. For a unique maximum in a $2\times2$ window and incoming gradient $4$, max pooling sends $4$ to that entry; average pooling sends $1$ to each of the four entries. Tied maxima need a defined selection or subgradient convention.
One full gradient-descent step
I will isolate the convolution and ReLU so every learned value fits on paper. Use the same $X$ and $K$ from the forward example, change the initial bias to $b=2$, and choose a $2\times2$ target map $Y$. This is a small regression exercise with squared error, separate from the classification objective above.
Find the gradients of all four weights and the bias, update them with learning rate $\eta=0.01$, and recompute the loss.
Add two to every entry of the earlier convolution output:
All entries are positive, so all four ReLU derivatives are one. Define a summed loss, with no division by the four outputs:
Subtract the target and square each residual:
The outer derivative of $\frac12 e^2$ with respect to residual $e$ is $e$. The derivative of $e=A-Y$ with respect to $A$ is one. Multiply those derivatives and the ReLU gate:
where $\odot$ means entry-by-entry multiplication. Each shared weight accumulates four contributions, in output order $(0,0),(0,1),(1,0),(1,1)$:
| Parameter | Gradient contributions | Sum |
|---|---|---|
| $K_{0,0}$ | $1\cdot1+1\cdot2+1\cdot0+1\cdot1$ | 4 |
| $K_{0,1}$ | $1\cdot2+1\cdot0+1\cdot1+1\cdot3$ | 6 |
| $K_{1,0}$ | $1\cdot0+1\cdot1+1\cdot2+1\cdot1$ | 4 |
| $K_{1,1}$ | $1\cdot1+1\cdot3+1\cdot1+1\cdot0$ | 5 |
| $b$ | $1+1+1+1$ | 4 |
Check the gradient shapes: $\nabla_KL$ is $2\times2$, matching $K$; $\partial L/\partial b$ is a scalar. For the center input pixel, all four output windows contribute:
Some paths cancel. The filter gradients still need all four paths.
Apply $\theta_{\mathrm{new}}=\theta-\eta\nabla_\theta L$ to each learned parameter:
| Parameter | Learning rate × gradient | Old value − update | New value |
|---|---|---|---|
| $K_{0,0}$ | $0.01\cdot4=0.04$ | $1-0.04$ | 0.96 |
| $K_{0,1}$ | $0.01\cdot6=0.06$ | $0-0.06$ | -0.06 |
| $K_{1,0}$ | $0.01\cdot4=0.04$ | $0-0.04$ | -0.04 |
| $K_{1,1}$ | $0.01\cdot5=0.05$ | $-1-0.05$ | -1.05 |
| $b$ | $0.01\cdot4=0.04$ | $2-0.04$ | 1.96 |
Recompute every output using those new values:
| Output | Products plus bias | Result |
|---|---|---|
| $Z^{\prime}_{0,0}$ | $0.96-0.12+0-1.05+1.96$ | 1.75 |
| $Z^{\prime}_{0,1}$ | $1.92+0-0.04-3.15+1.96$ | 0.69 |
| $Z^{\prime}_{1,0}$ | $0-0.06-0.08-1.05+1.96$ | 0.77 |
| $Z^{\prime}_{1,1}$ | $0.96-0.18-0.04+0+1.96$ | 2.70 |
All outputs remain positive, so ReLU leaves them unchanged. The new residuals and loss are
One update reduced this example’s loss from $2$ to $1.06075$. That checks the direction of this step; it does not establish generalization or guarantee that every learning rate will reduce the loss.
For the full ten-class architecture, the trainable parameter count is $224+1{,}168+170=1{,}562$: two convolutions and the final classifier. The second convolution contributes $16(8\cdot3\cdot3+1)=1{,}168$. ReLU and pooling contribute zero. During training, the loss gradient passes through the classifier, the pooling operations, and both convolutions. Every filter update adds the evidence from all locations where that filter was used.
Read next: Numerical gradient checking compares these derivatives against changes in the loss from small parameter perturbations.
-
PyTorch Conv2d documentation, for cross-correlation, channel layout, stride, padding, dilation, and output dimensions. The numerical examples here are constructed for this post. ↩ ↩2
-
Goodfellow, Bengio, and Courville, Deep Learning, chapter 9, especially §§9.1–9.3 on convolution, parameter sharing, and pooling. ↩
-
Araujo, Norris, and Sim, Computing Receptive Fields of Convolutional Neural Networks, 2019. The table here applies the single-path recurrence to the example classifier. ↩
-
Zhang, Making Convolutional Networks Shift-Invariant Again, ICML 2019, on downsampling and shift sensitivity. ↩
-
Stanford CS231n convolutional-network notes, on accumulating gradients over shared uses. The chain-rule derivation and numerical update here use the unflipped convention stated above. ↩
Support the writing
If this post helped, a coffee keeps the deep dives coming.
Buy me a coffee