Convolutional Neural Networks: How Machines Learn to See

🏁 Introduction: The Limits of Fully-Connected Networks

In our neural network from scratch post, we built a fully-connected (dense) network that achieved solid accuracy on MNIST. But there's a fundamental problem with that approach when it comes to images: it ignores spatial structure.

Consider a 28×28 MNIST image. We flattened it into a 784-dimensional vector. Every pixel was connected to every neuron in the next layer — 784 connections per neuron, regardless of whether those pixels are neighbors or on opposite ends of the image.

Now scale up to a real-world image: 224×224×3 (RGB) = 150,528 inputs. A single hidden layer with 1,000 neurons would need 150 million parameters — just for layer one. That's astronomically expensive, prone to overfitting, and ignores that nearby pixels are related in ways that distant pixels are not.

Convolutional Neural Networks (CNNs) solve this with two powerful ideas: local connectivity and parameter sharing.

🔍 The Convolution Operation

The core operation is a discrete convolution. A small matrix called a filter (or kernel) slides across the input image, computing a dot product at each position.

For a 2D input $I$ and filter $K$ of size $f \times f$:

$S(i, j) = (I * K)(i, j) = \sum_{m=0}^{f-1} \sum_{n=0}^{f-1} I(i+m,\ j+n) \cdot K(m, n)$

The result $S$ is called a feature map or activation map.

What Do Filters Learn?

Each filter learns to detect a specific local pattern:

• Edge detectors: respond strongly where brightness changes sharply
• Texture detectors: respond to repeated local patterns
• Shape detectors: in deeper layers, combine edges into curves and corners
• Object part detectors: in the deepest layers, detect eyes, wheels, faces

This hierarchy — from simple to complex features — mirrors what neuroscientists found in the mammalian visual cortex (the discovery that inspired CNNs).

Key Parameters

Padding: Adding zeros around the border of the input to control output size.

• valid (no padding): output shrinks by $f-1$ per dimension
• same (zero-padding): output size equals input size

Stride $s$: How many pixels the filter moves at each step.

Output size formula:

$\text{output size} = \left\lfloor \frac{n + 2p - f}{s} + 1 \right\rfloor$

Where $n$ is the input size, $p$ is the padding amount, $f$ is the filter size.

🧱 Building Blocks of a CNN

1. Convolutional Layer

The convolutional layer applies $k$ different filters to the input, producing $k$ feature maps. Each filter has its own learnable weights.

import numpy as np

def conv2d(X, W, b, stride=1, pad=0):
    """
    X: input of shape (H, W, C_in)
    W: filters of shape (f, f, C_in, C_out)
    b: biases of shape (C_out,)
    """
    H, W_in, C_in = X.shape
    f, _, _, C_out = W.shape

    # Add padding
    X_pad = np.pad(X, ((pad, pad), (pad, pad), (0, 0)))

    H_out = (H + 2 * pad - f) // stride + 1
    W_out = (W_in + 2 * pad - f) // stride + 1
    Z = np.zeros((H_out, W_out, C_out))

    for h in range(H_out):
        for w in range(W_out):
            for c in range(C_out):
                hs, ws = h * stride, w * stride
                patch = X_pad[hs:hs+f, ws:ws+f, :]
                Z[h, w, c] = np.sum(patch * W[:, :, :, c]) + b[c]

    return Z

Why parameter sharing matters: A 3×3 filter applied to a 224×224 image uses only 9 weights (plus bias) to produce one feature map, regardless of input size. Compare that to a dense layer's 150,528 weights per neuron. This is why CNNs are tractable.

2. Activation Layer (ReLU)

After each convolution, we apply a non-linearity. ReLU is the standard choice:

$\text{ReLU}(z) = \max(0, z)$

Without this, stacking convolutions would just be a single linear transformation — no deeper representation power.

3. Pooling Layer

Pooling downsamples the spatial dimensions, making the representation smaller and more computationally efficient. It also introduces translation invariance — a slight shift in the input doesn't completely change the output.

Max Pooling takes the maximum value in each window:

$P(i, j) = \max_{m, n \in \text{window}} S(i \cdot s + m,\ j \cdot s + n)$

def max_pool2d(X, pool_size=2, stride=2):
    H, W, C = X.shape
    H_out = (H - pool_size) // stride + 1
    W_out = (W - pool_size) // stride + 1
    out = np.zeros((H_out, W_out, C))

    for h in range(H_out):
        for w in range(W_out):
            hs, ws = h * stride, w * stride
            out[h, w, :] = np.max(
                X[hs:hs+pool_size, ws:ws+pool_size, :],
                axis=(0, 1)
            )
    return out

4. Flatten + Fully-Connected Layer

After several conv+pool blocks, the feature maps are flattened into a vector and fed into dense layers for the final classification. This is where our familiar neural network machinery from before takes over.

🏛️ Classic Architectures

LeNet-5 (1998) — Yann LeCun

The OG CNN, designed for digit recognition (sound familiar?):

Input (32×32) → Conv(6 filters, 5×5) → Pool → Conv(16 filters, 5×5) → Pool → FC(120) → FC(84) → Output(10)

AlexNet (2012) — The ImageNet Revolution

AlexNet won the 2012 ImageNet challenge with a 15.3% error rate vs runner-up's 26.2%. It ignited the deep learning revolution:

• 8 layers (5 conv, 3 FC)
• ReLU activations (instead of tanh/sigmoid)
• Dropout for regularization
• Data augmentation
• GPU training (two GPUs for 5-6 days!)

Modern Trend: ResNets and Beyond

Deep networks (20+ layers) suffered from the vanishing gradient problem — gradients became tiny during backpropagation, making early layers learn almost nothing.

ResNets (2015) solved this with skip connections (residual connections):

$\text{output} = F(x) + x$

This "shortcut" lets the gradient flow directly backwards through the addition, bypassing potentially problematic layers. ResNet-152 (152 layers!) became possible and won ImageNet 2015.

🧠 Backpropagation Through Convolutions

Backprop through conv layers involves computing gradients with respect to the filter weights and the input. The key insight: backprop through a convolution is also a convolution (with a flipped filter).

For filter gradients:

$\frac{\partial L}{\partial W} = X * \frac{\partial L}{\partial Z}$

For input gradients (needed to propagate to earlier layers):

$\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Z} *_{\text{full}} W^{\text{flip}}$

This is why deep learning frameworks implement "transposed convolutions" — they're the backward pass of a regular convolution.

📦 CNN in PyTorch (5 Lines)

While we've done the math and NumPy implementation, in practice you'd use a framework:

import torch.nn as nn

model = nn.Sequential(
    nn.Conv2d(in_channels=1, out_channels=32, kernel_size=3, padding=1),
    nn.ReLU(),
    nn.MaxPool2d(kernel_size=2, stride=2),
    nn.Conv2d(32, 64, kernel_size=3, padding=1),
    nn.ReLU(),
    nn.MaxPool2d(2, 2),
    nn.Flatten(),
    nn.Linear(64 * 7 * 7, 10)
)

This simple architecture would achieve >99% accuracy on MNIST — far beyond our fully-connected version.

🌍 Applications Beyond Images

CNNs aren't limited to 2D images:

• 1D CNNs: audio signals, time series, DNA sequences
• 3D CNNs: video analysis, medical volumetric data (CT/MRI scans)
• Graph CNNs: molecular property prediction, social networks

The convolution principle — local pattern detection with parameter sharing — is universally powerful.

🧭 Final Thoughts

CNNs teach us a profound lesson about inductive bias: by building our prior knowledge about image structure (local correlations, translation invariance) directly into the architecture, we can learn far more efficiently than a naive fully-connected approach.

• Convolutions detect local spatial patterns
• Pooling builds scale and translation invariance
• Hierarchical stacking builds complex representations from simple ones

In the next post, we'll tackle a different kind of data — sequences — and introduce the Transformer architecture and self-attention, which has reshaped not just NLP but computer vision and biology too.

📌 Next Steps

• Implement a LeNet-style CNN from scratch using NumPy (full forward + backward pass)
• Experiment with different filter sizes and depths on CIFAR-10
• Visualize what each filter has learned using gradient-based visualization

---

Written by Breye.