Convolution

The unifying principle

For two functions $f$ and $g$, convolution is defined as

$(f * g)(t) = \int_{-\infty}^{\infty} f(\tau)\, g(t - \tau)\, d\tau$

and in the discrete case

$(f * g)[n] = \sum_{k=-\infty}^{\infty} f[k]\, g[n-k].$

Three structural facts make this operation universal:

1. The flip-and-slide. The argument $g(t-\tau)$ reflects $g$ ($\tau \to -\tau$) and shifts it by $t$. As $t$ varies you sweep the kernel across $f$.
2. Commutativity & linearity. $f*g = g*f$, and $*$ is bilinear. So "signal" and "kernel" are interchangeable, and superposition holds.
3. The Convolution Theorem. Convolution becomes pointwise multiplication under the Fourier transform: $\mathcal{F}\{f*g\} = \mathcal{F}\{f\}\cdot\mathcal{F}\{g\}.$ This is why convolution is the natural operation for any linear time-invariant (LTI) system — it diagonalizes in the frequency basis.

Everything below is a special case of this one integral/sum.

How it shows up in each field

Maths — the analytic operation & probability sums

Convolution is a binary operation on functions turning the space $L^1(\mathbb{R})$ into a commutative algebra (multiplication = $*$, no identity except the Dirac delta $\delta$, where $f*\delta = f$).

Its sharpest probabilistic incarnation: if $X$ and $Y$ are independent random variables with densities $p_X, p_Y$, the density of their sum is

$p_{X+Y}(z) = (p_X * p_Y)(z) = \int p_X(x)\, p_Y(z-x)\, dx.$

Physics — impulse response of linear systems

A physical LTI system (an RLC circuit, a damped oscillator, an optical lens) is fully characterized by its impulse response $h(t)$: its output when kicked by $\delta(t)$. For any input $x(t)$, the output is

$y(t) = (x * h)(t) = \int x(\tau)\, h(t-\tau)\, d\tau.$

AI-ML — the convolutional layer

A CNN layer convolves an input feature map with learned kernels. (Framework "convolution" is technically cross-correlation — no kernel flip — but the sliding-weighted-sum structure is identical.) In 2D:

$(I * K)[i,j] = \sum_{m}\sum_{n} I[i+m,\, j+n]\, K[m,n].$

The kernel $K$ is learned, and weight-sharing (same $K$ at every location) gives translation equivariance: shift the input, the output shifts identically. This is exactly LTI's time-invariance, spatialized.

Coding/CS — algorithms, polynomials & FFT

Discrete convolution is a core primitive. Polynomial multiplication is convolution of coefficient vectors:

$\Big(\sum a_i x^i\Big)\Big(\sum b_j x^j\Big) = \sum_n \Big(\sum_k a_k b_{n-k}\Big) x^n = \sum_n (a*b)[n]\, x^n.$

Naïvely $O(N^2)$; via the Convolution Theorem and FFT it drops to $O(N\log N)$: transform, pointwise-multiply, inverse-transform.

Why this bridge matters

• Independence ⇄ multiplication. The probabilist's "sums of independent variables convolve" and the engineer's "Fourier turns convolution into multiplication" are the same theorem: MGFs/characteristic functions of independent sums multiply.
• Impulse response = kernel = Green's function. The physicist's intuition — characterize a system by its response to a spike, then everything else follows by superposition — is exactly why a CNN filter is a learnable local feature detector.
• Diagonalization insight transfers. Knowing convolution is "just multiplication in the eigenbasis" (Fourier/frequency domain) explains: why FFT accelerates polynomial multiply, why CNNs are cheap in frequency space, why filtering suppresses frequency bands, and why the CLT smooths toward Gaussians (repeated multiplication of characteristic functions).
• Equivariance is portability. Translation-invariance in physics (LTI) → weight sharing in ML → shift-invariant filters in signal processing. One symmetry principle, four vocabularies.

Connections

• 03-Fourier-Transform
• 07-Linear-Time-Invariant-Systems
• 11-Central-Limit-Theorem
• 14-Convolutional-Neural-Networks
• 18-Greens-Functions
• 22-FFT-and-Fast-Multiplication
• 05-Probability-Densities
• 09-Impulse-Response

#bridge