Sampling & Monte Carlo

The unifying principle

Every field below secretly wants to compute an expectation:

$$\mathbb{E}_{p}[f] = \int f(x)\, p(x)\, dx$$

The Monte Carlo estimator draws $x_1,\dots,x_N \sim p$ and reports

$$\hat{\mu}_N = \frac{1}{N}\sum_{i=1}^N f(x_i).$$

Two theorems make this work:

• Law of Large Numbers: $\hat{\mu}_N \xrightarrow{\text{a.s.}} \mathbb{E}_p[f]$.
• Central Limit Theorem: the error shrinks as

$$\hat{\mu}_N - \mathbb{E}_p[f] \sim \mathcal{N}\!\left(0,\ \tfrac{\sigma^2}{N}\right),\qquad \text{error} \propto \frac{1}{\sqrt{N}}.$$

The crucial, field-agnostic fact: the $1/\sqrt{N}$ rate is independent of dimension. A 1000-dimensional integral is no worse than a 3-D one — which is why deterministic grids fail (they scale as $N^{-1/d}$) and sampling wins.

How it shows up in each field

Maths — numerical integration & MCMC

The primal setting. To estimate $\int_{[0,1]^d} f$, draw uniform points and average.

Physics — statistical mechanics

Thermodynamic averages are expectations over the Boltzmann distribution:

$$p(x) = \frac{1}{Z}e^{-\beta E(x)},\qquad Z = \sum_x e^{-\beta E(x)},\quad \beta = \tfrac{1}{k_B T}.$$

$Z$ is an intractable sum over $2^{N}$ configurations — exactly the "known up to a constant" case. The Metropolis algorithm was invented here (1953) to sample spin states.

AI-ML — Bayesian inference & generative models

Posterior expectations $\mathbb{E}_{p(\theta\mid D)}[f(\theta)]$ are Monte Carlo estimates; the evidence $p(D)=\int p(D\mid\theta)p(\theta)d\theta$ is the ML analogue of the partition function $Z$.

• Hamiltonian Monte Carlo borrows physics' energy $E = -\log p$ and simulates dynamics.
• Diffusion models sample by reversing a stochastic (Langevin) process.
• Dropout / REINFORCE estimate gradients $\nabla_\theta \mathbb{E}[R]$ by sampling.

Stock-Market — option pricing & risk

An option's fair price is a discounted expectation under the risk-neutral measure $Q$:

$$V_0 = e^{-rT}\,\mathbb{E}_Q\!\left[\text{payoff}(S_T)\right].$$

Simulate the underlying via geometric Brownian motion:

$$S_{t+\Delta} = S_t \exp\!\left[(r-\tfrac{\sigma^2}{2})\Delta + \sigma\sqrt{\Delta}\,Z\right],\quad Z\sim\mathcal N(0,1),$$

generate $N$ price paths, average the payoffs, discount.

Why this bridge matters

• The risk-neutral measure $Q$ is importance sampling. Girsanov's change of measure in finance is mathematically the reweighting $p/q$ — recognizing this lets you import variance-reduction tricks (control variates, antithetic variates) from ML/statistics into pricing.
• $Z$ = partition function = evidence = normalizing constant. The same intractable object haunts physics, Bayesian ML, and Gibbs-style economic models; MCMC solves all three identically.
• Intuition transfer: physicists' notion of temperature becomes ML's annealing / tempering for escaping local modes; ML's Langevin dynamics is literally an overdamped physical SDE; finance's path simulation is a special case of sampling an SDE's law.
• The curse of dimensionality is defeated the same way everywhere: never grid — always sample.

Connections

• 01 Probability & Expectation
• 02 Markov Chains & MCMC
• 03 Statistical Mechanics & Partition Functions
• 04 Bayesian Inference
• 05 Generative & Diffusion Models
• 06 Stochastic Differential Equations
• 07 Risk-Neutral Pricing & Black–Scholes
• 08 Variance Reduction & Importance Sampling

#bridge