Uniform, Exponential, and Beta distributions

The Uniform Distribution

Derivation from First Principles

WHY uniform? We want "no preference"—the probability density must be constant across $[a, b]$ and zero elsewhere.

WHAT does that mean mathematically? Let the constant density be $c$ . Since probability must integrate to 1:

$\int_{a}^{b} c \, dx = 1$

HOW do we solve?

$c \cdot (b - a) = 1 \implies c = \frac{1}{b-a}$

Key Properties (Derived)

Expected Value:

$E[X] = \int_{a}^{b} x \cdot \frac{1}{b-a} \, dx = \frac{1}{b-a} \cdot \frac{x^2}{2} \Big|_a^b = \frac{1}{b-a} \cdot \frac{b^2 - a^2}{2} = \frac{b+a}{2}$

WHY? The mean is exactly the midpoint—makes sense for a symmetric distribution!

Variance:

$E[X^2] = \int_{a}^{b} x^2 \cdot \frac{1}{b-a} \, dx = \frac{1}{b-a} \cdot \frac{x^3}{3} \Big|_a^b = \frac{b^3 - a^3}{3(b-a)} = \frac{a^2 + ab + b^2}{3}$

$\text{Var}(X) = E[X^2] - (E[X])^2 = \frac{a^2 + ab + b^2}{3} - \frac{(a+b)^2}{4} = \frac{(b-a)^2}{12}$

WHY this matters? Variance grows with the square of the range—wider intervals mean more uncertainty.

Worked example Uniform in Random Initialization

Problem: Initialize weights for a neural network layer with 100 inputs. Use Xavier initialization: sample from $U(-\frac{1}{\sqrt{n}}, \frac{1}{\sqrt{n}})$ where $n=100$ . Solution:

$a = -\frac{1}{\sqrt{100}} = -0.1$ , $b = 0.1$
PDF: $f(x) = \frac{1}{0.2} = 5$ for $x \in [-0.1, 0.1]$
Mean: $\mu = \frac{-0.1 + 0.1}{2} = 0$ (centered at zero, preventing bias)
Variance: $\sigma^2 = \frac{(0.2)^2}{12} = \frac{0.04}{12} = 0.00333$ WHY this step? Variance $\approx \frac{1}{n}$ keeps gradients stable—not too small (vanishing) or large (exploding).

Figure — Uniform, Exponential, and Beta distributions

The Exponential Distribution

Derivation from Poisson Process

WHY exponential? In a Poisson process, events are independent and occur at constant rate $\lambda$ . What's the probability that NO events occur in time $t$ ?

From Poisson distribution: $P(\text{0 events in time } t) = e^{-\lambda t}$

WHAT is the CDF? The probability that the first event occurs BEFORE time $t$ is:

$F(t) = P(T \leq t) = 1 - P(T > t) = 1 - e^{-\lambda t}$

HOW do we get the PDF? Differentiate the CDF:

$f(t) = \frac{d}{dt}F(t) = \frac{d}{dt}(1 - e^{-\lambda t}) = \lambda e^{-\lambda t}$

The Memoryless Property

CLAIM: $P(T > s + t \mid T > s) = P(T > t)$

WHY is this profound? "If you've waited $s$ time units without an event, the probability of waiting another $t$ is the same as if you just started." The process "forgets" the past.

Proof:

$P(T > s+t \mid T > s) = \frac{P(T > s+t)}{P(T > s)} = \frac{e^{-\lambda(s+t)}}{e^{-\lambda s}} = e^{-\lambda t} = P(T > t)$

WHERE does this matter? Exponential is the ONLY continuous distribution with this property. Models radioactive decay, time between server requests, or failure of components with constant hazard rate.

Deriving the Mean:

$E[T] = \int_0^\infty t \cdot \lambda e^{-\lambda t} \, dt$

Use integration by parts: $u = t$ , $dv = \lambda e^{-\lambda t} dt$

$= \left[-t e^{-\lambda t}\right]_0^\infty + \int_0^\infty e^{-\lambda t} \, dt =0 + \frac{1}{\lambda} = \frac{1}{\lambda}$

WHY this makes sense? If events happen at rate $\lambda = 5$ per hour, average time between events is $\frac{1}{5}$ hour = 12 minutes.

Worked example Exponential in A/B Testing

Problem: Users arrive at rate $\lambda = 0.5$ per second. What's the probability the next user arrives within 3 seconds? What's the 95th percentile wait time?

Solution:

$P(T \leq 3) = 1 - e^{-0.5 \cdot 3} = 1 - e^{-1.5} \approx 1 - 0.223 = 0.777$ WHY this step? Plug $t=3$ into the CDF formula directly.
For95th percentile: Solve $1 - e^{-0.5t} = 0.95$
$e^{-0.5t} = 0.05$
$-0.5t = \ln(0.05) = -2.996$
$t = 5.99$ seconds WHY this matters? In 95% of cases, the next user arrives within 6 seconds. Used for setting timeouts and SLA targets.

The Beta Distribution

Derivation from the Conjugate Prior Perspective

WHY does Beta exist? Suppose we flip a coin with unknown bias $p$ . After observing $\alpha - 1$ heads and $\beta - 1$ tails, what distribution describes our belief about $p$ ?

WHAT form should it take? We want:

Support on $[0, 1]$ (since $p$ is a probability)
Flexibility to represent different beliefs (uniform ignorance, strong certainty, etc.)
Mathematical convenience (conjugate to Binomial)

The Beta function normalizes the distribution:

$B(\alpha, \beta) = \int_0^1 x^{\alpha-1}(1-x)^{\beta-1} \, dx = \frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)}$

where $\Gamma(n) = (n-1)!$ for integers.

Understanding Shape via Parameters

HOW do $\alpha$ and $\beta$ control the shape?

$\alpha = \beta = 1$ : $\text{Beta}(1, 1) = \text{Uniform}(0, 1)$ — total ignorance
$\alpha > \beta$ : Distribution skewed toward 1 (more successes observed)
$\alpha < \beta$ : Distribution skewed toward 0 (more failures observed)
$\alpha, \beta > 1$ : Unimodal (single peak)
$\alpha, \beta < 1$ : U-shaped (bimodal, certainty near extremes)
Large $\alpha, \beta$ : Narrow distribution (high confidence)

Deriving the Mean:

$E[X] = \int_0^1 x \cdot \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha, \beta)} \, dx = \frac{1}{B(\alpha, \beta)} \int_0^1 x^\alpha (1-x)^{\beta-1} \, dx$

The integral is $B(\alpha+1, \beta)$ :

$= \frac{B(\alpha+1, \beta)}{B(\alpha, \beta)} = \frac{\Gamma(\alpha+1)\Gamma(\beta)/\Gamma(\alpha+\beta+1)}{\Gamma(\alpha)\Gamma(\beta)/\Gamma(\alpha+\beta)}$

$= \frac{\alpha \cdot \Gamma(\alpha)}{(\alpha+\beta) \cdot \Gamma(\alpha+\beta)} = \frac{\alpha}{\alpha+\beta}$

WHY this makes sense? If you observe $\alpha-1$ heads and $\beta-1$ tails, you expect the true probability to be near $\frac{\alpha-1}{\alpha+\beta-2}$ , which for large counts approaches $\frac{\alpha}{\alpha+\beta}$ .

Worked example Beta in Bayesian A/B Testing

Problem: We're testing a new "Buy Now" button. Start with prior $\text{Beta}(2, 2)$ (slight belief in 50% click rate). Observe 8 clicks and 12 non-clicks. What's the posterior distribution?

Solution:

Prior: $\text{Beta}(2, 2)$ — $E[p] = \frac{2}{4} = 0.5$ , fairly uncertain
Likelihood: 8 successes, 12 failures (Binomial)
Posterior: $\text{Beta}(2+8, 2+12) = \text{Beta}(10, 14)$

WHY this works? Beta is conjugate to Binomial—just add counts!

Posterior mean: $\frac{10}{24} \approx 0.417$
Posterior mode: $\frac{10-1}{24-2} = \frac{9}{22} \approx 0.409$ Interpretation: We now believe the true click rate is around 41%, with more certainty than before.

Compare to maximum likelihood: MLE would give $\frac{8}{20} = 0.4$ , but ignores prior knowledge and is less robust with small samples.

Worked example Beta for Thompson Sampling

Problem: Multi-armed bandit with 3 slot machines. Each has unknown win probability. Use Thompson Sampling to balance exploration and exploitation. Setup:

Machine A: $\text{Beta}(5, 5)$ — tried 8 times, 4 wins
Machine B: $\text{Beta}(10, 2)$ — tried 10 times, 9 wins
Machine C: $\text{Beta}(2, 8)$ — tried 8 times, 1 win

Algorithm: Each round, sample $p_A \sim \text{Beta}(5,5)$ , $p_B \sim \text{Beta}(10,2)$ , $p_C \sim \text{Beta}(2,8)$ . Pull the arm with highest sample.

WHY this works?

Machine B has high mean ( $\frac{10}{12} \approx 0.83$ ) → often sampled highest → exploited
But occasionally A or C sample higher → exploration happens naturally
More uncertainty (small $\alpha + \beta$ ) → more variable samples → more exploration

HOW does this beat epsilon-greedy? Thompson Sampling automatically adapts exploration—uncertain arms get tried more, confident arms get exploited more.

Connections Between the Three

Property	Uniform $(a, b)$	Exponential $(\lambda)$	Beta $(\alpha, \beta)$
Support	$[a, b]$	$[0, \infty)$	$[0, 1]$
Use case	No prior info	Waiting times	Probabilities
Memoryless?	No	Yes	No
Shape	Constant	Decreasing	Flexible
Special cases	Uniform $(0,1)$ = Beta $(1,1)$	Exp $(1)$ = Gamma $(1,1)$	Beta $(1,1)$ = Uniform

Recall Feynman Explanation (for a12-year-old)

Imagine you have three magic dice: The Flat Die (Uniform): Every number has the exact same chance. Like a perfectly fair spinner—no cheating, every spot equally likely. We use this when we have zero information and want to be completely fair.

The Forgetful Die (Exponential): This models waiting. Like waiting for the next bus. The weird thing? It doesn't matter how long you've waited—your chance of the bus coming in the next minute is ALWAYS the same. It's like the universe has amnesia! This works for things like radioactive atoms decaying or customers arriving at a store randomly.

The Belief Die (Beta): This one is about guessing probabilities. Imagine you're trying to figure out if a coin is fair. You flip it a few times. The Beta distribution describes how confident you are about what the true "heads probability" is. Get more data? The Beta updates, becoming more confident and accurate. It's like your brain getting smarter with each flip!

Connections

Continuous Random Variables and PDFs
Bayesian Inference and Conjugate Priors
Maximum Likelihood Estimation
Poisson Process
Thompson Sampling
Xavier and He Initialization
Monte Carlo Methods

#flashcards/ai-ml

What are the support (valid ranges) for Uniform, Exponential, and Beta distributions? :: Uniform: $[a, b]$ (finite interval), Exponential: $[0, \infty)$ (non-negative reals), Beta: $[0, 1]$ (unit interval)

What is the PDF of Uniform

(a, b)

f(x) = \frac{1}{b-a}

for

a \leq x \leq b

, zero otherwise. The constant ensures total probability = 1.

What is the mean and variance of Uniform $(a, b)$ ? :: Mean: $\frac{a+b}{2}$ (midpoint), Variance: $\frac{(b-a)^2}{12}$ (grows with square of range)

What is the memoryless property of the exponential distribution?

P(T > s+t \mid T > s) = P(T > t)

— probability of waiting additional time

t

is independent of how long you've already waited.

What is the PDF of Exponential

(\lambda)

f(t) = \lambda e^{-\lambda t}

for

t \geq 0

, where

\lambda

is the rate parameter (events per unit time).

What is the relationship between rate

\lambda

and mean waiting time in Exponential?

Mean

E[T] = \frac{1}{\lambda}

. Higher rate → shorter average wait. If

\lambda = 4

events/hour, average wait is

\frac{1}{4}

hour = 15 minutes.

What is the CDF of Exponential

(\lambda)

F(t) = 1 - e^{-\lambda t}

for

t \geq 0

. Used to find probability of event occurring within time

t

What is the PDF of Beta

(\alpha, \beta)

f(x) = \frac{1}{B(\alpha, \beta)} x^{\alpha-1}(1-x)^{\beta-1}

for

x \in [0,1]

, where

B(\alpha, \beta) = \frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)}

is the normalizing Beta function.

What is the mean of Beta

(\alpha, \beta)

E[X] = \frac{\alpha}{\alpha + \beta}

. If

\alpha = 5, \beta = 3

, mean =

\frac{5}{8} = 0.625

How does Beta

(\alpha, \beta)

update with new Binomial data?

If prior is Beta

(\alpha, \beta)

and you observe

k

successes in

n

trials, posterior is Beta

(\alpha + k, \beta + n - k)

. This is conjugacy!

What does Beta

(1, 1)

equal?

Uniform

(0, 1)

. It represents complete ignorance about a probability—every value

[0,1]

equally likely.

What happens to Beta

(\alpha, \beta)

shape when both parameters are large?

Distribution becomes more concentrated (lower variance), representing high confidence. Example: Beta

(100, 100)

is tightly peaked at 0.5.

What does

\alpha > \beta

indicate in Beta

(\alpha, \beta)

Distribution skewed toward 1 (right-skewed), indicating more evidence for successes than failures. Mean

> 0.5

Why is Uniform

(0,1)

important in simulation?

Most random number generators produce Uniform

(0,1)

samples. These can be transformed to ANY other distribution using inverse transform sampling:

X = F^{-1}(U)

where

U \sim \text{Uniform}(0,1)

What is the mode of Beta

(\alpha, \beta)

when

\alpha, \beta > 1

Mode

= \frac{\alpha - 1}{\alpha + \beta - 2}

. This is the peak of the distribution, the most likely value.

Why is Exponential the only memoryless continuous distribution?

The memoryless property

P(X > s+t \mid X > s) = P(X > t)

uniquely characterizes the exponential distribution through a functional equation that forces the form

e^{-\lambda t}

What ML application uses Uniform for initialization?

Xavier/Glorot initialization uses Uniform

(-\sqrt{\frac{6}{n_{in}+n_{out}}}, \sqrt{\frac{6}{n_{in}+n_{out}}})

to initialize neural network weights, keeping gradient variance stable.

What is Thompson Sampling and how does Beta help?

Thompson Sampling for multi-armed bandits: maintain Beta

(\alpha_i, \beta_i)

for each arm

i

, sample from each Beta, pull arm with highest sample. Beta naturally balances exploration (uncertain arms vary more) and exploitation (high-mean arms favored).

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Chalo is baat ko simple tarike se samajhte hain. Ye teen distributions ML mein bahut kaam aati hain, aur har ek ka apna "personality" hai. Uniform distribution ka matlab hai "koi favouritism nahi" — jaise interval [a, b] ke andar har point equally likely hai. Isliye jab humein kuch nahi pata hota, ya jab hum neural network ke weights randomly initialize karte hain, tab Uniform kaam aata hai. Iska density constant hota hai, aur wo constant $\frac{1}{b-a}$ isliye aata hai taaki total probability exactly 1 ban jaye — kyunki koi bhi PDF ka area 1 hona chahiye. Mean midpoint pe aata hai (obvious hai, symmetric jo hai), aur variance range ke square pe depend karta hai — matlab jitna wide interval, utni zyada uncertainty.

Exponential distribution ki soul hai "waiting time" — agli event kab aayegi. Jaise agla customer kab aayega, ya component kab fail hoga, ya neuron kab spike karega. Iska derivation Poisson process se aata hai: agar events rate $\lambda$ pe hoti hain, to "time $t$ tak koi event nahi" ki probability $e^{-\lambda t}$ hoti hai. Isko 1 se minus karke CDF milta hai, aur usko differentiate karke PDF $\lambda e^{-\lambda t}$ nikalta hai. Iski sabse mast property hai memorylessness — matlab past ka koi asar nahi, distribution ko yaad nahi rehta ki tum kitni der se wait kar rahe ho. Ye continuous version hai geometric distribution ka.

Aur Beta distribution to sabse interesting hai — ye khud probability ki probability model karta hai! Matlab jab tum kisi probability ke baare mein hi unsure ho (jaise coin kitna biased hai, ya click-through rate kya hoga), tab Beta tumhara "belief" represent karta hai 0 se 1 ke beech. Bayesian ML mein iska bada role hai kyunki ye Bernoulli/Binomial ka conjugate prior hai — iska matlab jab tum naya data dekhte ho, to update karna mathematically clean aur elegant ho jaata hai. Ye teeno distributions samajhna isliye zaroori hai kyunki random initialization, sampling, aur Bayesian reasoning — ye sab ML ke core foundations hain, aur inke bina modern algorithms ki intuition adhoori reh jaati hai.

Uniform, Exponential, and Beta distributions

Core Intuition

The Uniform Distribution

Derivation from First Principles

Key Properties (Derived)

The Exponential Distribution

Derivation from Poisson Process

The Memoryless Property

The Beta Distribution

Derivation from the Conjugate Prior Perspective

Understanding Shape via Parameters

Connections Between the Three

Connections

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Probability & Statistics

Connections