Common distributions (Bernoulli, Binomial, Poisson)

Overview

These three discrete probability distributions model different types of counting experiments. Understanding them is critical for classification, A/B testing, rare event prediction, and Bayesian inference in ML.

---

1. Bernoulli Distribution

PMF (Probability Mass Function): $P(X=x) = p^x (1-p)^{1-x}, \quad x \in \{0, 1\}$

Why this form? When $x=1$, we get $p^1 \cdot (1-p)^0 = p$. When $x=0$, we get $p^0 \cdot (1-p)^1 = 1-p$. The exponents "select" the right probability.

---

2. Binomial Distribution

PMF: $P(X=k) = \binom{n}{k} p^k (1-p)^{n-k}, \quad k = 0, 1, 2, \ldots, n$

Derivation from first principles:

1. Why $p^k (1-p)^{n-k}$? We want exactly $k$ successes and $n-k$ failures. If trials are independent, probability = (product of individual probabilities).

2. Why $\binom{n}{k}$? The $k$ successes can occur in any order. The number of ways to choose $k$ positions out of $n$ is $\binom{n}{k} = \frac{n!}{k!(n-k)!}$. Each arrangement has the same probability $p^k (1-p)^{n-k}$, so we multiply by the count.

---

3. Poisson Distribution

PMF: $P(X=k) = \frac{\lambda^k e^{-\lambda}}{k!}, \quad k = 0, 1, 2, \ldots$

Derivation as limit of Binomial:

Imagine $n$ very large, $p$ very small, but $np = \lambda$ (constant average rate).

Start with Binomial: $P(X=k) = \binom{n}{k} p^k (1-p)^{n-k}$

Substitute $p = \lambda/n$: $P(X=k) = \frac{n!}{k!(n-k)!} \left(\frac{\lambda}{n}\right)^k \left(1 - \frac{\lambda}{n}\right)^{n-k}$

Simplify: $= \frac{n(n-1)\cdots(n-k+1)}{n^k} \cdot \frac{\lambda^k}{k!} \cdot \left(1 - \frac{\lambda}{n}\right)^n \cdot \left(1 - \frac{\lambda}{n}\right)^{-k}$

As $n \to \infty$:

• $\frac{n(n-1)\cdots(n-k+1)}{n^k} \to 1$ (leading terms dominate)
• $\left(1 - \frac{\lambda}{n}\right)^n \to e^{-\lambda}$ (definition of $e$)
• $\left(1 - \frac{\lambda}{n}\right)^{-k} \to 1$

Result: $P(X=k) = \frac{\lambda^k e^{-\lambda}}{k!}$

Why does this make sense? Poisson is Binomial with "infinite trials, infinitesimal success probability, finite expected count." Perfect for rare events.

---

Connections Between Distributions

Distribution	When to Use	Parameters	Mean	Variance
Bernoulli	Single binary trial	$p$	$p$	$p(1-p)$
Binomial	Fixed # of trials	$n, p$	$np$	$np(1-p)$
Poisson	Rare events, continuous opportunities	$\lambda$	$\lambda$	$\lambda$

Key insight: Binomial → Poisson when $n \to \infty$, $p \to 0$, $np = \lambda$ constant.

---

Common Mistakes

---

ML Applications

1. Logistic Regression output: Bernoulli likelihood (model $P(y=1|x)$ for binary labels).
2. A/B Testing: Binomial models for conversion counts. Compare $p_A$ vs $p_B$.
3. Anomaly Detection: Poisson for rare events (fraud, system failures). If observed count >> $\lambda$, flag as anomaly.
4. Count Data Regression: Poisson regression for predicting counts (# of purchases, # of bugs).
5. Naive Bayes with binary features: Each feature is Bernoulli conditional on class.

---

Recall Feynman Explanation (ELI12)

Imagine you have a magic coin. When you flip it once, it either lands on heads (success) or tails (failure). That's Bernoulli – one flip, two outcomes.

Now, flip that same coin 10 times and count how many heads you get. That's Binomial – multiple flips, count the wins. The math tells you: "If my coin is50-50, what are the chances I get exactly 7 heads out of 10?" The formula has two parts: one says "what's the chance of this exact pattern (HHHHTT)?" and another part says "how many different ways can I arrange 7 heads in 10 flips?"

Poisson is for super rare things. Like, "How many typos are in a 1000-page book?" You don't flip 1000 coins—each word could have a typo, but it's really unlikely. Poisson is the shortcut when you have a bazillion chances for something rare to happen, and you just want to know "how many will I see?" The cool part: the average and the "spreadiness" are the same number!

All three are about counting, but pick the right one: one flip = Bernoulli, fixed flips = Binomial, rare stuff happening in a big space/time = Poisson.

---

[!mnemonic] BP Memory Hook "Bern-ONE, Bi-MANY, Poi-RARE"

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Chalo, in teen distributions ka core intuition samajhte hain simple language mein. Sabse basic hai Bernoulli — yeh ek single yes/no experiment hai, jaise ek email spam hai ya nahi, ek user click karega ya nahi. Bas do outcomes: success (probability $p$) ya failure (probability $1-p$). Iska mean simply $p$ hota hai, aur variance $p(1-p)$ — jo maximum tab hoti hai jab $p=0.5$, kyunki tab sabse zyada uncertainty hoti hai (dono outcomes equally likely). Jab $p$ 0 ya 1 ke paas ho, tab hum almost sure hote hain, toh variance kam.

Ab Binomial actually Bernoulli ka hi extension hai — agar tum ek Bernoulli trial ko $n$ baar repeat karo (jaise 100 emails bhejo), aur poocho "kitne successes mile?", toh wo Binomial hai. Iski PMF mein do parts hote hain: $p^k(1-p)^{n-k}$ ek specific arrangement ki probability deta hai, aur $\binom{n}{k}$ count karta hai ki $k$ successes kitne alag-alag orders mein aa sakte hain. Isiliye mean $np$ ban jaata hai (linearity of expectation se — bas $n$ Bernoullis ko add kar do) aur variance $np(1-p)$. Aur Poisson aata hai jab $n$ bahut bada ho par success rare ho (jaise per 1000 lines mein kitne bugs) — yeh Binomial ka hi limiting case hai.

Yeh matter isliye karta hai kyunki real ML problems mein tumhe constantly counting aur probability estimate karni padti hai — A/B testing mein "kitne users convert honge", classification mein "yeh spam hai ki nahi", ya fraud detection mein rare events predict karna. In distributions ko samajhne se tum sirf predictions nahi, balki uncertainty bhi quantify kar paate ho (variance se), jo real-world decisions mein critical hota hai. Bayesian inference aur probabilistic models ka toh yeh foundation hi hai, toh yahan strong base banana zaroori hai.