Sample spaces, events, and axioms of probability

What is a sample space?

Why do we need this? Without defining "all possibilities," we can't measure "how likely" something is. It's like asking "what fraction?" without saying "fraction of what?"

Examples of sample spaces

What is an event?

Why subsets? Because we often care about groups of outcomes. "Rolling an even number" is the event {2, 4, 6}, not a single outcome.

Types of events

• Simple event: single outcome, e.g., {H} or {(3,5)}
• Compound event: multiple outcomes, e.g., {sum≥ 10} = {(4,6), (5,5), (5,6), (6,4), (6,5), (6,6)}
• Certain event: Ω itself (probability = 1)
• Impossible event: ∅ (probability = 0)

Define event A = "sum is 7" $A = \{(1,6), (2,5), (3,4), (4,3), (5,2), (6,1)\}$ $|A| = 6$

Define event B = "at least one die shows 6" $B = \{(6,1), (6,2), .., (6,6), (1,6), (2,6), .., (5,6)\}$ $|B| = 11$ Why 11, not 12? (6,6) is counted once, not twice (inclusion-exclusion principle)

Event operations:

• A ∪ B = "sum is 7 OR at least one 6" (union)
• A ∩ B = {(1,6), (6,1)} = "sum is 7 AND at least one 6" (intersection)
• A^c = "sum is NOT 7" (complement, all outcomes except A)

The three axioms of probability

Axiom 1 (Non-negativity): $P(E) \geq 0 \quad \forall E \subseteq \Omega$

Axiom 2 (Normalization): $P(\Omega) = 1$

Axiom 3 (Countable Additivity): For mutually exclusive events $E_1, E_2, ..$: $P(E_1 \cup E_2 \cup ..) = P(E_1) + P(E_2) + ...$ (if $E_i \cap E_j = \emptyset$ for $i \neq j$)

Why these three?

Axiom 1 WHY: Probabilities are fractions of certainty. Negative probability has no physical meaning. You can't be "−20% sure."

Axiom 2 WHY: SOMETHING must happen. If you flip a coin, you get heads or tails (assuming it doesn't land on edge, which is outside our model). The total probability of all possible outcomes is certainty = 100% = 1.

Axiom 3 WHY: If events can't happen together (mutually exclusive), the probability of "A or B" is just "add them up." If you can't roll (1,1) AND (2,3) simultaneously, then P({(1,1)} ∪ {(2,3)}) = P({(1,1)}) + P({(2,3)}).

What about non-mutually-exclusive events? That's NOT Axiom 3. We derive it:

Derivation:

1. Write $A \cup B = A \cup (B \cap A^c)$ (split B into "B-only" part)
2. These are disjoint: $A$ and $B \cap A^c$ don't overlap
3. By Axiom 3: $P(A \cup B) = P(A) + P(B \cap A^c)$
4. But $B = (B \cap A) \cup (B \cap A^c)$, disjoint again
5. So $P(B) = P(B \cap A) + P(B \cap A^c)$
6. Rearrange: $P(B \cap A^c) = P(B) - P(A \cap B)$
7. Substitute into step 3: $P(A \cup B) = P(A) + P(B) - P(A \cap B)$ ∎

Why this step? We had to avoid double-counting the overlap $A \cap B$.

Deriving useful properties from axioms

Proof:

1. $A$ and $A^c$ are mutually exclusive: $A \cap A^c = \emptyset$
2. $A \cup A^c = \Omega$ (every outcome is either in A or not)
3. By Axiom 2: $P(\Omega) = 1$
4. By Axiom 3: $P(A \cup A^c) = P(A) + P(A^c)$
5. Combine: $1 = P(A) + P(A^c)$
6. Rearrange: $P(A^c) = 1 - P(A)$ ∎

ML intuition: If your spam classifier says P(spam) = 0.7, then P(not-spam) = 0.3. They must sum to 1.

Proof:

1. Write $B = A \cup (B \cap A^c)$ (B is A plus extra stuff)
2. These are disjoint
3. By Axiom 3: $P(B) = P(A) + P(B \cap A^c)$
4. By Axiom 1: $P(B \cap A^c) \geq 0$
5. Therefore: $P(B) \geq P(A)$ ∎

Intuition: "Bigger events can't have smaller probability." If B contains all of A plus more, B is at least as likely.

Computing probabilities in finite sample spaces

For finite Ω with equally likely outcomes (like fair dice, random draws):

Why this works:

1. Each outcome ω has the same probability: $P(\{\omega\}) = \frac{1}{|\Omega|}$
2. Event E is a union of disjoint outcomes: $E = \{\omega_1\} \cup \{\omega_2\} \cup ...$
3. By Axiom 3: $P(E) = \sum_{\omega \in E} P(\{\omega\}) = |E| \cdot \frac{1}{|\Omega|}$

Why this step? Each of the 36 outcomes is equally likely (fair dice). The event "sum=7" contains 6 of them.

Estimated: $P(\text{spam}) \approx \frac{300}{1000} = 0.3$ $P(\text{not-spam}) \approx \frac{700}{1000} = 0.7$

Check: $0.3 + 0.7 = 1$ ✓ (Axiom 2)

ML connection: This is maximum likelihood estimation for a Bernoulli distribution.

Common mistakes

Why it feels right: For disjoint events, this IS true (Axiom 3). Beginers overgeneralize.

The fix: Check if events overlap. If $A \cap B \neq \emptyset$, use inclusion-exclusion: $P(A \cup B) = P(A) + P(B) - P(A \cap B)$

Example: Two dice, A = "first die is 6", B = "sum is 10"

• $P(A) = 6/36 = 1/6$
• $P(B) = 3/36 = 1/12$ (pairs: (4,6), (5,5), (6,4))
• $A \cap B = \{(6,4)\}$, so $P(A \cap B) = 1/36$
• $P(A \cup B) = 1/6 + 1/12 - 1/36 = 6/36 + 3/36 - 1/36 = 8/36 = 2/9$

If we'd just added: $1/6 + 1/12 = 1/4 \neq 2/9$ ✗

Why it feels right: Mathematically, |Ω| = 2. Classical probability gives 1/2.

The fix: Equally likely outcomes require symmetry (fair coin, fair die random draw). Weather is NOT symmetric. The axioms allow non-uniform probabilities. We need data or a model.

The fix: Axiom 1 and 2 bound P(E) ∈ [0, 1]. If you compute something outside this range, you made an algebra error or your model is incoherent.

Connections to ML/AI

• Conditional Probability builds on events: P(A|B) requires understanding what events A, B mean
• Bayes' Theorem rests on these axioms; without them, Bayesian inference is undefined
• Random Variables are functions from Ω to ℝ; the sample space is the domain
• Probability Distributions assign probabilities to events in a structured way
• Maximum Likelihood Estimation uses frequentist probability: P(data|model) as |favorable|/|total| in the limit
• Classification Metrics (precision, recall) are conditional probabilities over confusion matrix events
• Entropy and Information measures uncertainty over a probability distribution built from these axioms

The sample space is "all the marbles in the bag" — every possible one you could pick. That's 10 marbles total.

An event is something you care about, like "I picked a red marble" or "I picked red or blue." Events are just groups of marbles.

The three rules of probability (axioms) are:

1. Chances can't be negative. You can't have "minus 30% chance" of picking red — that's nonsense.
2. SOME marble will be picked (assuming you do pick). So the chance of "any marble at all" is 100%, which we write as 1.
3. If two events can't happen at the same time (like picking one marble that's both red AND blue — impossible), then the chance of "red OR blue" is just: chance of red + chance of blue.

That's it! These three simple rules let us calculate probabilities for everything in AI and ML.

Summary

• Sample space Ω: all possible outcomes, exhaustive and mutually exclusive
• Event E: a subset of Ω we assign probability to
• Three axioms: Non-negativity, normalization, additivity (for disjoint events)
• Derived properties: complement rule, inclusion-exclusion, monotonicity
• Classical probability: P(E) = |E|/|Ω| when outcomes are equally likely
• ML foundation: all probabilistic models (Bayes, logistic regression, generative models) rest on these axioms

#flashcards/ai-ml

Concept Map

Hinglish (regional understanding)