Bias, fairness, and discrimination metrics

Core Concepts

WHY it matters: A "fair" model to a statistician (low error rate) can be deeply unfair to a sociologist (disparate impact across groups). We need multiple mathematical definitions because fairness is context-dependent.

The Impossibility of Perfect Fairness

KEY INSIGHT: Most fairness metrics are mutually exclusive. You cannot satisfy all simultaneously except in trivial cases.

1. Demographic Parity (Statistical Parity) $P(\hat{Y} = 1 | A = 0) = P(\hat{Y} = 1 | A = 1)$ WHERE:

$\hat{Y}$ : model prediction (1 = positive outcome, e.g., "approve loan")
$A$ : protected attribute (0/1, e.g., Group A/B)

WHY this definition: Equal acceptance rates across groups. Ensures representation in positive outcomes.

DERIVATION: Start with the fairness principle "the chance of a positive prediction should not depend on group membership." Translate to probability: the positive-prediction rate must be the same across groups. Formally, for any two groups $a$ and $a'$ : $P(\hat{Y} = 1 | A = a) = P(\hat{Y} = 1 | A = a')$ This is equality across groups, NOT equality to the marginal rate $P(\hat{Y}=1)$ . (The marginal is just the population-weighted average of the group rates; requiring each group to equal it is only an equivalent statement when there are exactly two groups, but stating it as " $= P(\hat Y=1)$ " is misleading and we avoid it.)

2. Equalized Odds (Error Rate Balance) $P(\hat{Y} = 1 | Y = y, A = 0) = P(\hat{Y} = 1 | Y = y, A = 1) \quad \forall y \in \{0,1\}$

WHY this definition: Equal true positive rates (TPR) AND equal false positive rates (FPR) across groups. Ensures errors don't concentrate in one group.

DERIVATION: Consider the confusion matrix for each group. For fair treatment, both "when we're right" (TPR) and "when we're wrong" (FPR) should match: \begin{align} \text{TPR}_A &= \frac{TP_A}{TP_A + FN_A} = \text{TPR}_B \ \text{FPR}_A &= \frac{FP_A}{FP_A + TN_A} = \text{FPR}_B \end{align}

3. Predictive Parity (Calibration) $P(Y = 1 | \hat{Y} = 1, A = 0) = P(Y = 1 | \hat{Y} = 1, A = 1)$

WHY this definition: Equal precision across groups. If the model says "approve," it should be equally trustworthy for both groups.

DERIVATION: Precision measures "of those we predicted positive, how many were truly positive?" For fairness, this success rate should be group-independent: $\text{Precision}_A = \frac{TP_A}{TP_A + FP_A} = \text{Precision}_B$

Why These Metrics Conflict (The Math)

SETUP: Two groups with different base rates:

Group A: $P(Y=1|A=0) = 0.5$ (50% actually qualify)
Group B: $P(Y=1|A=1) = 0.3$ (30% actually qualify)

CLAIM: We cannot satisfy both Equalized Odds AND Predictive Parity when base rates differ (and the classifier is imperfect).

PROOF (the exact identity that forces the conflict): For each group, precision (PPV), FPR, and base rate $b = P(Y=1|A=a)$ are linked by an algebraic identity. Start from the definition of FPR and TPR and expand $P(\hat{Y}=1)$ by the law of total probability: $P(\hat{Y}=1|A=a) = \underbrace{\text{TPR}\cdot b}_{\text{true positives}} + \underbrace{\text{FPR}\cdot(1-b)}_{\text{false positives}}$ Precision is true positives over all positives: $\text{PPV} = \frac{\text{TPR}\cdot b}{\text{TPR}\cdot b + \text{FPR}\cdot(1-b)}$ Solving for FPR in terms of PPV, TPR, and $b$ gives the Chouldechova identity: $\text{FPR} = \frac{b}{1-b}\cdot\frac{1-\text{PPV}}{\text{PPV}}\cdot\text{TPR}$

WHY this matters: Suppose we impose Equalized Odds, so TPR and FPR are equal across both groups. Then the equation forces $\frac{b_A}{1-b_A}\cdot\frac{1-\text{PPV}_A}{\text{PPV}_A} = \frac{b_B}{1-b_B}\cdot\frac{1-\text{PPV}_B}{\text{PPV}_B}$ Since $b_A = 0.5 \neq 0.3 = b_B$ , the base-rate factors $\frac{b}{1-b}$ differ ( $1.0$ vs $0.43$ ), so $\text{PPV}_A \neq \text{PPV}_B$ . Equal error rates therefore force unequal precision — Predictive Parity is violated. QED.

(Note: we do NOT assume perfect classification. A perfect classifier would make $\hat Y = Y$ , which trivially satisfies everything but never equalizes prediction rates when base rates differ — so the interesting, real-world case is the imperfect classifier above.)

Practical Metrics & Computation

$\text{Disparate Impact} = \frac{P(\hat{Y}=1|A=\text{protected})}{P(\hat{Y}=1|A=\text{privileged})}$

Legal threshold: Ratio $< 0.8$ suggests discrimination (U.S. EEOC guideline).

EXAMPLE CALCULATION:

Model approves 60% of Group A applications
Model approves 40% of Group B applications
Disparate Impact = $\frac{0.40}{0.60} = 0.67 < 0.8$ ❌ Fails fairness test

WHY 0.8? Empirical threshold balancing statistical variation vs. systematic bias. Not a sharp boundary, but a screening tool.

$d(\hat{Y}_i, \hat{Y}_j) \leq L \cdot d(X_i, X_j)$

WHERE:

$d(\hat{Y}_i, \hat{Y}_j)$ : difference in predictions
$d(X_i, X_j)$ : distance in feature space
$L$ : Lipschitz constant (sensitivity bound)

WHY this definition: "Similar individuals should get similar predictions." Prevents arbitrary discrimination between near-identical applicants.

DERIVATION: Formalize "treat similar people similarly" as a smoothness constraint. Borrow from Lipschitz continuity in analysis: output change bounded by input change scaled by $L$ .

EXAMPLE: Two applicants with identical credit scores, income, debt:

Applicant 1: Predicted score 0.72
Applicant 2: Predicted score 0.39
Feature distance: $||X_1 - X_2|| = 0.01$ (tiny difference in ZIP code)

If $L = 5$ , required: $|0.72 - 0.39| \leq 5 \cdot 0.01 = 0.05$ . But $0.33 > 0.05$ ❌ Violates individual fairness.

WHY this step? Exposes when tiny, irrelevant feature changes cause huge prediction swings—a red flag for hidden bias.

Measuring Group Fairness: Worked Example

SCENARIO: A bank uses ML to predict who will default on a loan. Here $\hat{Y}=1$ means "predicted to default" (a negative outcome for the applicant — those flagged are denied). We audit on labeled historical data where $Y=1$ means the applicant actually defaulted.

To avoid confusing "approvals" with "predicted defaults," we lay out the full confusion matrix per group.

Group A (1000 applicants, 200 actual defaulters):

	Actually defaulted ( $Y=1$ )	Did not default ( $Y=0$ )	Row total
Predicted default ( $\hat{Y}=1$ )	$TP_A = 180$	$FP_A = 120$	$300$
Predicted no default ( $\hat{Y}=0$ )	$FN_A = 20$	$TN_A = 680$	$700$
Column total	$200$	$800$	$1000$

Group B (1000 applicants, 300 actual defaulters):

	Actually defaulted ( $Y=1$ )	Did not default ( $Y=0$ )	Row total
Predicted default ( $\hat{Y}=1$ )	$TP_B = 270$	$FP_B = 130$	$400$
Predicted no default ( $\hat{Y}=0$ )	$FN_B = 30$	$TN_B = 570$	$600$
Column total	$300$	$700$	$1000$

STEP 1: Demographic Parity — rate of being flagged (predicted default): $P(\hat{Y}=1|A=0) = \frac{300}{1000} = 0.30, \qquad P(\hat{Y}=1|A=1) = \frac{400}{1000} = 0.40$ WHY they differ? Group B is flagged 33% more often. Since flagging leads to denial, this is a red flag for disparate impact. (Why this step? We divide the row total of predicted-positives by group size — never mixing with true labels.)

STEP 2: Equalized Odds — TPR uses the $Y=1$ column, FPR uses the $Y=0$ column: $\text{TPR}_A = \frac{TP_A}{TP_A+FN_A} = \frac{180}{200} = 0.90, \qquad \text{TPR}_B = \frac{270}{300} = 0.90 \;\checkmark$ $\text{FPR}_A = \frac{FP_A}{FP_A+TN_A} = \frac{120}{800} = 0.15, \qquad \text{FPR}_B = \frac{130}{700} = 0.186$ WHY this matters: TPRs match (0.90 = 0.90) but FPRs differ (0.15 vs 0.186) — Group B's non-defaulters get wrongly flagged more often. Equalized Odds is partially violated on FPR. (Why this step? TPR is always $TP/(TP+FN)$ , never a row total — the earlier "impossible TPR > 1" bug came from dividing by the wrong denominator.)

STEP 3: Predictive Parity — precision uses the predicted-positive row: $\text{Precision}_A = \frac{TP_A}{TP_A+FP_A} = \frac{180}{300} = 0.60, \qquad \text{Precision}_B = \frac{270}{400} = 0.675$ WHY they differ? Of those flagged, 60% of Group A truly default vs 67.5% of Group B. (Why this step? Precision divides by total predicted positives = $TP+FP$ = the row total 300 or 400 — NOT by the number of approvals or the group size.)

CONCLUSION: Fails Demographic Parity, fails FPR-part of Equalized Odds, and shows unequal precision. Because base rates differ (20% vs 30% default), the Chouldechova identity guarantees we cannot fix all three at once. Investigate:

Base-rate difference (Group B defaults more often) — is it real or a proxy artifact?
Feature quality (missing data for Group A?)
Historical bias in who ended up in the labeled dataset.

Common Mistakes & Fixes

WHY IT FEELS RIGHT: If race isn't in the dataset, the model can't discriminate by race.

WHY IT'S WRONG: Proxy variables (ZIP code, name, school) correlate with race. The model learns indirect discrimination.

THE FIX:

Measure fairness metrics on protected groups even if not used in training
Use adversarial debiasing: train a secondary model to predict the protected attribute from your model's hidden layers. Penalize if it succeeds (forces model to "forget" group information).

FORMULA: $\mathcal{L} = \mathcal{L}_{\text{task}} - \lambda \mathcal{L}_{\text{adversary}}$ WHERE $\mathcal{L}_{\text{adversary}}$ tries to predict $A$ from hidden representations.

WHY IT FEELS RIGHT: Same error rate for everyone.

WHY IT'S WRONG: Accuracy mixes TPR and TNR. You can have equal accuracy with wildly different error types. Example:

Group A: 90% accuracy (95% TPR, 85% TNR)
Group B: 90% accuracy (70% TPR, 99% TNR)

Group B gets fewer false alarms but misses more true positives. If the outcome is medical diagnosis, Group B's cancer goes undetected.

THE FIX: Report disaggregated metrics (TPR, FPR, precision, recall) per group, not just accuracy.

WHY IT FEELS RIGHT: Dividing $TP$ by the group's total feels like "how many did we get right out of everyone."

WHY IT'S WRONG: That's not precision — it's the positive detection rate scaled weirdly. Precision must divide by the row total of predicted positives ( $TP+FP$ ). Confusing this with approvals or group size produces nonsense (like the impossible TPR $> 1$ bug where $270/300$ was mistakenly written as a rate over a wrong denominator).

THE FIX: Anchor every metric to its correct slice of the confusion matrix:

TPR $= TP/(TP+FN)$ → divide by the $Y=1$ column
FPR $= FP/(FP+TN)$ → divide by the $Y=0$ column
Precision $= TP/(TP+FP)$ → divide by the $\hat Y=1$ row

Mitigation Strategies

Assign instance weights to balance representation: $w_i = \frac{P(A=a)\,P(Y=y)}{P(A=a, Y=y)}$

WHY: Upweight underrepresented (group, label) pairs. Forces the model to pay attention to minority cases.

DERIVATION: We want training to "see" $A$ and $Y$ as independent. If Group B with $Y=1$ appears rarely, weight each such instance higher so the weighted joint equals the product of marginals: $w_i \cdot P(A=a, Y=y) = P(A=a)\,P(Y=y)$

Choose different decision thresholds per group to satisfy fairness constraints: $\hat{Y}_a = \mathbb{1}[\hat{p}(X) > \tau_a]$

EXAMPLE:

Require Equalized Odds
Group A: Set $\tau_A = 0.5$ to get TPR=0.85
Group B: Set $\tau_B = 0.4$ to also get TPR=0.85

WHY legal? If justified by base rate differences and doesn't perpetuate historical bias.

When to Use Which Metric

Metric	Use Case	Example
Demographic Parity	Equal representation in opportunities	College admissions
Equalized Odds	Equal error rates (justice system)	Bail decisions, recidivism
Predictive Parity	Trust in positive predictions	Medical diagnosis
Individual Fairness	Prevent arbitrary local discrimination	Insurance quotes

WHY context matters: The "right" metric depends on the harm model. False negatives in cancer screening ≠ false negatives in spam detection.

Recall Feynman Explanation (ELI12)

Imagine you and your friend both apply for the school soccer team. You both run the same speed, kick equally well. But the coach has a "scoring system" that gives extra points for height. You're short, your friend is tall. Is that fair? Demographic Parity says: "Same percentage of short and tall kids should make the team." Sounds fair, but what if taller kids really are better goalkeepers?

Equalized Odds says: "Of the kids who'd be great players (truth), the coach should pick the same fraction of short and tall. Of the kids who'd be mediocre (truth), the coach should wrongly pick the same fraction." This is like saying "make the same types of mistakes for both groups."

Predictive Parity says: "When the coach says 'you'll be a star,' that should be equally accurate for short and tall kids." If the coach is right 80% of the time for tall kids but only 50% for short kids, that's unfair—short kids get false hope or unfair doubt.

The wild part? You can't satisfy all three at once if short and tall kids have different actual skill distributions. Math proves it. So we have to choose which fairness we care about most for each situation.

Connections

Algorithmic accountability: Fairness metrics are the quantitative foundation for accountability frameworks
Disparate impact theory: Legal concept formalized as the 80% rule
Counterfactual fairness: Asks "would outcome change if protected attribute flipped?"
Intersectionality in ML: Fairness across combinations of protected attributes (race × gender)
COMPAS risk assessment: Real-world case study violating Predictive Parity
Fairness-accuracy tradeoffs: Theoretical limits on achieving both
Causal fairness: Using causal graphs to distinguish legitimate vs. discriminatory paths

#flashcards/ai-ml

What is the 80% rule (disparate impact) :: A practical fairness threshold: selection rate for protected group / selection rate for privileged group ≥ 0.8. Below 0.8 suggests discrimination (U.S. EEOC guideline).

Define Demographic Parity mathematically :: $P(\hat{Y}=1|A=0) = P(\hat{Y}=1|A=1)$ . Equal positive-prediction rates across groups (not equal to the marginal rate), regardless of ground truth.

Define Equalized Odds :: $P(\hat{Y}=1|Y=y, A=0) = P(\hat{Y}=1|Y=y, A=1)$ for both $y=0$ and $y=1$ . Equal TPR and FPR across groups.

Define Predictive Parity :: $P(Y=1|\hat{Y}=1, A=0) = P(Y=1|\hat{Y}=1, A=1)$ . Equal precision (PPV) across groups.

What identity forces Equalized Odds and Predictive Parity to conflict? :: The Chouldechova identity $\text{FPR} = \frac{b}{1-b}\cdot\frac{1-\text{PPV}}{\text{PPV}}\cdot\text{TPR}$ . If base rates $b$ differ but TPR and FPR are equalized, PPV must differ.

What is Individual Fairness (Lipschitz condition) :: $d(\hat{Y}_i, \hat{Y}_j) \leq L \cdot d(X_i, X_j)$ . Similar individuals (in feature space) receive similar predictions. Bounds output sensitivity to input perturbations.

What are proxy variables in bias :: Features correlated with protected attributes (e.g., ZIP code ↔ race). Removing the protected attribute from training doesn't prevent discrimination if proxies remain.

Name three sources of ML bias :: (1) Historical bias (training data reflects past discrimination), (2) Representation bias (undersampled groups), (3) Measurement bias (label/feature quality varies by group).

How do you compute precision (PPV) from a confusion matrix? :: $\text{Precision} = TP/(TP+FP)$ — divide true positives by the total predicted positives (the $\hat Y=1$ row), never by group size or number of approv

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho yaar, is note ka core idea ye hai ki jab hum machine learning models banate hain—jaise loan approve karna ya scholarship dena—tab model kabhi-kabhi kuch groups ke saath unfair treatment karta hai, chahe hum usse race ya gender directly na bhi batayein. Kyun? Kyunki model proxy variables seekh leta hai—jaise neighborhood ya "cultural fit" score—jo secretly protected attributes ke saath correlate karte hain. Iska matlab historical discrimination model ke andar chupke se ghus jaata hai. Isliye humein fairness metrics chahiye—ye mathematical definitions hain jo is chupi hui bias ko measure karti hain, taaki hum ethical principles ko actual engineering constraints mein convert kar sakein.

Ab yahan sabse important baat samajh lo: fairness ki ek nahi, teen alag-alag definitions hain—Demographic Parity (dono groups ka acceptance rate barabar ho), Equalized Odds (errors yaani TPR aur FPR dono groups mein barabar ho), aur Predictive Parity (jab model "approve" bole toh dono groups ke liye equally trustworthy ho). Har definition apni jagah sahi hai, but context ke hisaab se important hoti hai. Ek statistician ke liye "fair" model ka matlab low error hai, lekin ek sociologist ke liye fair matlab groups ke beech disparate impact na ho. Yehi wajah hai ki hum sirf ek metric pe rely nahi kar sakte.

Aur asli twist ye hai—jo Chouldechova ka Impossibility Theorem kehta hai—ki ye metrics mutually exclusive hain. Jab do groups ke base rates alag hote hain (jaise ek group mein 50% qualify karte hain aur dusre mein 30%), tab tum Equalized Odds aur Predictive Parity dono ek saath satisfy nahi kar sakte, agar classifier perfect nahi hai. Ye ek mathematical fact hai, koi engineering limitation nahi. Isliye ye matter karta hai: real-world mein tumhe choose karna padega ki konsi fairness apne use-case ke liye zyada zaroori hai—tum sabko ek saath khush nahi kar sakte, aur is trade-off ko samajhna hi ek responsible AI engineer banne ki pehli seedhi hai.