Logistic regression — sigmoid, cross-entropy loss

Overview

Logistic regression is the fundamental algorithm for binary classification problems in aerospace ML. Instead of predicting continuous values (like altitude), we predict probabilities (like "will this component fail?" or "is this satellite image cloudy?"). It maps any real-valued input to a probability between 0 and 1 using the sigmoid function, then trains by minimizing cross-entropy loss.

Core Components

The sigmoid function does exactly this. Think of it as a "probability translator" — it takes your model's raw confidence score and converts it to a valid probability.

1. The Sigmoid Function

Derivation from First Principles:

Why this specific form? Start with the odds ratio. If probability of class 1 is $p$, the odds are: $\text{odds} = \frac{p}{1-p}$

In logistic regression, we assume the log-odds are linear in features: $\log\left(\frac{p}{1-p}\right) = z = \mathbf{w}^T \mathbf{x} + b$

Why? This is the key assumption — the effect of features is multiplicative on odds, but additive on log-odds. Now solve for $p$:

$\frac{p}{1-p} = e^z$ $p = e^z(1-p)$ $p = e^z - pe^z$ $p(1 + e^z) = e^z$ $p = \frac{e^z}{1 + e^z}$

Multiply numerator and denominator by $e^{-z}$: $p = \frac{1}{e^{-z} + 1} = \sigma(z)$

Key Properties:

• $\sigma(0) = 0.5$ (decision boundary)
• $\sigma(z) \to 1$ as $z \to \infty$
• $\sigma(z) \to 0$ as $z \to -\infty$
• Derivative: $\sigma'(z) = \sigma(z)(1 - \sigma(z))$ (convenient for backprop!)

Derivative Derivation: $\frac{d}{dz}\sigma(z) = \frac{d}{dz}\left(\frac{1}{1+e^{-z}}\right)$ Using chain rule with $u = 1 + e^{-z}$: $= -\frac{1}{u^2} \cdot (-e^{-z}) = \frac{e^{-z}}{(1+e^{-z})^2}$ $= \frac{1}{1+e^{-z}} \cdot \frac{e^{-z}}{1+e^{-z}} = \sigma(z) \cdot \frac{e^{-z}}{1+e^{-z}}$ $= \sigma(z) \cdot \left(1 - \frac{1}{1+e^{-z}}\right) = \sigma(z)(1-\sigma(z))$

Why this step? The derivative form makes gradient calculations elegant — no exponentials in the gradient updates!

2. Cross-Entropy Loss

Cross-entropy from information theory does exactly this.

For $N$ training examples: $J(\mathbf{w}, b) = -\frac{1}{N}\sum_{i=1}^{N} [y^{(i)} \log(\hat{y}^{(i)}) + (1-y^{(i)}) \log(1-\hat{y}^{(i)})]$

Derivation from Maximum Likelihood:

Assume each label $y_i$ is drawn from Bernoulli distribution with probability $\hat{y}_i$: $P(y_i | \hat{y}_i) = \hat{y}_i^{y_i} (1-\hat{y}_i)^{1-y_i}$

Why this form? When $y_i=1$, this is $\hat{y}_i$. When $y_i=0$, this is $1-\hat{y}_i$. Perfect compact notation.

Likelihood of all data (assuming independence): $L = \prod_{i=1}^N \hat{y}_i^{y_i} (1-\hat{y}_i)^{1-y_i}$

Take log-likelihood (products become sums, easier to optimize): $\log L = \sum_{i=1}^N [y_i \log \hat{y}_i + (1-y_i) \log(1-\hat{y}_i)]$

Maximize log-likelihood = minimize negative log-likelihood: $J = -\frac{1}{N} \log L = -\frac{1}{N}\sum_{i=1}^{N} [y_i \log \hat{y}_i + (1-y_i) \log(1-\hat{y}_i)]$

Why this step? Maximizing likelihood finds parameters most consistent with observed data. Negative makes it a loss to minimize.

Intuition Check:

• If $y=1$ and $\hat{y}=0.9$: loss = $-\log(0.9) = 0.105$ (small)
• If $y=1$ and $\hat{y}=0.1$: loss = $-\log(0.1) = 2.303$ (large!)
• If $y=0$ and $\hat{y}=0.9$: loss = $-\log(0.1) = 2.303$ (large!)

$\frac{\partial J}{\partial w_j} = \frac{1}{N}\sum_{i=1}^{N} (\hat{y}^{(i)} - y^{(i)}) x_j^{(i)}$

Derivation:

For single example, chain rule: $\frac{\partial L}{\partial w_j} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z} \cdot \frac{\partial z}{\partial w_j}$

Step 1: Loss derivative w.r.t. prediction $\frac{\partial L}{\partial \hat{y}} = -\frac{y}{\hat{y}} + \frac{1-y}{1-\hat{y}}$

Why? Derivative of $-y\log\hat{y}$ is $-y/\hat{y}$, derivative of $-(1-y)\log(1-\hat{y})$ is $(1-y)/(1-\hat{y})$.

Step 2: Sigmoid derivative (we derived earlier) $\frac{\partial \hat{y}}{\partial z} = \hat{y}(1-\hat{y})$

Step 3: Linear part derivative $\frac{\partial z}{\partial w_j} = x_j$

Combine: $\frac{\partial L}{\partial w_j} = \left[-\frac{y}{\hat{y}} + \frac{1-y}{1-\hat{y}}\right] \cdot \hat{y}(1-\hat{y}) \cdot x_j$ $= \left[-y(1-\hat{y}) + (1-y)\hat{y}\right] x_j$ $= [-y + y\hat{y} + \hat{y} - y\hat{y}] x_j$ $= (\hat{y} - y) x_j$

Why this step? The cross-entropy and sigmoid derivatives perfectly cancel the complex fractions, leaving a beautifully simple gradient!

Average over $N$ examples: $\frac{\partial J}{\partial w_j} = \frac{1}{N}\sum_{i=1}^{N} (\hat{y}^{(i)} - y^{(i)}) x_j^{(i)}$

Gradient descent update: $w_j := w_j - \alpha \frac{\partial J}{\partial w_j}$

Worked Examples

Problem: Predict if a turbine blade will fail based on operating hours. Dataset:

• $x^{(1)} = 1000$ hours, $y^{(1)} = 0$ (no failure)
• $x^{(2)} = 5000$ hours, $y^{(2)} = 1$ (failure)

Initialize: $w = 0.0001$, $b = -0.3$ (features unnormalized for illustration; small $w$ chosen so $z$ stays in a sane range).

Forward Pass (Example 2): $z^{(2)} = w \cdot x^{(2)} + b = 0.0001 \cdot 5000 - 0.3 = 0.5 - 0.3 = 0.2$ $\hat{y}^{(2)} = \sigma(0.2) = \frac{1}{1 + e^{-0.2}} = \frac{1}{1.2214} = 0.5498$

Why this step? We compute the linear combination first, then pass through sigmoid to get probability. Note we deliberately use $w=0.0001$ throughout so that $z$ does not saturate the sigmoid.

Loss for this example: $L^{(2)} = -[1 \cdot \log(0.5498) + 0 \cdot \log(0.4502)] = -\log(0.5498) = 0.598$

Gradient: $\frac{\partial L^{(2)}}{\partial w} = (\hat{y}^{(2)} - y^{(2)}) \cdot x^{(2)} = (0.5498 - 1) \cdot 5000 = -2251$

Why negative? Prediction is too low (0.55 vs 1), so we need to increase $w$ to increase prediction. Negative gradient means descent will add to $w$.

Update ($\alpha = 10^{-6}$): Because the raw feature ($x=5000$) is large, the gradient is large, so we pick a tiny learning rate to avoid overshooting: $w := 0.0001 - 10^{-6} \cdot (-2251) = 0.0001 + 0.002251 = 0.002351$

Why this step? With unnormalized features the gradient magnitude scales with $x$, so $\alpha$ must be small. (In practice we normalize features — see Mistake 2 — which lets us use a larger, more stable $\alpha$.) After the update $w$ increases, making the model predict a higher probability for high operating hours.

Problem: Binary classifier for cloud presence. Features: average pixel brightness ($x_1$), brightness variance ($x_2$).

Training point: $\mathbf{x} = [180, 45]$, $y = 1$ (cloudy) Parameters: $\mathbf{w} = [0.01, -0.02]$, $b = -1.5$

Prediction: $z = 0.01(180) + (-0.02)(45) - 1.5 = 1.8 - 0.9 - 1.5 = -0.6$ $\hat{y} = \sigma(-0.6) = \frac{1}{1 + e^{0.6}} = \frac{1}{1 + 1.8221} = \frac{1}{2.8221} = 0.3543$

Why this step? Negative $z$ means model thinks "not cloudy" (prob < 0.5), but truth is cloudy ($y=1$) — error!

Loss: $L = -[1 \cdot \log(0.3543)] = 1.0376$

Gradients: $\frac{\partial L}{\partial w_1} = (\hat{y} - y) x_1 = (0.3543 - 1)(180) = -116.23$ $\frac{\partial L}{\partial w_2} = (0.3543 - 1)(45) = -29.06$

Why this step? Both gradients are negative because prediction is too low. Updates will increase both weights to make the model more "pro-cloudy" for these feature values.

Common Mistakes

Wrong Thinking: "Sigmoid gives 0.7, so output is 0.7?"

Why It Feels Right: Sigmoid outputs a number, so use it directly.

The Fix: $\hat{y}$ is a probability, not the final class. Apply threshold:

prediction_class = 1 if y_hat >= 0.5 else 0

For aerospace safety applications, you might use threshold 0.3 (prefer false alarms over missed failures).

Why This Works: Probabilities inform decisions based on cost/benefit. A 0.7 probability doesn't mean "70% of the answer is class 1" — it means "70% confidence in class 1."

Wrong Code:

# x1 in [0, 1000] hours, x2 in [0, 1] boolean
z = w1 * x1 + w2 * x2 + b

Why It Fails: $w_1$ will dominate because $x_1$ scale is huge. Gradient updates become unstable and force you to use a tiny learning rate (as we saw in Example 1).

The Fix: Standardize features: $x_j^{\text{norm}} = \frac{x_j - \mu_j}{\sigma_j}$

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

Why This Works: All features contribute equally. Gradients have similar magnitudes, making optimization converge faster and letting you use a larger, stable learning rate.

Wrong Math: Computing $L = -[y \log y]$ instead of $L = -[y \log \hat{y}]$

Why It Happens: Confusing true label $y$ with predicted probability $\hat{y}$.

The Fix: Cross-entropy measures divergence between:

• True distribution: $y$ (either 0 or 1)
• Predicted distribution: $\hat{y}$ (probability from sigmoid)

Always: $L = -[y \log \hat{y} + (1-y)\log(1-\hat{y})]$

Study Aids

Cross-Entropy: "Punish Confident Wrongs" $-\log$ grows huge as probability → 0. If model is confident (high $\hat{y}$) but wrong ($y=0$), loss explodes.

The sigmoid function is like a special ruler that turns any measurement (shine level) into a percentage between 0% and 100%. If shine is really high, sigmoid says "almost 100% heads." If shine is really low, sigmoid says "almost 0% heads" (so tails!). If shine is medium, sigmoid says "50% — could be either."

Now, how do we learn what "high" and "low" shine mean? We show the computer lots of coins with their true answers. Cross-entropy is the "wrongness score" — if you said "90% heads" but it was tails, you get a HUGE penalty. If you said "51% heads" and it was heads, tiny penalty. The computer adjusts its ruler to make the wrongness score as small as possible.

In aerospace, instead of coins, we might be guessing "will this engine part break?" based on temperature and vibration. Same math!

Connections

• Linear Regression: Logistic regression uses linear combination $z = \mathbf{w}^T\mathbf{x} + b$, but adds sigmoid for classification
• Gradient Descent: Weight updates use $\nabla J$ derived here; same optimization algorithm
• Softmax Regression: Multi-class generalization — sigmoid becomes softmax for $K > 2$ classes
• Neural Networks: Sigmoid is an activation function; cross-entropy is the standard classification loss
• Maximum Likelihood Estimation: Cross-entropy loss is negative log-likelihood of Bernoulli model
• ROC Curves and AUC: Evaluate logistic regression by varying the decision threshold
• Regularization (L1, L2): Prevent overfitting by adding $\lambda \|\mathbf{w}\|$ to loss function
• Bayesian Interpretation: Logistic regression approximates $P(y=1|\mathbf{x})$ directly

Flashcards

#flashcards/coding

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)