Logistic regression and the sigmoid function

WHAT is logistic regression?

WHY the sigmoid, specifically? (Derivation from odds)

We want a function that turns a linear score into a probability. Start from odds, not probability.

Formula Derivation: log-odds → sigmoid

Step 1. Let $p = P(y=1\mid x)$ . The odds are $\dfrac{p}{1-p}\in(0,\infty)$ . Why this step? Probability lives in $[0,1]$ (bounded), but a linear model outputs anything. Odds live in $(0,\infty)$ — closer, but still one-sided.

Step 2. Take the log-odds (called the logit): $\ln\dfrac{p}{1-p}\in(-\infty,\infty)$ . Why this step? Now the range matches a linear function's range exactly. So we can boldly set $\ln\frac{p}{1-p} = w^\top x + b = z.$

Step 3. Solve for $p$ . Exponentiate: $\dfrac{p}{1-p} = e^{z}$ . Why? Undo the log to isolate $p$ .

Step 4. $p = e^{z}(1-p) \Rightarrow p + p e^{z} = e^{z} \Rightarrow p(1+e^z)=e^z$ $p = \frac{e^z}{1+e^z} = \frac{1}{1+e^{-z}} = \sigma(z).$ Why? Divide top and bottom by $e^z$ to get the clean canonical form.

So the sigmoid is not arbitrary — it is the inverse of the log-odds. Assuming log-odds are linear in $x$ forces the sigmoid.

HOW do we train it? (The loss and its gradient)

We can't use squared error nicely (it's non-convex for sigmoid). Instead use maximum likelihood.

Formula Derivation of the loss (cross-entropy)

Step 1. For one example, the label is Bernoulli: $P(y\mid x) = p^{y}(1-p)^{1-y}, \quad p=\sigma(z).$ Why? If $y=1$ it gives $p$ ; if $y=0$ it gives $1-p$ . One formula, both cases.

Step 2. Likelihood of all $m$ examples = product. Products are hard, so take the log: $\ell = \sum_i \big[y_i\ln p_i + (1-y_i)\ln(1-p_i)\big].$ Why log? Turns product into sum, and log is monotonic so the maximizer is unchanged.

Step 3. We minimize the negative (a cost to descend): $J = -\frac{1}{m}\sum_i \big[y_i\ln p_i + (1-y_i)\ln(1-p_i)\big].$ This is the binary cross-entropy (log loss). It is convex in $w$ , so gradient descent finds the global minimum.

The decision boundary

Worked examples

Worked example Example 2 — One gradient step

Two features, $w=(0,0)$ , $b=0$ , $\alpha=0.5$ . One data point $x=(1,2)$ , $y=1$ .

Step 1: $z=0 \Rightarrow p=\sigma(0)=0.5$ . Why? Starting weights give a neutral guess.
Step 2: error $= p-y = 0.5-1 = -0.5$ .
Step 3: $\partial J/\partial w_1 = (-0.5)(1)=-0.5$ ; $\partial J/\partial w_2=(-0.5)(2)=-1$ .
Step 4: $w_1 \leftarrow 0 - 0.5(-0.5)=0.25$ ; $w_2\leftarrow 0-0.5(-1)=0.5$ .
Why did weights go up? We under-predicted a positive example, so the model pushes $z$ higher next time.

Common mistakes

Flashcards

What does the sigmoid function map its input to?

The open interval

(0,1)

, interpreted as a probability.

Write the sigmoid formula.

\sigma(z)=\dfrac{1}{1+e^{-z}}

What is

\sigma(0)

0.5

What is the derivative

\sigma'(z)

\sigma(z)(1-\sigma(z))

What quantity does logistic regression assume is linear in

x

The log-odds (logit),

\ln\frac{p}{1-p}=w^\top x+b

Why not use squared error for logistic regression?

It becomes non-convex and its gradient saturates when very wrong; cross-entropy is convex.

State the binary cross-entropy loss for one example.

-[y\ln p + (1-y)\ln(1-p)]

with

p=\sigma(z)

What is the gradient

\partial J/\partial w_j

\frac{1}{m}\sum_i (p_i-y_i)x_{ij}

— (pred−target)×input.

Where is the decision boundary?

Where

w^\top x + b = 0

(equivalently

p=0.5

); it is linear.

If log-odds

=1.1

, what are the odds and

p

Odds

=e^{1.1}\approx3

, so

p=0.75

Is logistic regression classification or regression?

Classification (name is historical).

Recall Feynman: explain to a 12-year-old

Imagine a machine that gives a "yes-ness score" — big positive means "definitely yes", big negative means "definitely no", zero means "no idea". But a score like 7 or −3 is confusing. So we pass it through a special squishy slide (the sigmoid) that turns any score into a number between 0 and 1 — like a percentage of confidence. Score of 0 → 50%. Big score → near 100%. We then teach the machine by nudging its dials whenever it's too confident and wrong, using the simple rule "how wrong you were × the input".

Connections

Linear Regression — supplies the linear score $z=w^\top x+b$ that sigmoid squashes.
Cross-Entropy Loss — the training objective derived from maximum likelihood.
Gradient Descent — optimizer that uses the $(p-y)x$ gradient.
Maximum Likelihood Estimation — the principle that justifies the loss.
Softmax Regression — multi-class generalization of the sigmoid.
Decision Boundaries — why the boundary here is a hyperplane.
Neural Networks — sigmoid as an activation function / a single logistic neuron.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, linear regression toh ek number predict karta hai jo kuch bhi ho sakta hai — minus infinity se plus infinity tak. Par jab sawaal "haan ya na" ka ho (spam hai ya nahi, tumor malignant hai ya nahi), tab humein ek probability chahiye jo 0 aur 1 ke beech ho. Iske liye hum pehle ek linear score nikalte hain $z = w^\top x + b$ , aur phir usko sigmoid function se dabaa dete hain: $\sigma(z)=1/(1+e^{-z})$ . Ye har score ko 0 aur 1 ke beech le aata hai, jise hum $P(y=1)$ maante hain.

Sigmoid koi random function nahi hai. Agar tum log-odds ( $\ln\frac{p}{1-p}$ ) ko linear maan lo, toh algebra se solve karne par automatically sigmoid nikal aata hai. Isliye ye natural choice hai. Boundary wahan hoti hai jahan $z=0$ , matlab $p=0.5$ — aur ye boundary hamesha ek seedhi line/hyperplane hoti hai, isiliye ise "linear" model kehte hain.

Training ke liye squared error mat use karo — wo yahan non-convex ho jaata hai aur galat jagah gradient zero ho jaata hai. Iski jagah cross-entropy (log loss) use karte hain, jo maximum likelihood se aata hai aur convex hota hai. Sabse pyari baat: gradient ban jaata hai bilkul simple — $(p - y)\cdot x$ , yaani (prediction minus asli answer) guna input. Bilkul linear regression jaisa, bas prediction ki jagah sigmaid output.

Yaad rakhne ka trick: "Log-odds go linear, then squash." Aur sigmoid ka derivative? "s times one-minus-s" — $\sigma(1-\sigma)$ . Bas itna pakka kar lo toh logistic regression tumhaara dost ban jaayega.

Logistic regression and the sigmoid function

WHAT is logistic regression?

WHY the sigmoid, specifically? (Derivation from odds)

HOW do we train it? (The loss and its gradient)

The decision boundary

Worked examples

Common mistakes

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Linear & Logistic Regression

Connections