Bias terms and their role: Worked examples — AI-ML deep dive

This page is the "exhaustive drill" companion to the bias topic (the parent). Read that first if you have not — here we assume you know the one line $z = w^{⊤} x + b,$ where $z$ is the pre-activation (the number the neuron computes before squashing it), $w$ is the list of weights (one multiplier per input), $x$ is the list of inputs, and $b$ is the bias (a single added number). Everything below is a chase through every kind of case this formula can produce.

The scenario matrix

Every case this topic can throw at you falls into one of these cells. Each worked example is tagged with the cell(s) it covers.

Cell	What varies	Example(s)
A Positive bias	$b > 0$ — line/threshold shifts one way	Ex 1
B Negative bias	$b < 0$ — line/threshold shifts the other way	Ex 2
C Zero bias (degenerate)	$b = 0$ — forced through origin	Ex 3
D Zero input	all $x_{i} = 0$ , only bias survives	Ex 3, Ex 4
E Multi-input neuron	$w, x$ are vectors, threshold is a plane	Ex 4
F Sign of the gradient	$\partial L / \partial b$ positive vs negative → $b$ goes down vs up	Ex 5
G Limiting behaviour	$b \to + \infty$ / $b \to - \infty$ for sigmoid	Ex 6
H Real-world word problem	translate a story into $w, b$	Ex 7
I Exam twist	" $- b / w$ ", threshold algebra, negative $w$	Ex 8
J Minibatch gradient	average of per-example $δ$ 's	Ex 9

Two symbols we will lean on:

$σ (z) = \frac{1}{1 + e ^{- z}}$ — the sigmoid, an S-shaped squasher that maps any real $z$ into $(0, 1)$ . See Activation functions (sigmoid, ReLU, tanh). It equals exactly $0.5$ when $z = 0$ , because then $e^{- 0} = 1$ and $\frac{1}{1 + 1} = \frac{1}{2}$ .
$δ = \frac{\partial L}{\partial z}$ — the neuron's error signal, the sensitivity of the loss $L$ to the pre-activation. From the parent, $\frac{\partial L}{\partial b} = δ$ .

Example 1 — Positive bias shifts the line up (Cell A)

Example 2 — Negative bias shifts the line down (Cell B)

Example 3 — Zero bias & zero input: the degenerate cell (Cells C, D)

Worked example Why a bias-free neuron can never fit a constant

Data: $(x, y) = (0, 4), (1, 4)$ — a flat target $y = 4$ . Model without bias: $z = w x$ . Can any $w$ fit it?

Forecast: Try to find a winning $w$ in your head before continuing.

At $x = 0$ : $z = w \cdot 0 = 0$ for every $w$ . But we need $z = 4$ . Error $= ∣0 - 4∣ = 4$ , no matter what. Why? With $b = 0$ , the line is pinned to the origin. Zero input forces zero output.
Best possible fit. Minimise MSE $L = \frac{1}{2} [(w \cdot 0 - 4)^{2} + (w \cdot 1 - 4)^{2}]$ . Only the second term depends on $w$ ; set it to zero with $w = 4$ . Then $L = \frac{1}{2} (16) + 0 = 8$ . Why? Even the optimal bias-free model leaves loss $8 > 0$ .
Add a bias. $z = w x + b$ with $w = 0, b = 4$ gives $z = 4$ everywhere. $L = 0$ . Why? The bias supplies the constant the origin-pinned line could not.

Verify: Bias-free best loss $= 8$ (checked in VERIFY); bias-full loss $= 0$ . This is Cell C (zero bias fails) meeting Cell D (zero input) meeting the parent's core claim.

Example 4 — Multi-input neuron: bias shifts a plane (Cell E, D)

Worked example Two inputs, one threshold

Neuron $z = w_{1} x_{1} + w_{2} x_{2} + b$ with $w = (2, - 1)$ , $b = - 3$ . (a) Compute $z$ at $x = (0, 0)$ . (b) Find where the neuron switches on ( $z = 0$ ).

Forecast: At the all-zero input, what survives?

(a) Zero input: $z = 2 (0) + (- 1) (0) + (- 3) = - 3$ . Why? Both weighted terms vanish; only $b$ remains — the bias is the output at the origin (Cell D generalised to a vector).
(b) Decision boundary: set $z = 0$ : $2 x_{1} - x_{2} - 3 = 0 \Rightarrow x_{2} = 2 x_{1} - 3$ . Why? The neuron flips from "off" ( $z < 0$ ) to "on" ( $z > 0$ ) across the line $z = 0$ . Here it is a straight line in the $(x_{1}, x_{2})$ plane; the bias sets its offset from the origin.
Check the offset. With $b = 0$ the boundary would pass through $(0, 0)$ ; with $b = - 3$ its intercept moves to $x_{2} = - 3$ at $x_{1} = 0$ .

Verify: Point $(1.5, 0)$ : $z = 2 (1.5) - 0 - 3 = 0$ ✓ — sits exactly on the boundary. The bias budged the whole separating line, just like it budged the 1-D line in Ex 1–2.

Example 5 — Sign of the bias gradient (Cell F)

Worked example Which way does

$b$ move? Identity neuron $a = z = w x + b$ , squared-error loss $L = \frac{1}{2} (a - y)^{2}$ . Current $w = 1, b = 1$ , sample $(x, y) = (2, 3)$ , learning rate $η = 0.1$ . Do one bias update.

Forecast: Is the output too high or too low? Will $b$ increase or decrease?

Forward pass: $a = 1 \cdot 2 + 1 = 3$ . Wait — that's exactly $y = 3$ ! Let me instead use $b = 2$ : $a = 1 \cdot 2 + 2 = 4$ . Why? We need a non-zero error to see the sign; with $b = 2$ the output overshoots.
Error signal: $δ = \frac{\partial L}{\partial z} = (a - y) \cdot 1 = (4 - 3) = 1$ . Why? $\partial L / \partial a = (a - y)$ and $\partial a / \partial z = 1$ (identity). Positive $δ$ means output is too big.
Bias gradient: $\frac{\partial L}{\partial b} = δ = 1$ (because $\partial z / \partial b = 1$ ; see Backpropagation and the chain rule). Why? Bias gradient equals the pure error — no input factor.
Update: $b \leftarrow b - η δ = 2 - 0.1 (1) = 1.9$ . Why? Gradient descent subtracts the gradient; a positive gradient lowers $b$ , shrinking the too-large output.

Verify: New output $a = 1 \cdot 2 + 1.9 = 3.9$ , closer to $3$ than $4$ was ✓. Positive error → $b$ went down. Flip the sign of the error (output too small) and $b$ would go up.

Example 6 — Limiting behaviour of a sigmoid bias (Cell G)

Worked example Push the bias to the extremes

Neuron $a = σ (x + b)$ at fixed $x = 0$ . What happens to $a$ as $b \to + \infty$ ? As $b \to - \infty$ ? As $b = 0$ ?

Forecast: A "very eager" neuron and a "very reluctant" one — what output do they saturate to?

$b = 0$ : $a = σ (0) = \frac{1}{1 + 1} = 0.5$ . Why? Neutral bias → the balanced midpoint of the S-curve.
$b \to + \infty$ : $z = b \to + \infty$ , $e^{- z} \to 0$ , so $a \to \frac{1}{1 + 0} = 1$ . Why? A huge positive bias makes the neuron fire almost always — "eager". At $b = 10$ : $a = σ (10) \approx 0.99995$ .
$b \to - \infty$ : $e^{- z} = e^{∣ b ∣} \to + \infty$ , so $a \to \frac{1}{\infty} = 0$ . Why? A huge negative bias makes it almost never fire — "reluctant". At $b = - 10$ : $a \approx 0.0000454$ .

Verify: $σ (10) \approx 0.9999546$ and $σ (- 10) \approx 0.0000454$ sum to $1$ ✓ (sigmoid symmetry $σ (- z) = 1 - σ (z)$ ). The bias slides the whole S-curve; at the limits the output saturates at 1 or 0.

Example 7 — Real-world word problem (Cell H)

Worked example The "spam alarm" neuron

A spam detector fires ( $z > 0$ ) based on one feature $x =$ number of suspicious words. Domain rule: fire when $x \geq 5$ . Weight is fixed at $w = 1$ . What bias makes the threshold exactly $x = 5$ ?

Forecast: Recall the threshold is where $z = 0$ . Solve for $b$ .

Threshold condition: neuron fires when $z = w x + b > 0$ , i.e. $x > - b / w$ . Why? The boundary is $z = 0$ ; rearranging gives the input value that flips the neuron.
Set threshold to 5: $- b / w = 5$ with $w = 1 \Rightarrow - b = 5 \Rightarrow b = - 5$ . Why? We want the switch point at exactly $x = 5$ ; a negative bias raises the bar.
Interpret. $b = - 5$ is a "reluctant" neuron: it needs 5 suspicious words before its weighted sum overcomes the offset.

Verify: At $x = 5$ : $z = 1 (5) + (- 5) = 0$ (exactly on threshold) ✓. At $x = 6$ : $z = 1 > 0$ fires ✓. At $x = 4$ : $z = - 1 < 0$ stays quiet ✓. The bias encoded the business rule "need 5 words".

Example 8 — Exam twist: negative weight & the $- b / w$ formula (Cell I)

Example 9 — Minibatch bias gradient (Cell J)

Recall Checks

Recall In Example 3, what is the best possible loss of the bias-free model?

$L = 8$ — the $x = 0$ point contributes $\frac{1}{2} (0 - 4)^{2} = 8$ no matter what $w$ is, since $z = w \cdot 0 = 0$ .

Recall In Example 8, why does the neuron fire for

small $x$ ? Because $w = - 2$ is negative; dividing the inequality $z > 0$ by $w$ flips it to $x < 3$ . Sign of $w$ sets the firing direction; $- b / w$ sets the location.

Recall Why is the batch bias gradient just the mean of the

$δ$ 's (not input-weighted)? Because $\partial z / \partial b = 1$ for every example, so each contributes $δ^{(k)} \cdot 1$ ; averaging gives $\frac{1}{m} \sum δ^{(k)}$ .

Connections

3.1.13 Bias terms and their role (Hinglish)
Perceptron and the step function
Weighted sum and linear combinations
Activation functions (sigmoid, ReLU, tanh)
Backpropagation and the chain rule
Affine transformations