Bias terms and their role: Exercises — AI-ML deep dive

Before we start, let me re-anchor the four symbols we will lean on the whole way, because you may never use a symbol you have not seen defined:

Two activations appear below, so define them once:

Level 1 — Recognition

L1.1

State, in one equation, the pre-activation of a single neuron with $n$ inputs, and name every symbol.

Recall Solution

$z = \sum_{i = 1}^{n} w_{i} x_{i} + b = w^{⊤} x + b .$

$x_{i}$ : the $i$ -th input number.
$w_{i}$ : the weight on that input (multiplies it).
$b$ : the bias, a single scalar added to the sum (multiplies nothing).
$z$ : the pre-activation, fed next into an activation $ϕ$ .

L1.2

Which of these is the bias term acting correctly? (a) $z = w x \cdot b$ , (b) $z = w x + b$ , (c) $z = w (x + b)$ .

Recall Solution

(b). The bias is added on top of the weighted sum, not multiplied into it. (a) multiplies it (wrong), and (c) folds it inside the input so it gets scaled by $w$ — that is $w x + w b$ , a different number unless $w = 1$ .

L1.3

For a linear neuron $z = w x + b$ , what is the value of $z$ when $x = 0$ ? What does this tell you $b$ is, geometrically?

Recall Solution

At $x = 0$ : $z = w \cdot 0 + b = b$ . So $b$ is the $y$ -intercept of the line — the height where the line crosses the vertical axis. It is exactly the piece a bias-free model ( $z = w x$ , forced through the origin) can never produce.

Level 2 — Application

L2.1

A linear neuron is $z = w x + b$ with $w = 2, b = 5$ . Compute $z$ at $x = 0$ , $x = 3$ , and $x = - 4$ .

Recall Solution

$x = 0$ : $z = 2 (0) + 5 = 5$ .
$x = 3$ : $z = 2 (3) + 5 = 11$ .
$x = - 4$ : $z = 2 (- 4) + 5 = - 3$ . Notice every output is "slope $\times x$ " plus the same constant $5$ — that constant is the bias.

L2.2

A sigmoid neuron is $a = σ (w x + b)$ with $w = 1, b = - 4$ . At which input $x$ is the neuron half-activated ( $a = 0.5$ )?

Recall Solution

$σ$ equals $0.5$ exactly when its argument is $0$ . So set the pre-activation to zero: $w x + b = 0 \Rightarrow x = - \frac{b}{w} = - \frac{- 4}{1} = 4.$ The neuron is half-on at $x = 4$ . Because $b$ went negative, the switch point slid to the right — the neuron became "more reluctant," needing $x$ as large as $4$ before it half-fires.

L2.3

For the loss $L = \frac{1}{2} (a - y)^{2}$ with a linear neuron $a = z = w x + b$ , compute the bias gradient $\partial L / \partial b$ when $a = 5, y = 3$ . Then apply one update step $b \leftarrow b - η \partial L / \partial b$ with $η = 0.1$ and old $b = 5$ .

Recall Solution

Chain rule, linking loss → activation → pre-activation → bias: $\frac{\partial L}{\partial b} = \partial L / \partial a (a - y) \cdot \partial a / \partial z 1 \cdot \partial z / \partial b 1 = (a - y) .$ With $a = 5, y = 3$ : $\partial L / \partial b = 5 - 3 = 2$ . Update: $b \leftarrow 5 - 0.1 (2) = 4.8$ . The output was too high, so lowering $b$ pulls it back toward the target — exactly the right direction.

Level 3 — Analysis

L3.1

Prove that a bias-free linear model $z = w x$ can never fit the two points $(0, 3)$ and $(1, 5)$ with zero error, but $z = w x + b$ can. Give the fitting $w, b$ .

Recall Solution

Bias-free: at $x = 0$ , $z = w \cdot 0 = 0$ for every $w$ . But the target there is $3$ , so the error at that point is $∣0 - 3∣ = 3$ no matter what — never zero. With bias: we need two conditions:

$(0, 3)$ : $w (0) + b = 3 \Rightarrow b = 3$ .
$(1, 5)$ : $w (1) + b = 5 \Rightarrow w + 3 = 5 \Rightarrow w = 2$ . So $w = 2, b = 3$ gives $z = 2 x + 3$ , passing exactly through both points, $L = 0$ . The bias supplied the intercept the origin-locked model was missing.

L3.2

Two neurons feed identical inputs but have biases $b_{1} = + 6$ and $b_{2} = - 6$ (both $w = 1$ , sigmoid). Which one is "eager" and which is "reluctant"? Justify using the threshold $- b / w$ .

Recall Solution

Thresholds: neuron 1 fires half-on at $x = - b_{1} / w = - 6$ ; neuron 2 at $x = - b_{2} / w = + 6$ .

Neuron 1 is eager: even at $x = - 6$ (a small/negative push) it is already half-activated, so for ordinary inputs it is strongly on. Large positive bias = loose switch.
Neuron 2 is reluctant: it needs $x$ up to $+ 6$ before it half-fires. Large negative bias = stiff switch.

L3.3

Explain why the bias gradient is $\partial L / \partial b = δ$ while a weight's is $\partial L / \partial w_{i} = δ x_{i}$ . What consequence does this have when an input $x_{i}$ happens to be $0$ ?

Recall Solution

Both start from $δ = \partial L / \partial z$ (the neuron's error signal). The final factor differs:

\frac{\partial z}{\partial w_i}=x_i.$$ So bias picks up a factor $1$ (input-independent) while weight $w_i$ picks up $x_i$. **Consequence:** if $x_i=0$, then $\partial L/\partial w_i = \delta\cdot 0 = 0$ — that weight gets *no* update from this example. But the bias still updates by the full $\delta$, because its derivative is $1$, not $0$. The bias keeps learning even when inputs vanish.

Level 4 — Synthesis

L4.1

A minibatch of $m = 3$ examples produces per-example error signals $δ^{(1)} = 2, δ^{(2)} = - 1, δ^{(3)} = 5$ at one neuron. Using the averaged-batch rule, compute $\partial L / \partial b$ and the updated bias if old $b = 1, η = 0.3$ .

Recall Solution

Averaged bias gradient: $\frac{\partial L}{\partial b} = \frac{1}{m} \sum_{k = 1}^{m} δ^{(k)} = \frac{2 + ( - 1 ) + 5}{3} = \frac{6}{3} = 2.$ Update: $b \leftarrow 1 - 0.3 (2) = 1 - 0.6 = 0.4$ .

L4.2

You have a sigmoid neuron with $w = 2$ and want it to be half-activated exactly at $x = 1.5$ . What bias $b$ achieves this? Verify by computing the pre-activation at $x = 1.5$ .

Recall Solution

Half-activation requires threshold $- b / w = 1.5$ , so $b = - 1.5 w = - 1.5 (2) = - 3$ . Check: at $x = 1.5$ , $z = w x + b = 2 (1.5) + (- 3) = 3 - 3 = 0$ , and $σ (0) = 0.5$ . ✓ So $b = - 3$ places the "switch" precisely at $x = 1.5$ .

L4.3

A ReLU neuron uses $ϕ (z) = max (0, z)$ . Suppose it currently outputs $0$ for all training inputs (a "dead" neuron), because $z < 0$ everywhere and $max (0, z) = 0$ . The gradient of a dead ReLU is $0$ , so it can never revive. Explain how a positive initial bias would have prevented this, and give the smallest integer bias that makes $z \geq 0$ for the worst-case input where $w x = - 1.7$ .

Recall Solution

A dead ReLU has $z = w x + b < 0$ for every input, so its output is stuck at $0$ and its gradient is $0$ — no learning can move it. A small positive bias pushes the pre-activation up, keeping $z \geq 0$ for at least some inputs, so the ReLU stays in its "on" region where the gradient is $1$ and updates can flow. For the worst-case input, $z = - 1.7 + b \geq 0 \Rightarrow b \geq 1.7$ . Smallest integer: $b = 2.$ This is exactly why ReLU biases are often initialized to a small positive constant rather than $0$ .

Level 5 — Mastery

L5.1

Show that for a sigmoid neuron with $w = 0$ , the bias alone controls the output, and it is the same constant for every input. Compute the output when $b = 2$ . What has the neuron become?

Recall Solution

With $w = 0$ : $z = 0 \cdot x + b = b$ , independent of $x$ . So the output is $a = σ (b), the same value for all inputs.$ For $b = 2$ : $a = σ (2) = \frac{1}{1 + e ^{- 2}} \approx 0.8808$ . The neuron has become a constant output unit — it ignores its input entirely and just emits a fixed number set by the bias. This is the degenerate edge case: no slope, pure offset.

L5.2

Consider $L = \frac{1}{2} (a - y)^{2}$ with a sigmoid neuron $a = σ (z), z = w x + b$ . Derive $\partial L / \partial b$ in full, then evaluate it at $x = 1, w = 1, b = 0, y = 1$ . (Use $σ^{'} (z) = σ (z) (1 - σ (z))$ .)

Recall Solution

Chain rule with three links: $\frac{\partial L}{\partial b} = \partial L / \partial a (a - y) \cdot \partial a / \partial z σ (z) (1 - σ (z)) \cdot \partial z / \partial b 1 = (a - y) σ (z) (1 - σ (z)) .$ Evaluate: $z = 1 (1) + 0 = 1$ , so $a = σ (1) = \frac{1}{1 + e ^{- 1}} \approx 0.7311$ .

$(a - y) = 0.7311 - 1 = - 0.2689$ .
$σ (z) (1 - σ (z)) = 0.7311 (1 - 0.7311) = 0.7311 (0.2689) \approx 0.1966$ .
Product: $\partial L / \partial b \approx (- 0.2689) (0.1966) \approx - 0.0529$ . Negative gradient → gradient descent increases $b$ , pushing $z$ up and $a$ toward the target $1$ . ✓

L5.3

Prove the general threshold formula: for a single-input activation neuron $a = ϕ (w x + b)$ where $ϕ$ has its "half-on" value at argument $0$ (like sigmoid or tanh's zero-crossing), the switch input is $x^{⋆} = - b / w$ . Then discuss the degenerate case $w = 0$ : does a threshold exist?

Recall Solution

The switch happens where the argument of $ϕ$ is $0$ : $w x^{⋆} + b = 0 \Rightarrow x^{⋆} = - \frac{b}{w} (w \neq = 0) .$ Degenerate case $w = 0$ : the formula divides by zero, which signals no finite threshold exists. Indeed with $w = 0$ the output is the constant $σ (b)$ (from L5.1): the neuron never "switches" as $x$ varies — it is flat. So the threshold concept only makes sense when $w \neq = 0$ ; the sign of $w$ also decides whether increasing $x$ turns the neuron on ( $w > 0$ ) or off ( $w < 0$ ).

Connections

Weighted sum and linear combinations
Perceptron and the step function
Activation functions (sigmoid, ReLU, tanh)
Backpropagation and the chain rule
Weight initialization strategies
Regularization (L1, L2)
Affine transformations