Bias terms and their role: Question bank — AI-ML deep dive

Reminders of the symbols you'll need (all built in the parent Bias terms and their role note):

$z = w^{⊤} x + b$ — the pre-activation (weighted sum plus bias, before the activation).
$ϕ$ — the activation function (see Activation functions (sigmoid, ReLU, tanh)).
$δ = \partial L / \partial z$ — the neuron's error signal (from Backpropagation and the chain rule).
$- b / w$ — the threshold input value where a sigmoid neuron is half-on.

True or false — justify

TF1. "A bias-free neuron can still fit the line

y = 2 x

True —

y = 2 x

passes through the origin, so

z = w x

with

w = 2

works; the bias is only needed when the line must miss the origin (a nonzero intercept).

TF2. "The bias multiplies one of the inputs."

False — the bias is added to the weighted sum, not multiplied by any input; it is the weight on a dummy input that is permanently

1

(see Weighted sum and linear combinations).

TF3. "One bias number is shared across all neurons in a layer."

False — every neuron has its own bias; a layer of

H

neurons carries a bias vector

b \in R^{H}

, one scalar per neuron.

TF4. "Because

\partial z / \partial b = 1

, the bias gradient equals the neuron's error

δ

and does not scale with the input."

True — weight gradients are

δ x_{i}

(input-scaled), but the bias gradient is the "pure" error

δ

, independent of how big the inputs are.

TF5. "Adding a bias makes a linear neuron nonlinear."

False —

z = w x + b

is still a straight line; it is an affine map (linear plus a shift). Nonlinearity comes only from the activation

ϕ

TF6. "Setting every bias to 0 causes the same symmetry problem that zero weights cause."

False — identical weights across neurons make them compute the same thing and get identical gradients (broken learning); biases don't route the inputs, so zero biases are a perfectly fine, standard starting point.

TF7. "For a ReLU neuron, a large negative bias can make it output zero for almost every input."

True — ReLU outputs

max (0, z)

; if

b

is very negative,

z = w x + b

stays below

0

for most

x

, so the unit is "dead" — which is exactly why ReLU biases are sometimes seeded with a small positive constant.

TF8. "Regularizing the bias with L2 improves generalization just like regularizing weights."

False — a bias is a single scalar per neuron adding negligible capacity; penalizing it just fights the model's ability to shift its mean output, adding error without curbing overfitting (see Regularization (L1, L2)).

TF9. "Making

b

more positive makes a sigmoid/step neuron easier to activate."

True — the neuron fires roughly when

z > 0

, i.e.

w^{⊤} x > - b

; a bigger

b

lowers the threshold

- b

, so the neuron is "eager" and fires for smaller inputs.

TF10. "If you use the absorbed-bias trick (append a constant

1

), you no longer need a special update rule for the bias."

True — once the bias is literally the weight on the constant-

1

input, ordinary gradient descent updates it like any other weight; the trick is why the same rule already applies.

Spot the error

SE1. "The bias lets the neuron rotate its decision line to any angle."

Wrong verb — weights rotate/tilt the line (they set slope and orientation); the bias shifts it parallel to itself. Rotation vs. translation is the whole weight-vs-bias distinction.

SE2. "

\partial L / \partial b = δ x

, same form as a weight gradient."

Wrong — it's

\partial L / \partial b = δ

with no

x

factor, because

\partial z / \partial b = 1

whereas

\partial z / \partial w = x

. The missing input factor is the key point.

SE3. "Because

- b

is the threshold, the neuron activates when the input

x

exceeds

- b

Wrong for

w \neq = 1

— the neuron activates when

z = w x + b > 0

, i.e.

x > - b / w

. The threshold on the input is

- b / w

, not

- b

;

- b

is the threshold on the weighted sum.

SE4. "A minibatch of

m

examples gives

m

separate biases to average."

Wrong — there is still one bias per neuron; what you average are the

m

per-example gradients:

\partial L / \partial b = \frac{1}{m} \sum_{k} δ^{(k)}

. One parameter, many gradient contributions.

SE5. "Since bias is 'the weight on a 1', we should initialize it randomly to break symmetry."

Wrong — symmetry-breaking is a weight concern; biases are typically initialized to

0

(or a small positive constant for ReLU), and this causes no symmetry problem.

SE6. "A perceptron with a step activation and no bias can still put its decision boundary anywhere."

Wrong — a bias-free boundary

w^{⊤} x = 0

always passes through the origin; it can only rotate about that point. See Perceptron and the step function.

Why questions

WHY1. Why does a bias-free neuron force its decision boundary through the origin?

Setting

z = 0

gives

w^{⊤} x = 0

, and

x = 0

always satisfies it — the origin is always on the boundary, so no offset is possible without

b

WHY2. Why is the bias gradient input-independent while the weight gradient is not?

The bias adds a constant, so

\partial z / \partial b = 1

; a weight multiplies its input, so

\partial z / \partial w_{i} = x_{i}

. The chain rule then carries that

x_{i}

into the weight gradient but leaves the bias gradient as the bare error

δ

WHY3. Why do we sometimes initialize ReLU biases to a small positive value instead of

0

A slightly positive bias keeps the pre-activation

z

above

0

early on, so more units stay active and pass gradient — this avoids "dead" ReLUs that never learn.

WHY4. Why is the bias trained by the same gradient descent rule as the weights?

Via the absorbed-bias trick it literally is a weight (on the constant-

1

input), so it inherits the identical update

b \leftarrow b - η δ

WHY5. Why do we usually exclude biases from L2 regularization?

L2 shrinks parameters toward

0

to reduce capacity, but shrinking a bias toward

0

merely re-pins outputs near the origin — it removes the neuron's freedom to set its baseline without meaningfully reducing overfitting.

WHY6. Why does shifting the bias slide an entire sigmoid S-curve left or right rather than reshaping it?

Because

σ (w x + b) = σ (w (x + b / w))

is the same curve evaluated at a shifted input; changing

b

only relocates the half-activation point

- b / w

, leaving the shape (set by

w

) untouched.

WHY7. Why is the bias described as controlling the "threshold" while weights control "slope"?

Weights determine how steeply and in which direction

z

responds to inputs (slope/orientation), whereas the bias sets the crossover point

z = 0

where activation flips — the threshold.

Edge cases

EC1. What happens to the threshold

- b / w

if the weight

w \to 0

The threshold

- b / w

blows up (to

\pm \infty

): with

w = 0

the neuron ignores the input entirely and outputs the constant

ϕ (b)

regardless of

x

— no meaningful threshold exists.

EC2. If all inputs are exactly zero, what is the neuron's output?

z = w^{⊤} 0 + b = b

, so the output is

ϕ (b)

— the bias alone determines the "resting" activation when nothing drives the neuron.

EC3. Can a purely bias-driven neuron (all weights

0

) learn anything useful?

Only a constant — it outputs

ϕ (b)

for every input, so it can fit a fixed baseline but cannot respond to the data; useful only as an offset, never as a discriminator.

EC4. Over a minibatch, when is the accumulated bias gradient

\frac{1}{m} \sum_{k} δ^{(k)}

exactly zero even though individual errors are nonzero?

When the per-example errors

δ^{(k)}

cancel (positive and negative errors balance), the bias stops moving — the neuron's baseline is already centered even if individual predictions are off.

EC5. With a symmetric activation like

tanh

, does the bias still act as a shift?

Yes —

tanh (w x + b)

is the same S-curve shifted so its zero-crossing sits at

x = - b / w

; symmetry of the activation doesn't remove the bias's translating role.

EC6. If two neurons in a layer end up with identical weights but different biases, are they redundant?

No — equal weights give the same orientation, but different biases place their thresholds at different inputs, so they can still respond to distinct regions and are not duplicates.

Connections

Bias terms and their role
Perceptron and the step function
Weighted sum and linear combinations
Activation functions (sigmoid, ReLU, tanh)
Backpropagation and the chain rule
Weight initialization strategies
Regularization (L1, L2)
Affine transformations