Foundations — Bias terms and their role

This page assumes you have seen nothing. Every symbol the parent note throws at you — $x$, $w$, $\sum$, ${\mathbf{w}}^{\top }\mathbf{x}$, $\varphi$, $\sigma$, $\delta$, $\partial$ — is built here from the ground up, in the order that lets each one lean on the one before it.

0. A number line and a point on it

Before symbols, a picture. Everything a neuron does happens to numbers, and numbers live on a line.

• A number is just a position on a line. Right of the middle mark ($0$) is positive, left is negative.
• The special point $0$ is the origin. "Passing through the origin" will become the villain of this whole topic — keep it in view.

1. Input $x$ — the thing the neuron sees

Why the topic needs it: a neuron exists to react to inputs. No $x$, nothing to react to.

When there are several inputs we label them $x_1,x_2,\dots ,x_n$ — the little number below (the subscript) is just a name tag, "input number 1", "input number 2". It is not a power or a multiplication.

2. Weight $w$ — how much an input counts

Picture (look at the red line): plot $z=wx$ for inputs $x$ along the bottom. It is a straight line through the origin. Changing $w$ tilts the line like a see-saw pinned at the centre. Notice what it can never do: the line always passes through $0$. This is exactly the trap the bias will later fix.

Why the topic needs it: weights set the slope/orientation of the neuron's response. But slope alone is not enough — see §7.

3. The sum symbol $\sum$ — add up many terms

Real neurons have many inputs, so we must add many products $w_1{x}_1+w_2{x}_2+\dots$. Writing that out is tedious, so mathematicians invented a shorthand.

Why this tool and not just "…": with $n$ possibly being thousands, $\sum$ is the only compact, exact way to say "add all of them" without knowing $n$ in advance.

4. Vectors and ${\mathbf{w}}^{\top }\mathbf{x}$ — bundling the sum

The parent writes ${\mathbf{w}}^{\top }\mathbf{x}$. That funny ${}^{\top }$ ("transpose") plus side-by-side vectors is a compact name for the exact same sum from §3:

Why this tool: once you have thousands of inputs, ${\mathbf{w}}^{\top }\mathbf{x}$ is one clean symbol for the whole weighted sum. It is the backbone of an affine transformation once we add the bias.

5. The pre-activation $z$ — the number before the switch

Why the topic needs it: $z$ is the single number that decides how strongly the neuron will react. The bias's whole job is to nudge $z$.

6. Activation $\varphi$ and the sigmoid $\sigma$ — turning $z$ into an output

The parent uses one specific $\varphi$ a lot: the sigmoid, written $\sigma$ (lowercase Greek "sigma", different job from the capital $\sum$).

Why this switch: a hard step (Perceptron and the step function) is on/off with no in-between, which is impossible to nudge smoothly during learning. The sigmoid is the smooth cousin — and the point where it reads $0.5$ is the threshold the bias will slide. That's the whole reason the bias matters for a neuron.

7. Bias $b$ — the star of the show

Now every prerequisite is in place, so the parent's headline symbol finally earns its entrance.

• Slope vs offset: the weight $w$ tilts the line (§2); the bias $b$ slides it up/down. Two independent controls.
• Why we need it: in §2 the line was chained to the origin. Adding $b$ snaps that chain — now the neuron can output a nonzero value even when every input is $0$. That is the entire reason bias exists.
• All cases: $b>0$ lifts the line up / slides the sigmoid's switch left (fires more easily). $b<0$ pushes it down / slides the switch right (harder to fire). $b=0$ recovers the origin-locked line — the special starting case, not a general one.

Why "$-b/w$" is the threshold (used in the parent): the sigmoid flips at $z=0$, i.e. $wx+b=0$, i.e. $x=-b/w$. So $-b/w$ is the input value where the neuron is half-on.

8. Calculus symbols $\partial$ and $\delta$ — how the bias gets learned

The parent's learning section uses two more symbols. Built from zero:

Why the bias is special here: because $b$ is added (§7), nudging $b$ by a tiny amount moves $z$ by exactly the same amount, so $\partial z/\partial b=1$. That makes $\partial L/\partial b=\delta$ — the pure error, no input factor attached. (A weight instead gets $\partial L/\partial w_i=\delta x_i$, scaled by its input.)

Prerequisite map

Each box is built in a section above; every arrow means "you need the source to understand the target". Notice how origin feeds everything — it's the trap; and bias is what escapes it.

Equipment checklist

Test yourself — say the answer out loud before revealing.

Connections

• Parent: Bias terms and their role (3.1.13)
• 3.1.13 Bias terms and their role (Hinglish)
• Weighted sum and linear combinations
• Affine transformations
• Activation functions (sigmoid, ReLU, tanh)
• Perceptron and the step function
• Backpropagation and the chain rule
• Weight initialization strategies
• Regularization (L1, L2)