Visual walkthrough — The perceptron model and history

This is the visual walkthrough of the perceptron model. We build the whole thing from nothing — no dot products, no hyperplanes, no learning rule assumed. By the end you will see why one straight line and one nudge-when-wrong rule is enough to learn.

A smart 12-year-old rule applies: if a symbol appears, it was drawn first.

---

Step 1 — A dot on a grid is a "thing to sort"

WHAT. We start with the plainest possible object: a point on a flat grid. The grid has two axes. The horizontal axis measures one number about our object; the vertical axis measures a second number. Call the horizontal number $x_1$ and the vertical number $x_2$.

So one object = two numbers = one dot. We write the pair as a list $\mathbf{x}=[x_1,x_2]$ and call it the input vector — "vector" just means "an ordered list of numbers", here a location.

WHY. Almost anything we want a machine to judge can be turned into numbers. An email becomes "how many $ signs ($x_1$), how many ! marks ($x_2$)". Sorting emails then means sorting dots. Pictures are easier to reason about than emails, so we look at dots.

PICTURE. Below: two clouds of dots. Magenta dots are class 1 (say "spam"), violet dots are class 0 ("not spam"). The whole job of the perceptron is one sentence: draw a line that keeps the colours apart.

---

Step 2 — A weighted sum turns a dot into one number

WHAT. We invent a single number that summarises the dot, called $z$:

$z=\underset{\text{importance of }{x}_1}{\underbrace{w_1}}\underset{\text{feature 1}}{\underbrace{x_1}}+\underset{\text{importance of }{x}_2}{\underbrace{w_2}}\underset{\text{feature 2}}{\underbrace{x_2}}+\underset{\text{base offset}}{\underbrace{b}}$

Each weight $w_i$ is a dial: it says how strongly feature $i$ pushes the answer up. The bias $b$ is a starting offset — a "head start or handicap" added no matter what the dot is.

WHY a weighted sum and not something fancier? Because it is the simplest way to blend several numbers into one while still letting each number matter a different amount. Multiplication-then-add is the least-assuming combination there is: if you doubled a feature's importance you double its dial $w_i$. We reach for anything more complicated only if this fails.

PICTURE. Think of $z$ as a height floating above the flat grid. Every $(x_1,x_2)$ dot gets lifted to a height $z$. Because $z$ is linear, these heights form a tilted flat sheet — a ramp over the grid. Weights set the tilt direction; bias sets how high the ramp sits.

---

Step 3 — Slice the ramp at zero to get a decision line

WHAT. A score is nice, but we need a yes/no answer $\hat{y}$ (the prediction, with a hat to mean "guess"). We use the simplest cut:

$\hat{y}=\{\begin{array}{ll}1& \text{if }z\ge 0\\ 0& \text{if }z<0\end{array}$

The place where $z=0$ is where the tilted ramp crosses the flat grid. That crossing is a straight line.

WHY cut at zero? We chose $0$ as the fence, and it costs nothing: the bias $b$ can slide the whole ramp up or down, so "$z\ge 0$" can be made to mean any threshold we like. Fixing the fence at $0$ and letting $b$ do the moving keeps the maths clean. (Tie rule: $z=0$ counts as class 1 — a convention we keep everywhere.)

PICTURE. The ramp from Step 2 is sliced by the flat grid. The intersection is the decision boundary. Everything the ramp lifts above zero → magenta side (predict 1); everything it dips below zero → violet side (predict 0).

This is the moment Linear Separability enters: the perceptron can only ever draw one straight line. If no straight line separates the colours, this machine is stuck (that is the famous XOR wall — see Multi-Layer Perceptron).

---

Step 4 — Why the weight arrow points across the line

WHAT. Collect the weights into their own arrow $\mathbf{w}=[w_1,w_2]$. The middle chunk of $z$ is the dot product:

$\mathbf{w}\cdot \mathbf{x}=w_1{x}_1+w_2{x}_2$

The dot product measures how much the dot $\mathbf{x}$ leans in the direction of the arrow $\mathbf{w}$. Lean the same way → positive; lean opposite → negative; sideways (at a right angle) → exactly zero.

WHY the dot product and not, say, distance? Because we need a quantity that is positive on one side of the line and negative on the other, flipping sign exactly as we cross. Distance is always positive — useless for telling sides apart. The dot product is the natural "which side, and how far along the arrow" meter, and it is precisely the $z$ (minus $b$) we already built.

PICTURE. The arrow $\mathbf{w}$ sits perpendicular to the decision line and points into the magenta (class 1) territory. Dots ahead of the arrow score positive; dots behind it score negative. The line itself is exactly the set of "sideways" dots where the dot product is $-b$.

---

Step 5 — Learning = nudge the arrow when you're wrong

WHAT. Start with any arrow, walk through examples, and only act on mistakes. For a training pair $(\mathbf{x},y)$:

$\mathbf{w}\leftarrow \mathbf{w}+\underset{\text{step size}}{\underbrace{\eta }}\underset{\text{error }\in \{-1,0,+1\}}{\underbrace{(y-\hat{y})}}\underset{\text{the dot itself}}{\underbrace{\mathbf{x}}},b\leftarrow b+\eta (y-\hat{y})$

Here $\eta$ (Greek "eta") is the learning rate — how big a nudge each mistake earns.

WHY exactly $+(y-\hat{y})\mathbf{x}$? Look at the error factor:

• Missed a class-1 dot ($y=1,\hat{y}=0$): error $=+1$, so $\mathbf{w}\leftarrow \mathbf{w}+\eta \mathbf{x}$ — the arrow swings toward that dot, raising its future $z$.
• Wrongly flagged a class-0 dot ($y=0,\hat{y}=1$): error $=-1$, so $\mathbf{w}\leftarrow \mathbf{w}-\eta \mathbf{x}$ — the arrow swings away, lowering its $z$.
• Correct ($y=\hat{y}$): error $=0$ — do nothing.

We add/subtract the dot itself because that is the exact direction that moves this dot's score the most, per Step 4. This is the humble ancestor of Gradient Descent and later Backpropagation.

PICTURE. A misclassified magenta dot is sitting behind the arrow. Adding $\eta \mathbf{x}$ (dashed) rotates $\mathbf{w}$ toward it; the decision line pivots so the dot lands on the correct side.

---

Step 6 — The degenerate cases (never let the reader fall through)

WHAT & WHY. Three corners of the world the rule must survive:

1. The tie, $z=0$. A dot lands exactly on the line. Our convention resolves this to class 1. In the parent's spam example the very first pass has $\mathbf{w}=0$, so $z=0$ for everything — every dot is predicted 1 until a class-0 mistake nudges the weights negative. Nothing breaks; the tie rule is what makes that first step well-defined.
2. All-zero input $\mathbf{x}=[0,0]$. Then $z=b$ alone. The bias must exist, or the origin dot could never be sorted. Updating on it changes $b$ but not $\mathbf{w}$ (since $\eta \cdot \text{error}\cdot 0=0$): the line shifts without tilting.
3. Not separable (e.g. XOR). No straight line works, so mistakes never stop and the arrow wobbles forever. The perceptron does not converge — this is a feature of the data, not a bug. The fix is more lines: Multi-Layer Perceptron. Smoother cousins (Logistic Regression, Support Vector Machines) also live here.

PICTURE. Left: a dot on the line (the tie) coloured as class 1. Middle: the origin dot moved only by $b$ (line slides, no tilt). Right: XOR's four dots — magenta and violet on opposite corners — with a red "no single line possible" mark.

---

The one-picture summary

Everything at once: dots → weighted-sum ramp → zero-slice line → perpendicular arrow → nudge-on-mistake. Read it left to right and the whole derivation replays.

Recall Feynman retelling (say it out loud)

Every thing I want to judge, I turn into a dot on a grid. I make up a score by multiplying each of the dot's numbers by an importance dial and adding a base offset — that's $z$. Positive score means "yes", negative means "no", and the fence is $z=0$, which draws a straight line on the grid. The importance dials form an arrow that points across the line toward the "yes" side; how far a dot leans along that arrow is its score. To learn, I only fix things when I'm wrong: if I missed a "yes" dot I swing the arrow toward it; if I wrongly flagged a "no" dot I swing away; if I was right I don't touch anything. If a straight line can split the colours, I'm guaranteed to find one; if it can't (like XOR), I keep wobbling forever and need more lines. That's the whole perceptron.

Recall

What single geometric object does $\mathbf{w}\cdot \mathbf{x}+b=0$ draw in 2D? ::: A straight line — the decision boundary. Which direction does the weight arrow $\mathbf{w}$ point? ::: Perpendicular to the line, into the class-1 region. When does the update rule change the weights? ::: Only on a mistake ($y\ne \hat{y}$); the error factor is then $\pm 1$. Why can't a single perceptron solve XOR? ::: XOR is not linearly separable — no single straight line separates the classes. What happens on an all-zero input? ::: $z=b$; an update shifts the bias only, sliding the line without tilting it.

Related: McCulloch-Pitts Neuron · Activation Functions · Linear Separability · Gradient Descent · back to the parent