Foundations — The perceptron model and history

Before you touch the parent note, you must be fluent in a handful of picture-ideas. Below, every symbol you will meet is built from nothing. Read top to bottom: each item uses only things defined above it.

1. A feature — one measured number

Picture one email. We measure two things about it: how many $ signs it has, and how many ! marks it has. Each is one number. That number is a feature.

• Plain words: one fact about one example, written as a number.
• The picture: a slider with a value on it — say the $-count slider sitting at 2.
• Why the topic needs it: the perceptron can only do arithmetic. So the world must first be turned into numbers. Features are that translation.

We write a feature with a lowercase letter and a small number below (a subscript): $x_1$ is "feature number 1", $x_2$ is "feature number 2". The subscript is just a name tag, not a multiplication.

2. A vector — several features held together

If one email has two features, we do not want two loose numbers floating around — we bundle them.

• Plain words: a list where position matters ($[2,5]$ is not $[5,2]$).
• The picture: an arrow starting at the corner $(0,0)$ and ending at the point $(x_1,x_2)$. See the figure below.
• Why the topic needs it: it lets us talk about one whole example as a single object $\mathbf{x}$, and lets us picture that example as a point (the arrow's tip) on a flat map.

What this figure shows: the single example $\mathbf{x}=[2,5]$ drawn two ways at once — a lavender arrow from the origin (carrying a direction and a length) and a coral dot at its tip (a location on the map). The dotted grey lines drop from the tip to the axes to show the feature values $x_1=2$ and $x_2=5$. The perceptron mostly cares about the dot — where does this example sit?

3. The weights — how much each feature counts

• Plain words: an importance dial for each feature. Big positive $w_i$ = "this feature strongly votes YES". Negative = "votes NO".
• The picture: the same arrow map, but now a second arrow $\mathbf{w}=[w_1,w_2]$ pointing in the "YES direction".
• Why the topic needs it: raw features don't know which of them matters. The weights are exactly the knobs the perceptron learns by turning.

4. Multiplication and the weighted sum

• Plain words: "count each feature's vote, scaled by how much that feature matters, then total the votes."
• The picture: two bars — one of height $w_1{x}_1$, one of height $w_2{x}_2$ — stacked into one taller bar.
• Why the topic needs it: this single total number is the perceptron's "level of enthusiasm" for saying class 1. One number is easy to compare against a threshold.

5. How many features? The number $n$ and the space ${\mathbb{R}}^n$

• Plain words: $n=2$ means two features (a flat map you can draw). $n=3$ means three features (a room). $n=784$ means a list of 784 numbers — you cannot draw it, but the rules are identical.
• The picture: for $n=2$, ${\mathbb{R}}^2$ is the whole flat sheet of paper; every point on it is one possible example. Larger $n$ just stacks more axes you cannot see.
• Why we name it now: both $\mathbf{x}$ and $\mathbf{w}$ are lists of the same length $n$, so both live in ${\mathbb{R}}^n$. Every geometric idea below — arrow, angle, side of a line — is proven in 2D but works unchanged for any $n$. We picture $n=2$; the algebra carries the rest.

6. The dot product — the weighted sum, renamed

The parent note writes $\mathbf{w}\cdot \mathbf{x}$. That dot is not ordinary multiplication of two lists — it is a specific recipe: multiply matching positions, then add. It is literally the weighted sum from Section 4, extended to all $n$ features and given a shorter name.

What this figure shows: the lavender arrow $\mathbf{w}$ is the "YES direction". Three example arrows test it. The mint arrow leans the same way as $\mathbf{w}$ → its dot product $\mathbf{w}\cdot \mathbf{x}$ is a large positive number. The butter arrow sits at a right angle to $\mathbf{w}$ → its dot product is exactly zero. The coral arrow leans the opposite way → its dot product is negative. That sign — positive / zero / negative — is the whole decision, and it works identically in ${\mathbb{R}}^n$ even though we can only draw ${\mathbb{R}}^2$.

7. The bias — sliding the fence

• Plain words: a built-in head start (or handicap). Even with all features zero, $z=b$.
• The picture: the fence line does not have to pass through the origin $(0,0)$ — the bias slides it sideways. Without $b$, the boundary is stuck at the corner; with $b$, it can sit anywhere.
• Why the topic needs it: in the AND-gate example the boundary is $x_1+x_2=1.5$, a line that does not pass through $(0,0)$. Only the bias can shift it there.

We name the whole total $z$. So $z$ is just shorthand for "weighted sum plus bias".

8. The step function — turning $z$ into a yes/no

• Plain words: if enthusiasm $z$ reaches the line, fire (output 1); otherwise stay quiet (output 0).
• The picture: a flat floor at height 0 that jumps straight up to height 1 the instant $z$ hits zero — a single stair step. See the figure.
• Why the topic needs it: a perceptron must give a decision, not a mood. The step crushes the continuous $z$ into exactly two answers.

What this figure shows: the horizontal axis is $z=\mathbf{w}\cdot \mathbf{x}+b$; the vertical axis is the prediction $\hat{y}$. For all $z<0$ the coral floor sits at height $0$; the instant $z$ reaches $0$ the value leaps to the mint level at height $1$ and stays there. The closed mint dot at $z=0$ and the open coral dot mark the parent's fixed convention: a tie ($z=0$) resolves to class 1, not class 0. The jump is instant — no in-between.

The symbol $\hat{y}$ (read "y-hat") is the perceptron's guess. The plain $y$ (no hat) is the true answer from the data. Keeping them apart is essential: learning happens only when $\hat{y}\ne y$.

9. The decision boundary — where $z=0$

• Plain words: the exact fence between the "predict 1" side and the "predict 0" side.
• The picture: in 2D it is a straight line; the arrow $\mathbf{w}$ pokes out of it at a right angle, pointing toward the class-1 side. In ${\mathbb{R}}^n$ the same equation defines a flat hyperplane, but the picture — a fence with $\mathbf{w}$ poking out perpendicular — is unchanged.
• Why the topic needs it: the whole perceptron is this line. Learning = moving this line until the two colours of dots fall on the correct sides. This is exactly the idea of Linear Separability.

What this figure shows: the lavender line is the boundary $\mathbf{w}\cdot \mathbf{x}+b=0$. The mint dots (class 1) sit on the side the grey $\mathbf{w}$ arrow points to; the coral dots (class 0) sit on the other side. The $\mathbf{w}$ arrow leaves the line at a right angle — it is perpendicular to the fence. If some straight line can split the colours cleanly, the data is linearly separable and the McCulloch-Pitts Neuron-style rule can succeed. If no line works (the XOR case), you need a Multi-Layer Perceptron.

10. The learning rate — how big a nudge

• Plain words: step size. Small $\eta$ = tiny careful nudges; large $\eta$ = big jumps that may overshoot.
• The picture: the fence line jumping a little (small $\eta$) versus leaping (large $\eta$) after a wrong guess.
• Why the topic needs it: it scales the size of every correction. This same "small step in the right direction" idea grows up into Gradient Descent used everywhere in deep learning.

The arrow "$\leftarrow$" means "becomes" — replace the old value with the new one. It is assignment, not equality.

11. The update rule — how the boundary moves

Now we assemble the previous ten ideas into the one moving part of the perceptron: the rule that fixes a mistake.

The piece $(y-\hat{y})$ is the error signal. Because both $y$ and $\hat{y}$ are $0$ or $1$, this error can only be $+1$, $-1$, or $0$.

Algebraic proof that the nudge points the right way. Suppose we make the false-negative mistake: true $y=1$ but $\hat{y}=0$, so the error is $y-\hat{y}=+1$. The update sets ${\mathbf{w}}_{\text{new}}=\mathbf{w}+\eta \mathbf{x},b_{\text{new}}=b+\eta .$ Recompute $z$ for the same example with the new weights: $z_{\text{new}}={\mathbf{w}}_{\text{new}}\cdot \mathbf{x}+b_{\text{new}}=(\mathbf{w}+\eta \mathbf{x})\cdot \mathbf{x}+(b+\eta ).$ Multiply out using the dot-product recipe: $z_{\text{new}}=\underset{=z_{\text{old}}}{\underbrace{\mathbf{w}\cdot \mathbf{x}+b}}+\eta (\mathbf{x}\cdot \mathbf{x})+\eta .$

• WHAT we just did: wrote the new score as the old score plus two extra pieces.
• WHY it matters: $\mathbf{x}\cdot \mathbf{x}=x_1^2+x_2^2+\cdots$ is a sum of squares, so it is always $\ge 0$; and $\eta >0$. Therefore $z_{\text{new}}\ge z_{\text{old}}$ — the score went up, exactly the direction that pushes the guess toward $1$.

For the mirror case (false positive: $y=0$, $\hat{y}=1$, error $-1$) every sign flips: we subtract $\eta \mathbf{x}$, and the same algebra gives $z_{\text{new}}=z_{\text{old}}-\eta (\mathbf{x}\cdot \mathbf{x})-\eta \le z_{\text{old}}$ — the score goes down, pushing the guess toward $0$. In both cases the correction moves the fence toward classifying this point correctly.

What this figure shows: a coral point that was on the wrong (class-0) side of the lavender old boundary. The error is $+1$, so we add $\eta \mathbf{x}$ to $\mathbf{w}$; the mint new boundary has rotated/shifted so the same point now sits on the correct class-1 side. The little grey arrow shows the change $\eta \mathbf{x}$ added to the weight vector — the mechanism behind the move.

Prerequisite map

Everything on the left is a foundation; all roads feed into The Perceptron. If any left-hand box is fuzzy, re-read its section above before opening the parent note.

Equipment checklist

Cover the right side and answer aloud. Reveal to check.