The perceptron model and history: Question bank

This page is a misconception hunt. Every item below targets a place where people quietly go wrong about the perceptron. Read the prompt, say your answer out loud, then reveal. Each answer gives you the reasoning, not just a verdict.

Before we start, one shared vocabulary reminder so no symbol is unearned:

$x$ is the input vector — the list of feature numbers describing one example.
$w$ is the weight vector — one importance number per feature.
$b$ is the bias — a fixed offset added to the sum.
$z = w \cdot x + b$ is the pre-activation, the single number the neuron computes before deciding.
$\overset{y}{^}$ is the prediction $\in {0, 1}$ ; the rule is $\overset{y}{^} = 1$ when $z \geq 0$ , else $0$ (the tie at $z = 0$ goes to class 1).
$y$ is the true label — the correct answer that comes with each training example, also $\in {0, 1}$ . Note the hat: $\overset{y}{^}$ is our guess, $y$ is the truth, and their gap $y - \overset{y}{^}$ is the error, one of $- 1$ , $0$ , or $+ 1$ .
$η$ (the Greek letter "eta") is the learning rate — a small positive number, typically $0.01$ to $1$ , that sets how big a step we take when correcting a mistake. Small $η$ = timid nudges, large $η$ = bold leaps.
The decision boundary is the set of points where $z = 0$ : a line in 2D, a plane in 3D, a hyperplane in general.

One picture to hold in your head

Everything below is easier if you can see the geometry. Here is the whole perceptron in one blueprint diagram — refer back to it whenever a question talks about "sides", "perpendicular", "shift" or "tilt".

Figure — The perceptron model and history

The amber line is the boundary $z = 0$ . The cyan arrow is $w$ , planted on the line and pointing into the class-1 side (where $z > 0$ ). Points on the arrow's side are labelled 1; points behind it are labelled 0.

Why the learning updates work — a picture

When a class-1 point is wrongly labelled 0, we do $w \leftarrow w + η x$ . This tips the arrow toward that point, which drags the boundary until the point ends up on the correct side. Watch the arrow swing:

Why convergence is guaranteed (when separable)

Intuition The geometric idea behind the Convergence Theorem

Suppose some perfect weight vector $w^{⋆}$ exists that separates the data — call the boundary it defines the "goal". Every time our perceptron makes a mistake and updates, two things provably happen at once:

Our $w$ becomes more aligned with $w^{⋆}$ (the dot product $w \cdot w^{⋆}$ grows by at least a fixed amount each mistake — the update always pushes us toward the goal direction).
Our $w$ can only grow so fast in length (its squared length increases by a bounded amount per mistake, because a mistake means $z$ had the wrong sign).

Point 1 says alignment climbs linearly in the number of mistakes $k$ ; point 2 says length grows only like $k$ . But a dot product can never exceed the product of lengths (that's just "cosine $\leq 1$ "). Something growing like $k$ cannot forever stay under something growing like $k$ — so the mistake count $k$ must stop at a finite value. That finite stopping point is convergence.

The figure below shows the two racing quantities: alignment (climbing straight) versus the length ceiling (climbing only as a square root). Where they would cross is where mistakes must end.

The one honest caveat: this whole argument assumed a separating $w^{⋆}$ exists. If the data isn't linearly separable, no goal exists, the alignment argument collapses, and the mistakes never stop.

True or false — justify

True or false: A perceptron can perfectly classify any dataset given enough training.

False. It only converges when the classes are linearly separable; if no single straight boundary can split them (e.g. XOR), the weights oscillate forever (see the convergence figure — no goal

w^{⋆}

exists).

True or false: If you scale every weight and the bias by the same positive constant, the decision boundary moves.

False. Multiplying

w

and

b

c > 0

turns

z = 0

into

cz = 0

, the same set of points — the boundary is unchanged; only the magnitude of

z

(its confidence-like size) scales.

True or false: The bias

b

changes the tilt (slope) of the decision line.

False. The slope in 2D is

- w_{1} / w_{2}

, which has no

b

in it. The bias only shifts the line parallel to itself; the weights set the tilt.

True or false: The perceptron outputs a probability of belonging to class 1.

False. It outputs a hard

0

1

from a step function — no in-between. If you want a probability you need a smooth activation like the one in Logistic Regression.

True or false: The Perceptron Convergence Theorem promises the algorithm finds the best separating line.

False. It promises some separating hyperplane in finite steps, not the widest-margin one. Finding the maximum-margin boundary is the job of Support Vector Machines.

True or false: On a correctly classified example the perceptron still nudges its weights a little.

False. The update is scaled by the error

(y - \overset{y}{^})

, which is

0

when the prediction is right, so the weights are left exactly unchanged.

True or false: The weight vector

w

points along the decision boundary.

False.

w

is perpendicular to the boundary (see the s01 diagram); it points from the boundary toward the class-1 half-space, since moving in the

w

direction increases

z

True or false: A perceptron with all weights zero and bias zero classifies every point as class 0.

False. With

w = 0, b = 0

every point gives

z = 0

, and

z = 0

resolves to class 1 by our tie convention — so everything is labelled 1, not 0.

True or false: Two different weight-and-bias sets can never produce the identical classifier.

False. Any positive rescaling (and infinitely many other combinations) gives the same boundary and same labels, so the

(w, b)

that solves a problem is never unique.

True or false: The 1969 Minsky–Papert critique proved neural networks can never work.

False. It proved a single perceptron can't solve non-separable problems like XOR; stacking layers (Multi-Layer Perceptron) removes that limit, which is what ended the AI Winter.

Spot the error

"To separate cats from dogs a perceptron draws a curve around the cat points." — What's wrong?

A perceptron can only draw a straight boundary (a hyperplane,

z = 0

). It cannot bend into a curve, so it fails whenever the true separation isn't linear.

"Learning rate

η

decides whether a misclassified point causes an update." — What's wrong?

The error

(y - \overset{y}{^})

decides whether to update;

η

only scales how big the update is. A tiny

η

still updates on every mistake, just by less.

"For the AND gate I set

b = 0

so that

(0, 0)

maps to 0." — What's wrong?

With

b = 0

, the point

(0, 0)

gives

z = 0

, which resolves to class 1, not 0. AND needs a negative bias (e.g.

- 1.5

) so the empty input falls below threshold.

"Since the perceptron sums the inputs, feature order matters." — What's wrong?

Addition is commutative:

w_{1} x_{1} + w_{2} x_{2} = w_{2} x_{2} + w_{1} x_{1}

. Reordering features (and their matching weights) gives the identical

z

; order carries no information.

"I'll normalise labels to

{- 1, + 1}

but keep the update rule

w \leftarrow w + η (y - \overset{y}{^}) x

unchanged." — What's wrong?

The

{0, 1}

update rule assumes

y - \overset{y}{^} \in {- 1, 0, + 1}

. Switching to

{- 1, + 1}

labels changes the arithmetic of the error term, so you must rederive the rule or your steps double in size.

"The bias is just an extra input, so I can drop it to simplify." — What's wrong?

Dropping

b

forces the boundary through the origin. Many datasets (like AND) need a boundary that doesn't pass through

(0, 0)

, so removing the bias makes them unsolvable.

"My data is separable but training never stops, so the theorem is wrong." — What's wrong?

Almost certainly the data isn't actually separable, or a coding bug prevents updates on ties. The convergence theorem is proven (see the racing-quantities figure); endless looping is evidence the linear-separability assumption is violated.

Why questions

Why is the update

w \leftarrow w + η (y - \overset{y}{^}) x

and not

w \leftarrow w + η (y - \overset{y}{^})

Multiplying by

x

moves

w

in the direction of the offending point (see the swinging-arrow figure), which is exactly what raises or lowers that point's

z

; a scalar shift would move

w

blindly and might not fix the specific mistake.

Why does a step function make the perceptron impossible to train with Gradient Descent?

The step function is flat everywhere except one point, so its derivative is

0

(or undefined) — gradient descent gets no slope to follow. Smooth activations were introduced precisely to give Backpropagation a usable gradient.

Why is the perceptron called a linear classifier when the step function is very non-linear?

"Linear" refers to the boundary

z = 0

, which is a linear equation in

x

. The step is just how we read off the side; the shape that separates the classes is a straight hyperplane.

Why does the perceptron trace back to the McCulloch-Pitts Neuron?

McCulloch–Pitts (1943) gave the all-or-nothing "sum inputs, fire past a threshold" model; Rosenblatt's key addition was a learning rule that adjusts the weights from data instead of hand-setting them.

Why does a larger learning rate not always mean faster learning?

A big

η

can overshoot — a single correction hurls the boundary past its target, undoing progress and causing wild oscillation. Convergence needs steps small enough to settle, not just large.

Why is the tie case

z = 0

worth pinning down explicitly?

Because

z = 0

is exactly the boundary, and whichever class we assign it changes predictions on edge points and the AND/OR bias values; leaving it ambiguous makes "correct" solutions look wrong.

Why does the Convergence Theorem guarantee a finite number of mistakes rather than merely "eventually"?

Because alignment with the goal

w^{⋆}

grows like the mistake count

k

while

w

's length grows only like

k

, and a dot product can't exceed the product of lengths — so

k

is forced to stop at a concrete finite bound, not just trend downward.

Edge cases

What does the perceptron predict for a point lying exactly on the decision boundary?

It gives

z = 0

, which by our fixed convention resolves to class 1 — the tie always goes up, never down.

If a dataset has one feature that is identical for every example, what happens to its weight?

That feature carries no discriminating information, so it behaves like part of the bias; its weight can drift, but changing it just shifts

z

by a constant and never helps separate the classes.

What happens if two training points have identical features but opposite labels?

The data is not linearly separable at that spot — no boundary can put the same

x

on two sides — so the perceptron will keep flip-flopping and never converge.

What does the boundary look like the moment before the very first update, starting from

w = 0, b = 0

There is no real boundary:

z = 0

for all points, so the whole space is labelled class 1 and the first misclassified negative example is what first tilts

w

away from zero.

If you feed a perceptron a brand-new point far outside the training range, what governs its label?

Only which side of the infinite hyperplane it falls on — the perceptron extrapolates a straight boundary forever, with no notion of "too far from the data" or uncertainty.

What is the smallest number of features (inputs) for which a single perceptron can already fail to separate the data?

Two features suffice: XOR lives in 2D with four points and no straight line can separate its two classes, which is the classic single-perceptron failure.

Recall Quick self-test

The bias only shifts / never tilts the boundary ::: shifts The error term $(y - \overset{y}{^})$ decides whether to update; $η$ decides ::: how big the update is $z = 0$ resolves to class ::: 1 XOR is unsolvable by one perceptron because it is not ::: linearly separable $y$ is the truth, $\overset{y}{^}$ is the guess; their difference is called the ::: error