The perceptron model and history: Exercises — AI-ML deep dive

This is a self-testing workout for the perceptron. Every problem states its level (L1 Recognition → L5 Mastery). Try it first, then open the collapsible solution. Numbers are all machine-checked.

Fixed convention everywhere: $z = w \cdot x + b$ , and $\overset{y}{^} = 1$ when $z \geq 0$ , $\overset{y}{^} = 0$ when $z < 0$ . So a tie ( $z = 0$ ) means class 1.

Level 1 — Recognition

You just read a formula; can you plug numbers into it?

Exercise 1.1 — Forward pass

A perceptron has weights $w = [2, - 1]$ and bias $b = 0.5$ . What does it predict for the input $x = [1, 3]$ ?

Recall Solution

WHY form $z = w \cdot x + b$ : each weight $w_{i}$ says "how strongly does feature $i$ vote for class 1?" The dot product $w \cdot x$ tallies all those weighted votes into one number, and $b$ shifts the whole tally up or down (the built-in threshold). So $z$ is the neuron's single "how excited am I?" score. WHAT we do: compute that score, then apply the step rule. $z = (2) (1) + (- 1) (3) + 0.5 = 2 - 3 + 0.5 = - 0.5$ Since $- 0.5 < 0$ , the rule gives $\overset{y}{^} = 0$ . Answer: $\overset{y}{^} = 0$ .

Exercise 1.2 — The tie case

Same perceptron $w = [2, - 1]$ , $b = 0.5$ . Predict for $x = [1, 2.5]$ .

Recall Solution

WHY the step at zero is the boundary: the equation $z = 0$ is exactly the dividing line between the two classes — one side has $z > 0$ (votes for 1), the other has $z < 0$ (votes for 0). The step function turns "which side of the line am I on?" into a crisp 0/1 label, and points sitting on the line ( $z = 0$ ) are the tie our convention sends to class 1. $z = (2) (1) + (- 1) (2.5) + 0.5 = 2 - 2.5 + 0.5 = 0$ $z = 0$ is a tie — this input lands exactly on the boundary. Our convention resolves ties to class 1. Answer: $\overset{y}{^} = 1$ .

Exercise 1.3 — Read the boundary

For $w = [3, 4]$ , $b = - 12$ , the decision boundary is the line $3 x_{1} + 4 x_{2} - 12 = 0$ . Write it in slope–intercept form $x_{2} = m x_{1} + c$ .

Recall Solution

WHY slope–intercept: it lets us draw the line — $m$ is steepness, $c$ is where it crosses the vertical axis. $4 x_{2} = - 3 x_{1} + 12 \Rightarrow x_{2} = - \frac{3}{4} x_{1} + 3$ Answer: slope $m = - \frac{3}{4}$ , intercept $c = 3$ .

Level 2 — Application

Now design or run the algorithm, not just evaluate it.

Exercise 2.1 — Build a NOT gate

A NOT gate has one input: $x = 0 \to y = 1$ and $x = 1 \to y = 0$ . Find weights $w$ and bias $b$ for a single-input perceptron that computes it.

Recall Solution

WHAT we need: $\overset{y}{^} = 1$ when $x = 0$ , $\overset{y}{^} = 0$ when $x = 1$ . Since output falls as input rises, the weight must be negative. Try $w = - 1$ , $b = 0.4$ :

$x = 0$ : $z = (- 1) (0) + 0.4 = 0.4 \geq 0 \Rightarrow \overset{y}{^} = 1$ ✓
$x = 1$ : $z = (- 1) (1) + 0.4 = - 0.6 < 0 \Rightarrow \overset{y}{^} = 0$ ✓ Answer: $w = - 1, b = 0.4$ (any $w < 0$ with $0 < b < ∣ w ∣$ works — e.g. $b = 0.5$ also works).

Exercise 2.2 — One update step

Start with $w = [0, 0]$ , $b = 0$ , learning rate $η = 0.5$ . The example is $x = [1, 1]$ , true label $y = 0$ . Perform one perceptron update and report the new $w, b$ .

Recall Solution

Predict: $z = 0 \geq 0 \Rightarrow \overset{y}{^} = 1$ . Error: $y - \overset{y}{^} = 0 - 1 = - 1$ (a false positive). Update $w \leftarrow w + η (y - \overset{y}{^}) x$ : $w_{1} = 0 + 0.5 (- 1) (1) = - 0.5, w_{2} = 0 + 0.5 (- 1) (1) = - 0.5$ $b = 0 + 0.5 (- 1) = - 0.5$ WHY these moved down: we said "spam" wrongly, so we push $z$ down for this point next time. Answer: $w = [- 0.5, - 0.5], b = - 0.5$ .

Exercise 2.3 — NAND gate weights

Verify that $w = [- 1, - 1]$ , $b = 1.5$ computes NAND (output is 1 unless both inputs are 1).

Recall Solution

NAND truth: $(0, 0) \to 1, (0, 1) \to 1, (1, 0) \to 1, (1, 1) \to 0$ .

$(0, 0)$ : $z = 1.5 \geq 0 \Rightarrow 1$ ✓
$(0, 1)$ : $z = - 1 + 1.5 = 0.5 \geq 0 \Rightarrow 1$ ✓
$(1, 0)$ : $z = - 1 + 1.5 = 0.5 \geq 0 \Rightarrow 1$ ✓
$(1, 1)$ : $z = - 2 + 1.5 = - 0.5 < 0 \Rightarrow 0$ ✓ All four match. NAND is confirmed. (Notice it is the AND weights, negated with a flipped bias.)

Level 3 — Analysis

Reason about why boundaries and separability behave as they do.

The figure below has two panels, both drawn on a chalkboard grid.

Left panel: the boundary line for $w = [1, 2]$ , $b = - 4$ (the equation $x_{1} + 2 x_{2} - 4 = 0$ ), drawn in chalk white. A blue arrow starts on the line and points in the direction of $w$ — notice it meets the line at a right angle. The yellow dot is $P = [4, 4]$ (labeled "class 1"), sitting on the side the blue arrow points toward; the pink dot is $Q = [0, 0]$ (labeled "class 0"), on the side it points away from. Axes are $x_{1}$ (horizontal) and $x_{2}$ (vertical).
Right panel: the four XOR points on the unit square. Yellow dots are the class-1 corners $(0, 1)$ and $(1, 0)$ ; pink dots are the class-0 corners $(0, 0)$ and $(1, 1)$ . A dashed blue line shows one attempted separator — trace it and you will see it always leaves one wrong-coloured dot on each side.

Figure — The perceptron model and history

Exercise 3.1 — Which side of the line?

A perceptron has $w = [1, 2]$ , $b = - 4$ . The weight vector $w$ points perpendicular to the boundary (this is the blue arrow in the left panel above). For each point, say the predicted class and confirm it sits on the side the arrow points to (class 1) or away from it (class 0): $P = [4, 4]$ , $Q = [0, 0]$ .

Recall Solution

WHY perpendicular: $z = w \cdot x + b$ grows fastest along $w$ ; the boundary is the level set $z = 0$ , always at right angles to that direction of steepest increase. That is the right angle you see between the blue arrow and the white line.

$P = [4, 4]$ : $z = 4 + 8 - 4 = 8 \geq 0 \Rightarrow \overset{y}{^} = 1$ . Positive $z$ = same side the arrow points (the yellow dot). ✓
$Q = [0, 0]$ : $z = 0 + 0 - 4 = - 4 < 0 \Rightarrow \overset{y}{^} = 0$ . Negative $z$ = opposite side (the pink dot). ✓ Answer: $P \to 1$ , $Q \to 0$ .

Exercise 3.2 — Why XOR is impossible

XOR is $(0, 0) \to 0, (0, 1) \to 1, (1, 0) \to 1, (1, 1) \to 0$ . Prove no perceptron can compute it. (Use the right panel of the figure to picture the four points.)

Recall Solution

WHAT we must find: a line putting the two "1" points ${(0, 1), (1, 0)}$ on one side and the two "0" points ${(0, 0), (1, 1)}$ on the other. Picture first (intuition, not proof): in the right panel the two yellow (class-1) corners and the two pink (class-0) corners alternate around the square. A single straight cut can certainly slice through a diagonal — that is why the picture alone is not a proof. To be rigorous we turn to algebra. Algebraic proof. Suppose weights $w_{1}, w_{2}, b$ exist. The perceptron's four required inequalities are:

$(0, 0) \to 0$ : $b < 0$ — call this (A)
$(1, 1) \to 0$ : $w_{1} + w_{2} + b < 0$ — call this (B)
$(0, 1) \to 1$ : $w_{2} + b \geq 0$ — call this (C)
$(1, 0) \to 1$ : $w_{1} + b \geq 0$ — call this (D)

Add (C) and (D): $(w_{1} + b) + (w_{2} + b) = w_{1} + w_{2} + 2 b \geq 0$ . Add (A) and (B): $b + (w_{1} + w_{2} + b) = w_{1} + w_{2} + 2 b < 0$ . The same quantity $w_{1} + w_{2} + 2 b$ is required to be both $\geq 0$ and $< 0$ — impossible. So no such weights exist. This is exactly the Linear Separability limit Minsky & Papert publicized — solved later by the Multi-Layer Perceptron.

Exercise 3.3 — Distance from boundary

For $w = [3, 4]$ , $b = - 12$ , how far is the point $x = [4, 4]$ from the decision boundary? (The signed distance is $\frac{w \cdot x + b}{∥ w ∥}$ .)

Recall Solution

WHY divide by $∥ w ∥$ : $z$ alone scales if we double the weights, but the geometry of the line doesn't change. Dividing by the length $∥ w ∥ = 3^{2} + 4^{2} = 5$ cancels that scaling, giving true perpendicular distance. $z = (3) (4) + (4) (4) - 12 = 12 + 16 - 12 = 16, distance = \frac{16}{5} = 3.2$ Answer: $3.2$ units, on the class-1 side (positive).

Level 4 — Synthesis

Combine the pieces: run learning to convergence, connect to neighbouring ideas.

Exercise 4.1 — Train OR from scratch

Start $w = [0, 0]$ , $b = 0$ , $η = 1$ . Train on OR: $(0, 0) \to 0, (0, 1) \to 1, (1, 0) \to 1, (1, 1) \to 1$ , presenting rows in order and cycling until all four are correct. Report the first weight set that classifies every row.

Recall Solution

Degenerate start — read this first. At initialization $w = [0, 0]$ , $b = 0$ , so for every input $z = 0 \cdot x_{1} + 0 \cdot x_{2} + 0 = 0$ . Since $z = 0$ is a tie, the fresh perceptron always predicts class 1 — regardless of the input! That is why the very first row with true label $0$ must produce an error and trigger an update. Don't be surprised the machine starts by getting every "class 0" example wrong: it literally has no opinion yet and defaults to 1.

Now track $z$ , prediction, error, update each step.

Epoch 1

$(0, 0), y = 0$ : $z = 0 \Rightarrow \overset{y}{^} = 1$ (the degenerate all-ones prediction), err $= - 1$ . Update: $w = [0, 0]$ (times 0), $b = 0 + 1 (- 1) = - 1$ . → $w = [0, 0], b = - 1$ .
$(0, 1), y = 1$ : $z = 0 - 1 = - 1 \Rightarrow \overset{y}{^} = 0$ , err $= + 1$ . Update: $w_{2} = 0 + 1 (1) (1) = 1$ , $b = - 1 + 1 = 0$ . → $w = [0, 1], b = 0$ .
$(1, 0), y = 1$ : $z = 0 + 0 = 0 \Rightarrow \overset{y}{^} = 1$ , err $= 0$ . No change. → $w = [0, 1], b = 0$ .
$(1, 1), y = 1$ : $z = 0 + 1 + 0 = 1 \geq 0 \Rightarrow \overset{y}{^} = 1$ , err $= 0$ . No change.

Epoch 2 — check all rows with $w = [0, 1], b = 0$ :

$(0, 0)$ : $z = 0 \Rightarrow \overset{y}{^} = 1$ , but $y = 0$ . Still wrong! err $= - 1$ . Update: $b = 0 - 1 = - 1$ . → $w = [0, 1], b = - 1$ .
$(0, 1)$ : $z = 1 - 1 = 0 \geq 0 \Rightarrow 1$ ✓
$(1, 0)$ : $z = 0 - 1 = - 1 < 0 \Rightarrow 0$ , but $y = 1$ . Wrong. err $= + 1$ . Update: $w_{1} = 0 + 1 = 1$ , $b = - 1 + 1 = 0$ . → $w = [1, 1], b = 0$ .
$(1, 1)$ : $z = 2 \geq 0 \Rightarrow 1$ ✓

Epoch 3 — check all with $w = [1, 1], b = 0$ :

$(0, 0)$ : $z = 0 \Rightarrow 1$ , but $y = 0$ . Wrong. err $= - 1$ . Update: $b = 0 - 1 = - 1$ . → $w = [1, 1], b = - 1$ .
$(0, 1)$ : $z = 1 - 1 = 0 \geq 0 \Rightarrow 1$ ✓
$(1, 0)$ : $z = 1 - 1 = 0 \geq 0 \Rightarrow 1$ ✓
$(1, 1)$ : $z = 2 - 1 = 1 \geq 0 \Rightarrow 1$ ✓

Epoch 4 — verify $w = [1, 1], b = - 1$ on all rows:

$(0, 0)$ : $z = - 1 < 0 \Rightarrow 0$ ✓
$(0, 1)$ : $z = 0 \geq 0 \Rightarrow 1$ ✓ · $(1, 0)$ : $z = 0 \geq 0 \Rightarrow 1$ ✓ · $(1, 1)$ : $z = 1 \geq 0 \Rightarrow 1$ ✓

Answer: $w = [1, 1], b = - 1$ classifies all four. This matches the hand-picked OR weights from the parent note (up to the boundary passing exactly through the two mid-points). The Perceptron Convergence Theorem guarantees this: OR is linearly separable, so learning terminates in finite passes.

Exercise 4.2 — Step vs. sigmoid

The perceptron uses a hard step. Logistic Regression replaces it with the sigmoid $σ (z) = \frac{1}{1 + e ^{- z}}$ . Compute $σ (0)$ and explain why this smooth version enables Gradient Descent where the step function cannot.

Recall Solution

$σ (0) = \frac{1}{1 + e ^{0}} = \frac{1}{1 + 1} = 0.5$ WHY it matters: the step function is flat everywhere except an infinite jump at $z = 0$ — its slope (derivative) is $0$ almost everywhere and undefined at the jump. Gradient Descent needs a nonzero slope to know which way to nudge weights. The sigmoid slopes smoothly ( $σ^{'} (z) = σ (z) (1 - σ (z)) > 0$ ), so every point gives a usable gradient. This smoothing is what makes Backpropagation possible in deep networks. See Activation Functions. Answer: $σ (0) = 0.5$ .

Level 5 — Mastery

Prove, generalize, or expose a subtlety.

Exercise 5.1 — Bias as an extra weight

Show that the bias $b$ can be absorbed as a weight $w_{0}$ on a constant "dummy" input $x_{0} = 1$ , so that $w \cdot x + b$ becomes a single dot product $\tilde{w} \cdot \tilde{x}$ .

Recall Solution

Define augmented vectors $\tilde{x} = [1, x_{1}, \dots, x_{n}]$ and $\tilde{w} = [w_{0}, w_{1}, \dots, w_{n}]$ with $w_{0} := b$ . $\tilde{w} \cdot \tilde{x} = w_{0} \cdot 1 + w_{1} x_{1} + \dots + w_{n} x_{n} = b + w \cdot x .$ Identical to the original. WHY this is useful: the update rule for $b$ , namely $b \leftarrow b + η (y - \overset{y}{^}) \cdot 1$ , is now just the ordinary weight update on a feature that is always $1$ . One formula handles everything — this is the "add a 1s column" trick every ML library uses.

Exercise 5.2 — Scaling doesn't change the classifier

Take any solution $(w, b)$ . Prove that $(2 w, 2 b)$ produces the exact same predictions on every input, yet the perceptron update treats them differently. Verify on $w = [1, 2], b = - 4$ , point $[4, 4]$ .

Recall Solution

Same predictions: $z^{'} = 2 w \cdot x + 2 b = 2 (w \cdot x + b) = 2 z$ . Since $2 > 0$ , $z^{'} \geq 0 ⟺ z \geq 0$ . Every sign is preserved, so $\overset{y}{^}$ is identical for all inputs. Check: original $z = 4 + 8 - 4 = 8 \geq 0 \Rightarrow 1$ . Scaled $z^{'} = 8 + 16 - 8 = 16 \geq 0 \Rightarrow 1$ . Same class. ✓ The subtlety: although the decision boundary is unchanged, the raw magnitude $z$ doubled — so the confidence-like value and the step size effect differ. This is precisely why the honest margin uses $z /∥ w ∥$ (Ex 3.3): $16/∥2 w ∥ = 16/ (25) = 8/ 5$ , identical to $8/ 5$ from the original. Normalized margin is invariant; raw $z$ is not.

Exercise 5.3 — A non-separable trap

Consider one-dimensional data: $x = 1 \to y = 1$ , $x = 2 \to y = 0$ , $x = 3 \to y = 1$ . Show no single perceptron $(w, b)$ can classify all three, and state what the algorithm does forever.

Recall Solution

A 1-D perceptron splits the line at one threshold $t = - b / w$ . With $w > 0$ : points right of $t$ are class 1, left are class 0 (or the mirror if $w < 0$ ). Either way, one threshold gives one contiguous class-1 region. But here the class-1 points ${1, 3}$ sit on both sides of the class-0 point ${2}$ — the labels go $1, 0, 1$ , an alternating pattern (the same structure that kills XOR). No single cut separates them. What the algorithm does: it never converges. It keeps making a mistake on some point every epoch and updating forever, weights oscillating. The Perceptron Convergence Theorem guarantees termination only for linearly separable data — this data is not. Practical fix: cap the epochs (the "pocket" variant keeps the best-so-far weights), or move to a Multi-Layer Perceptron.

Recall Self-check ledger (answers only)

1.1 ::: $\overset{y}{^} = 0$ 1.2 ::: $\overset{y}{^} = 1$ (tie → class 1) 1.3 ::: slope $- 3/4$ , intercept $3$ 2.1 ::: $w = - 1, b = 0.4$ 2.2 ::: $w = [- 0.5, - 0.5], b = - 0.5$ 2.3 ::: NAND confirmed 3.1 ::: $P \to 1, Q \to 0$ 3.2 ::: impossible (contradiction from the four inequalities) 3.3 ::: distance $3.2$ 4.1 ::: $w = [1, 1], b = - 1$ 4.2 ::: $σ (0) = 0.5$ 5.1 ::: $\tilde{w} \cdot \tilde{x} = b + w \cdot x$ 5.2 ::: same class both times; margin invariant 5.3 ::: non-separable, never converges