The perceptron model and history

Overview

The perceptron is the simplest artificial neuron and the foundation of all modern neural networks. Created by Frank Rosenblatt in 1957, it was the first algorithm that could learn from data to classify inputs into two categories. Understanding the perceptron is essential because every deep learning architecture—from simple feedforward networks to transformers—builds on the fundamental ideas introduced here.

Historical Context

The Timeline:

• 1943: McCulloch & Pitts propose mathematical model of biological neurons
• 1957: Rosenblatt invents the perceptron at Cornell Aeronautical Laboratory
• 1958: First hardware implementation (Mark I Perceptron) - could recognize simple patterns
• 1969: Minsky & Papert publish "Perceptrons", showing fundamental limitations (can't solve XOR)
• mid-1970s: The first "AI Winter" begins - major funding cuts and a sharp drop in neural-network research
• late 1980s: A second AI Winter follows after early expert-system hype fades
• 1986: Backpropagation (Rumelhart, Hinton, Williams popularization) revives neural networks by solving multi-layer training
• Today: Perceptrons are the building blocks inside billion-parameter models

The AI Winter happened because the perceptron's limitations were overinterpreted—people thought neural networks would never work, when they just needed multiple layers.

The Perceptron Model

Where:

• $\mathbf{x} = [x_1, x_2, \ldots, x_n]$ is the input vector (features)
• $\mathbf{w} = [w_1, w_2, \ldots, w_n]$ are weights (importance of each feature)
• $b$ is the bias term (threshold for activation)
• $\hat{y} \in \{0, 1\}$ is the prediction

Threshold convention (fixed): Throughout this note we use $\hat{y} = 1$ when $z \geq 0$ and $\hat{y} = 0$ when $z < 0$. So $z = 0$ is a tie that resolves to class 1, not class 0. We apply this consistently everywhere below.

Biological Inspiration

Real neurons have:

1. Dendrites (inputs) → Input features $x_i$
2. Synapses (connection strengths) → Weights $w_i$
3. Cell body (sums signals) → Weighted sum $\sum w_i x_i + b$
4. Axon (fires or doesn't) → Activation function (step function)

WHY this model? Biological neurons sum incoming signals, and if the sum exceeds a threshold, they "fire" an electrical spike. The perceptron mathematically mimics this all-or-nothing behavior.

Deriving the Perceptron from First Principles

GOAL: Create a function that separates two classes of data points.

Step 1: Linear Combination The most basic way to combine multiple inputs is a weighted sum: $z = w_1 x_1 + w_2 x_2 + \cdots + w_n x_n + b$

WHY? Each weight $w_i$ tells us "how important is feature $i$ for this decision?" Positive weight = feature pushes toward class 1; negative weight = pushes toward class 0.

Step 2: Decision Boundary We need to convert the continuous value $z$ into a binary decision. The simplest rule: $\hat{y} = \begin{cases} 1 & \text{if } z \geq 0 \\ 0 & \text{if } z < 0 \end{cases}$

WHY this threshold? The line $z = 0$ (equivalent to $\mathbf{w} \cdot \mathbf{x} + b = 0$) divides the input space into two half-spaces. All points on one side get labeled 1, all on the other get labeled 0.

Step 3: Geometric Interpretation The equation $\mathbf{w} \cdot \mathbf{x} + b = 0$ defines a hyperplane (a line in 2D, plane in 3D, etc.). The weight vector $\mathbf{w}$ is perpendicular to this hyperplane.

WHY? The dot product $\mathbf{w} \cdot \mathbf{x}$ measures how much $\mathbf{x}$ points in the direction of $\mathbf{w}$. Points on one side of the hyperplane have positive dot product, points on the other have negative.

Equivalently (in 2D): $w_1 x_1 + w_2 x_2 + b = 0$ $x_2 = -\frac{w_1}{w_2} x_1 - \frac{b}{w_2}$

This is a straight line with slope $-w_1/w_2$ and y-intercept $-b/w_2$.

The Perceptron Learning Algorithm

HOW does the perceptron learn? By adjusting weights when it makes mistakes.

Where $\eta$ is the learning rate (typically 0.01 to 1).

Derivation - WHY this rule?

Case 1: True label $y = 1$ but we predicted $\hat{y} = 0$ (false negative)

• Error: $y - \hat{y} = 1 - 0 = +1$
• Update: $\mathbf{w} \leftarrow \mathbf{w} + \eta \mathbf{x}$
• Effect: Increases $\mathbf{w} \cdot \mathbf{x}$, making prediction more likely to be 1 next time

Case 2: True label $y = 0$ but we predicted $\hat{y} = 1$ (false positive)

• Error: $y - \hat{y} = 0 - 1 = -1$
• Update: $\mathbf{w} \leftarrow \mathbf{w} - \eta \mathbf{x}$
• Effect: Decreases $\mathbf{w} \cdot \mathbf{x}$, making prediction more likely to be 0 next time

Case 3: Correct prediction

• Error: $y - \hat{y} = 0$
• Update: No change (weights stay same)

WHY does this converge? The Perceptron Convergence Theorem (1962) proves that if the data is linearly separable, the algorithm will find a separating hyperplane in finite steps.

Worked Examples

Truth table (treating False=0, True=1):

Solution - Finding Weights Manually:

We need: $w_1 x_1 + w_2 x_2 + b \geq 0$ when $x_1 = x_2 = 1$, and $< 0$ otherwise.

Try $w_1 = 1, w_2 = 1, b = -1.5$:

• $(0,0)$: $1(0) + 1(0) - 1.5 = -1.5 < 0$ → predicts 0 ✓
• $(0,1)$: $1(0) + 1(1) - 1.5 = -0.5 < 0$ → predicts 0 ✓
• $(1,0)$: $1(1) + 1(0) - 1.5 = -0.5 < 0$ → predicts 0 ✓
• $(1,1)$: $1(1) + 1(1) - 1.5 = 0.5 \geq 0$ → predicts 1 ✓

WHY these weights? Each input needs weight ~1 because both must be "on" to activate. Bias $-1.5$ ensures that even one input alone ($z = 1 - 1.5 = -0.5$) isn't enough.

Decision boundary: $x_1 + x_2 = 1.5$ (a line passing between (1,0)/(0,1) and (1,1))

Try $w_1 = 1, w_2 = 1, b = -0.5$:

• $(0,0)$: $-0.5 < 0$ → predicts 0 ✓
• $(0,1)$: $1 - 0.5 = 0.5 \geq 0$ → predicts 1 ✓
• $(1,0)$: $1 - 0.5 = 0.5 \geq 0$ → predicts 1 ✓
• $(1,1)$: $2 - 0.5 = 1.5 \geq 0$ → predicts 1 ✓

WHY? Only need one input to be 1, so bias $-0.5$ is a lower threshold.

Step-by-step training ($\eta = 0.1$, convention $\hat{y}=1$ when $z \geq 0$):

Initialize: $w_1 = 0, w_2 = 0, b = 0$

Iteration 1, Email 1: $\mathbf{x} = [2, 5], y = 1$

• Prediction: $z = 0(2) + 0(5) + 0 = 0 \geq 0$ → $\hat{y} = 1$ (consistent with our convention)
• Error: $1 - 1 = 0$ → no update needed (already correct)
• WHY? Because $z = 0$ resolves to class 1, and the true label is 1, so this example is classified correctly on the very first pass.

Email 2: $\mathbf{x} = [0, 1], y = 0$

• Prediction: $z = 0(0) + 0(1) + 0 = 0 \geq 0$ → $\hat{y} = 1$
• Error: $0 - 1 = -1$
• Update: $w_1 = 0 + 0.1(-1)(0) = 0$
• Update: $w_2 = 0 + 0.1(-1)(1) = -0.1$
• Update: $b = 0 + 0.1(-1) = -0.1$
• WHY? We predicted spam but it's not spam, so reduce the influence of "!" marks and lower the bias.

Email 3: $\mathbf{x} = [3, 7], y = 1$

• Prediction: $z = 0(3) + (-0.1)(7) + (-0.1) = -0.8 < 0$ → $\hat{y} = 0$
• Error: $1 - 0 = +1$
• Update: $w_1 = 0 + 0.1(1)(3) = 0.3$
• Update: $w_2 = -0.1 + 0.1(1)(7) = 0.6$
• Update: $b = -0.1 + 0.1(1) = 0$
• WHY? We missed a real spam email, so strengthen weights toward positive for these high-count features.

Continue for remaining examples and multiple epochs until no mistakes remain in a full pass...

After convergence: The perceptron finds a line separating spam from non-spam emails based on these features.

Limitations and the XOR Problem

XOR truth table:

Try to find a line separating the classes:

• Points $(0,1)$ and $(1,0)$ must be on the "1" side
• Points $(0,0)$ and $(1,1)$ must be on the "0" side

IMPOSSIBLE with a single straight line! No linear boundary can separate these points.

Why this mistake feels right: "The perceptron can learn AND and OR, so surely it can learn any logic function?" The fix: XOR requires a non-linear decision boundary (a curve or multiple lines). Solution: multi-layer perceptrons (neural networks with hidden layers) can solve XOR.

Steel-manning the limitation: The perceptron isn't "broken"—it's solving the problem it was designed for (linearly separable data). The real mistake was expecting a linear model to solve non-linear problems.

Why it feels right: More training often helps with noisy data or poor initialization.

The fix: If data is not linearly separable, infinite training won't help. The perceptron convergence theorem only applies when a solution exists. Check if your problem needs a non-linear model first.

Modern Relevance

Why study perceptrons in 2026?

1. Building block: Every neuron in modern networks (BERT, GPT, ResNet) is a perceptron followed by a non-linear activation
2. Initialization: Understanding linear separability helps debug why networks fail
3. Interpretability: Perceptron weights are the only fully interpretable neural model
4. SVMs: Support Vector Machines are sophisticated perceptrons with margin optimization
5. Historical literacy: Can't appreciate backpropagation without understanding what it solved

The perceptron's simplicity is its strength—it's the minimal viable learning algorithm.

You make a simple rule: "If (age × 3) + (has ticket × 10) is more than 15, let them in."

At first, your rule is bad (maybe you set it to 5 instead of 15), some kids get in and some adults are rejected. Each time you mess up, you adjust your numbers a tiny bit. If you let a kid in by mistake, you make the rule stricter (increase the 15). If you reject an adult, you make it looser.

After checking many people and adjusting each time, your rule gets really good! That's exactly how a perceptron learns—it's a bouncer that adjusts its rules based on mistakes.

The XOR problem is like: "Let in people who are EITHER old OR have a ticket, BUT NOT BOTH." A single straight-line rule can't do this—you'd need two different rules or a curved rule.

Key Formulas Summary

Connections

• McCulloch-Pitts Neuron - The 1943 predecessor without learning
• Linear Separability - Mathematical condition for perceptron convergence
• Activation Functions - Step function vs modern smooth functions
• Gradient Descent - Modern learning algorithm that replaced perceptron rule
• Multi-Layer Perceptron - How stacking perceptrons solves XOR
• Support Vector Machines - Perceptrons with margin optimization
• Logistic Regression - Perceptron with sigmoid activation and probabilistic interpretation
• Backpropagation - The algorithm that revived neural networks after the AI winter

#flashcards/ai-ml

Concept Map

Hinglish (regional understanding)