Foundations — Activation functions - sigmoid, tanh

Before you can read the parent note (topic: Activation functions — sigmoid, tanh), you must own every symbol it throws at you. This page builds each one from absolute zero, in the order they depend on each other. Nothing is used before it is drawn.

1. A number line and the symbol $\mathbb{R}$

Plain words. $\mathbb{R}$ (read "the real numbers") is the name for every number you can place on an infinite ruler: $-3$, $0$, $0.25$, $\pi$, a million — positives, negatives, zero, and all the decimals between.

The picture. A horizontal line stretching forever in both directions. Every dot on it is one real number.

Why the topic needs it. A neuron's raw signal $z$ can be any value on that ruler. The whole job of an activation function is to take this unbounded ruler and fold it into a short segment like $(0,1)$. You cannot talk about "squashing all of $\mathbb{R}$ into $(0,1)$" until you can picture that endless ruler.

2. The variables $x$, $w$, $b$, $z$ — what a neuron actually computes

Plain words.

• $x$ = the input, the numbers coming into the neuron (e.g. pixel brightness, or the output of a previous neuron).
• $w$ = the weight, how strongly the neuron cares about each input. Big weight = "this input matters a lot".
• $b$ = the bias, a constant nudge added on, letting the neuron shift its decision left or right.
• $z$ = the pre-activation, the single raw number the neuron builds before squashing.

The picture. Imagine a mixing desk. Each input slider $x$ is scaled by a knob $w$, all the scaled signals are summed, then one master offset $b$ is added. The needle reading on the master meter is $z$.

Why the topic needs it. Everything on the parent page starts with "take the pre-activation $z$…". If you don't know $z$ came from $w\cdot x+b$, the worked examples (like $z=(1)(2)+(-2)(1)+0.5=0.5$) look like magic.

3. Vectors and the bold letters $w=[1,-2]$, $x=[2,1]$

Plain words. A vector is just an ordered list of numbers bundled together. $x=[2,1]$ means "the first input is $2$, the second is $1$". Writing them as a list is shorthand so we don't need a separate letter for every single number.

The picture. Two slots in a row, each holding a number. $w$ has the same number of slots as $x$, because every input needs its own weight.

Why the topic needs it. Real neurons have hundreds of inputs. Bundling them as vectors lets us write one clean symbol $w\cdot x$ instead of a hundred-term sum.

4. The dot product $w\cdot x$

Plain words. The dot product pairs up the vectors slot by slot, multiplies each pair, then adds all the products into one number.

$w\cdot x=w_1{x}_1+w_2{x}_2=(1)(2)+(-2)(1)=2-2=0$

Why this tool and not another? We need to combine "how much the neuron cares" ($w$) with "what actually arrived" ($x$) into a single score. Slot-by-slot multiply-then-add is exactly the operation that says "reward matching inputs, penalise mismatches" and collapses a whole list into one number — precisely what a neuron wants.

5. The exponential $e^{-z}$ and the number $e$

Plain words. $e\approx 2.718$ is a special constant. Raising it to a power, $e^k$, gives a smooth exponential curve: it grows explosively for positive $k$ and shrinks toward (but never reaches) zero for negative $k$.

The picture. A curve that hugs the floor on the left, passes through height $1$ at $k=0$ (since $e^0=1$), then rockets upward on the right. The version we use, $e^{-z}$, is its mirror image: huge on the left, tiny on the right.

Why this tool and not another? We want a squasher that is smooth and symmetric and turns "big positive $z$" into "output near the top" and "big negative $z$" into "output near the bottom". The function $e^{-z}$ does exactly this cross-over: at $z\to +\infty$ it $\to 0$, at $z\to -\infty$ it $\to \infty$. Sticking it into $\frac{1}{1+e^{-z}}$ turns that into a clean slide from $0$ to $1$. No polynomial gives you this bounded, smooth, symmetric behaviour for free.

6. Functions and the notation $\sigma (z)$, $\tanh (z)$

Plain words. $\sigma (z)$ is read "sigma of $z$" and just means "the machine named $\sigma$, fed the input $z$". The Greek letter $\sigma$ ("sigma") is the traditional name for the sigmoid machine. $\tanh (z)$ is another such machine.

The picture. A box with an input arrow (carrying $z$) on the left and an output arrow (carrying the squashed value) on the right.

Why the topic needs it. This notation lets us name the squasher once and reuse it. When the parent writes $a=\sigma (Wx+b)$, it means "compute $z=Wx+b$, then push it through the sigmoid box, and call the result $a$".

7. Slope, and the derivative ${\sigma }^{\prime }(z)$

Plain words. The slope of a curve at a point is how steeply it is climbing there — "if I take one tiny step right, how far up do I go?". The derivative, written ${\sigma }^{\prime }(z)$ (read "sigma-prime of $z$"), is a new machine that reports the slope of $\sigma$ at every point $z$.

The picture. Zoom in on the S-curve until it looks like a straight ramp. The steepness of that little ramp is the derivative. Steep in the middle, nearly flat at the two ends.

Why this tool and not another? Learning means "nudge the weights in the direction that reduces error". To know which direction, we need the slope of every function in the chain. The derivative is the exact tool that answers "which way, and how fast, is the output changing?" — nothing else measures that.

8. The chain rule (why derivatives multiply)

Plain words. If a value flows through machine A, then machine B, the overall slope is the slope of A times the slope of B. Slopes multiply along a chain.

The picture. A conveyor belt: input → box A → box B → output. A tiny wiggle at the input gets amplified by A's steepness, then again by B's steepness. Total amplification = product of the two.

Why the topic needs it. The parent's Example 2 does exactly $\frac{\partial L}{\partial z}=\frac{\partial L}{\partial a}\times {\sigma }^{\prime }(z)=0.8\times 0.235=0.188$. That multiplication is the chain rule. It is also why saturated (flat) neurons kill the gradient: multiplying by a near-zero slope.

9. Putting it together

Read it top to bottom: the ruler $\mathbb{R}$ and vectors build $z$; the exponential and interval build the sigmoid box; the box gives a slope; slopes chain together — and only then are you ready for the full topic.

Equipment checklist

Test yourself — cover the right side and answer aloud.

Connections

• Parent topic: Sigmoid & Tanh — where all these symbols get used.
• Logistic Regression — a single sigmoid neuron; same $z=w\cdot x+b$.
• Backpropagation — where the chain rule and ${\sigma }^{\prime }(z)$ do their work.
• Vanishing and Exploding Gradients — the flat-tail slope story made rigorous.
• Weight Initialization — keeps $z$ off the flat tails.