Foundations — Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh)

Before you can read the parent note, you must be able to look at each symbol and hear what it says out loud. This page walks the entire alphabet of a neuron, from the humble subscript to the exponential $e^{-z}$, building each one on the one before it. Nothing here assumes you have seen algebra beyond "multiply and add".

1. A number that stands for something: the variable

Think of a cockpit gauge. The gauge labelled airspeed shows $150$ at one moment and $210$ a minute later. The label is $x$; the needle reading is the number inside. We use letters so we can talk about the reading without freezing it to one value.

The parent note uses $x$ for a sensor reading — airspeed, altitude, temperature. That is all $x$ ever means: one input number.

2. Many of the same thing: the subscript

Look at the figure. We have a row of gauges, one per input. Instead of inventing a fresh letter for each ($a$ for airspeed, $h$ for altitude, …) we reuse the single letter $x$ and change only the little label underneath. This is the only reason subscripts exist: to say "same kind of thing, different slot".

3. The counting shorthand: $i$, $n$, and $\sum$

Now the scary symbol. The parent note writes ${\sum }_{i=1}^n{w}_i{x}_i$.

The figure shows the loop unrolled. ${\sum }_{i=1}^3{w}_i{x}_i$ is literally the same as writing $w_1{x}_1+w_2{x}_3?\text{no —}{w}_1{x}_1+w_2{x}_2+w_3{x}_3.$ Same counter under both letters, so slot $1$ pairs with slot $1$. The $\sum$ is only there so we don't have to write a hundred + signs when there are a hundred inputs. Why the topic needs it: a real sensor bank has dozens of readings; the loop notation keeps the neuron formula one line long no matter how many.

4. Multiply-then-add together: the weighted sum $z$

Now we can read the heart of the neuron. Two words:

Picture a mixing desk (the figure). Each sensor feeds a slider $w_i$. Slide it up, that sensor's contribution to the final knob $z$ grows. The single output number $z$ is the "master volume" the neuron will react to. Why we need it: the whole point of learning is turning these dials — the network discovers, from data, which sensors matter for predicting pitch or spotting a fault.

5. The resting nudge: the bias $b$

Imagine a bathroom scale that reads $2\text{kg}$ with nothing on it. That constant $+2$ is a bias. In a neuron, if every input happens to be $0$, the weighted sum is $0$ too — without a bias the neuron would be forced to output "nothing" in that case. The bias gives it a resting value it can lean on, so it can fire even when inputs are quiet. Why the topic needs it: it lets each neuron pick where its decision line sits, not just how steep it is.

6. The bending machine: what a function $f(\cdot )$ is

The parent note's neuron is $y=f(z)$: take the master number $z$, run it through machine $f$, call the result $y$. The three machines the parent introduces — ReLU, sigmoid, tanh — are three different bending rules. To read them we need two last tools: the exponential and the idea of steepness.

7. Curves need this: the exponential $e^{-z}$

Sigmoid and tanh both contain $e^{-z}$. Let's earn it.

Follow the two curves in the figure. Why sigmoid uses $e^{-z}$: we want a machine whose output slides smoothly from $0$ to $1$. The trick $\frac{1}{1+e^{-z}}$ does exactly that:

• When $z$ is very negative, $e^{-z}$ is huge, so $\frac{1}{1+\text{huge}}\to 0$.
• When $z$ is very positive, $e^{-z}\to 0$, so $\frac{1}{1+0}\to 1$.
• At $z=0$, $e^0=1$, so $\frac{1}{1+1}=\frac{1}{2}$ — dead centre.

So $e^{-z}$ is the ingredient that gives sigmoid its smooth S-shape between $0$ and $1$. We use this tool and not a plain straight line precisely because we want a gentle, always-differentiable squash rather than a hard cut.

8. How steep is the curve here: derivative $f^{\prime }(z)$

The parent note keeps writing $f^{\prime }(z)$, ${\sigma }^{\prime }(z)$, ${\tanh }^{\prime }(z)$. Read the dash out loud as "prime".

Picture standing on the curve and asking "if I step a hair to the right, how much do I rise?" That rise-per-step is the derivative. Why the topic needs it: learning (see Backpropagation and Gradient Descent) works by nudging weights in the direction the slope points. If a machine is flat ($f^{\prime }\approx 0$) over a wide range — as sigmoid is for big $|z|$ — the nudge signal dies. That flat-region death is exactly the vanishing gradient the parent warns about.

9. Putting it together — the neuron in one breath

Read the parent's master formula now, symbol by symbol: $y=f\left({\sum }_{i=1}^n{w}_i{x}_i+b\right)$

• $x_i$ — the $i$-th input gauge (§1–3)
• $w_i$ — its importance dial (§4)
• ${\sum }_{i=1}^n$ — loop that multiply-adds all $n$ of them (§3–4)
• $+b$ — the resting nudge (§5)
• $f(\cdots )$ — the bending machine, one of ReLU / sigmoid / tanh (§6–7)
• $y$ — the single output number

That is the whole neuron. Everything else on the parent page is choosing which bending machine $f$ to use, and how steep ($f^{\prime }$) it is for learning.

Prerequisite map

Each foundation feeds the next; the two streams (the forward blend-and-bend, and the slope needed for learning) meet at the neuron, which is the doorway into the parent topic.

Equipment checklist

Test yourself — can you say each answer before revealing it?