Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh): Exercises

Before we start, one reminder of the four objects we will reuse constantly — read this even if you think you know it, because every solution leans on it.

Figure — Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh)

Look at the blueprint above (each curve is named in the legend, not just coloured): ReLU is a flat floor then a $45°$ ramp. Sigmoid is a gentle S trapped between $0$ and $1$ . Tanh is a steeper S trapped between $- 1$ and $1$ . The arrows on the figure point out each curve by name. Keep this picture in your head — most L1/L2 answers are just "which curve am I standing on, and where?"

Level 1 — Recognition

Problem 1.1

Which activation function can output the value $- 0.7$ ? (Choose from ReLU, sigmoid, tanh.)

Recall Solution 1.1

WHAT we look at: the range (the set of possible outputs) of each function.

ReLU output range is $[0, \infty)$ — it can never be negative (its floor is $0$ ).
Sigmoid range is $(0, 1)$ — strictly positive, never reaches $0$ or $1$ .
Tanh range is $(- 1, 1)$ — the only one that dips below zero.

$- 0.7$ lies between $- 1$ and $0$ , so it is only reachable by tanh.

WHAT IT LOOKS LIKE: on the figure, only the tanh curve (labelled in the legend) ever goes below the horizontal axis.

Problem 1.2

A neuron in a hidden layer of a 12-layer network needs an activation whose derivative does not shrink signals for positive inputs. Name it and give its full derivative (including the negative branch and the convention at $z = 0$ ).

Recall Solution 1.2

ReLU. For $z > 0$ , $ReLU (z) = z$ , so its slope is exactly $1$ . For $z < 0$ , $ReLU (z) = 0$ , a flat line with slope $0$ . At exactly $z = 0$ the two branches meet at a corner where the derivative is not truly defined; by convention we simply assign it $0$ (grouping it with the $z \leq 0$ side). Writing the whole thing: $ReLU^{'} (z) = {10 z > 0 z \leq 0$ A slope of $1$ on the active branch means backpropagation multiplies by $1$ — the signal passes through undimmed, which is why ReLU is the default for deep hidden layers. Contrast: Vanishing and Exploding Gradients happens when slopes below $1$ multiply together across many layers.

Level 2 — Application

Problem 2.1

A neuron has weights $w = (0.4, - 0.6)$ , bias $b = 1.0$ , inputs $x = (2.0, 3.0)$ . Compute $z$ , then apply ReLU.

Recall Solution 2.1

Step 1 — weighted sum (WHAT & WHY): we multiply each input by its weight to score its contribution, then add the bias nudge. Here $n = 2$ . $z = (0.4) (2.0) + (- 0.6) (3.0) + 1.0 = 0.8 - 1.8 + 1.0 = 0.0$ Step 2 — activation: ReLU at exactly $0$ : $ReLU (0) = max (0, 0) = 0.$ Output $y = 0$ . Note the edge case: $z = 0$ sits on the corner of the ReLU curve. By our definition $f (z) = 0$ for $z \leq 0$ , so the door is just closed.

Problem 2.2

For $z = 2$ , compute $σ (2)$ to 4 decimals, then use the identity $σ^{'} (z) = σ (z) (1 - σ (z))$ to get the derivative there.

Recall Solution 2.2

Step 1: $σ (2) = \frac{1}{1 + e ^{- 2}} = \frac{1}{1 + 0.135335} = \frac{1}{1.135335} \approx 0.8808$ .

Step 2 (WHY this identity): we don't want to re-differentiate from scratch every time; the identity lets us reuse the value we just computed. $σ^{'} (2) = 0.8808 (1 - 0.8808) = 0.8808 \times 0.1192 \approx 0.1050.$ Reading it: a slope of about $0.105$ — already well under the maximum $0.25$ . The sigmoid is starting to flatten (saturate) at $z = 2$ .

Problem 2.3

For $z = - 1.2$ , compute $tanh (- 1.2)$ and then $tanh^{'} (- 1.2)$ using $tanh^{'} (z) = 1 - tanh^{2} (z)$ .

Recall Solution 2.3

Step 1: $tanh (- 1.2) = \frac{e ^{- 1.2} - e ^{1.2}}{e ^{- 1.2} + e ^{1.2}} = \frac{0.301194 - 3.320117}{0.301194 + 3.320117} = \frac{- 3.018922}{3.621311} \approx - 0.8337$ .

Step 2 (WHY reuse the value): same trick as sigmoid — square the output and subtract from $1$ . $tanh^{'} (- 1.2) = 1 - (- 0.8337)^{2} = 1 - 0.6951 \approx 0.3049.$ Reading it: tanh at $z = - 1.2$ has a healthier slope ( $\approx 0.30$ ) than sigmoid did at $z = 2$ — that is the "stronger gradient near the middle" advantage of tanh.

Level 3 — Analysis

Problem 3.1

A student proposes a "two-layer linear network" with no activation: $h = W_{1} x + b_{1}, y = W_{2} h + b_{2} .$ Show algebraically that this is equivalent to a single-layer linear model, and state in one sentence why that is a problem.

Recall Solution 3.1

Substitute $h$ into $y$ : $y = W_{2} (W_{1} x + b_{1}) + b_{2} = (W_{2} W_{1}) x + (W_{2} b_{1} + b_{2}) .$ Let $W_{eff} = W_{2} W_{1}$ and $b_{eff} = W_{2} b_{1} + b_{2}$ . Then $y = W_{eff} x + b_{eff},$ which is a single linear layer. The problem in one sentence: no matter how many linear layers you stack, they collapse to one straight-line map, so the network can only draw straight decision boundaries and can never learn a curved pattern like XOR. The cure is inserting a non-linear $f$ between layers.

Problem 3.2

The maximum slope of sigmoid is $0.25$ , occurring at $z = 0$ . Prove this maximum value using $σ^{'} (z) = σ (z) (1 - σ (z))$ .

Recall Solution 3.2

WHY this method: $σ^{'}$ is a product of two things that add to $1$ : let $p = σ (z)$ , so $σ^{'} = p (1 - p)$ with $p \in (0, 1)$ .

Maximize $p (1 - p)$ : treat it as a downward parabola in $p$ . Its derivative w.r.t. $p$ is $1 - 2 p$ , zero at $p = 0.5$ . So the peak is at $p = σ (z) = 0.5$ , which happens at $z = 0$ . $σ_{m a x}^{'} = 0.5 (1 - 0.5) = 0.25.$ Consequence: every sigmoid slope is $\leq 0.25$ . In a 10-layer stack the product of slopes is $\leq 0.2 5^{10} \approx 9.5 \times 1 0^{- 7}$ — this is exactly the vanishing gradient problem, and why gradient descent stalls.

Problem 3.3

Verify the identity $tanh (z) = 2 σ (2 z) - 1$ at $z = 0.5$ numerically (4 decimals), confirming both forms agree.

Recall Solution 3.3

Left side: $tanh (0.5) = \frac{e ^{0.5} - e ^{- 0.5}}{e ^{0.5} + e ^{- 0.5}} = \frac{1.648721 - 0.606531}{1.648721 + 0.606531} = \frac{1.042190}{2.255252} \approx 0.4621$ .

Right side: $σ (2 \cdot 0.5) = σ (1) = \frac{1}{1 + e ^{- 1}} = \frac{1}{1.367879} \approx 0.7311$ ; then $2 (0.7311) - 1 = 0.4621$ .

Both give $\approx 0.4621$ — the identity holds. This is why tanh is "a stretched, shifted sigmoid": scale the input by $2$ , scale the output by $2$ , and shift down by $1$ to recenter on zero.

Level 4 — Synthesis

Problem 4.1

A control-output neuron for elevator deflection has weights $w = (0.5, - 0.25)$ , bias $b = 0.1$ , inputs $x = (1.0, 4.0)$ . The output must lie in $[- 1, 1]$ where $- 1$ = full down, $+ 1$ = full up. (a) Choose the correct activation and justify. (b) Compute the deflection.

Recall Solution 4.1

(a) Choice — WHY tanh: the output must be signed and bounded to $[- 1, 1]$ . ReLU can't go negative (so no "down"); sigmoid is stuck in $(0, 1)$ (no "down" either). Only tanh covers $(- 1, 1)$ symmetrically around zero — perfect for a control surface that moves both ways from neutral.

(b) Compute (here $n = 2$ ): $z = (0.5) (1.0) + (- 0.25) (4.0) + 0.1 = 0.5 - 1.0 + 0.1 = - 0.4.$ $tanh (- 0.4) = \frac{e ^{- 0.4} - e ^{0.4}}{e ^{- 0.4} + e ^{0.4}} = \frac{0.670320 - 1.491825}{0.670320 + 1.491825} = \frac{- 0.821505}{2.162145} \approx - 0.3799.$ Reading it: elevator deflects about $38.0%$ of maximum downward (negative = down). Compare to Sensor Fusion in Aerospace, where such signed outputs feed the flight controller.

Problem 4.2

An anomaly detector's output neuron gives $z = 3.5$ . (a) Which activation makes the output a probability, and why? (b) Compute that probability, and (c) using Binary Cross-Entropy Loss $L = - ln (p)$ for a truly-anomalous sample (label $= 1$ ), compute the loss.

Recall Solution 4.2

(a) We need output in $[0, 1]$ readable as "chance of anomaly," so sigmoid. ReLU could output $3.5$ (not a probability); tanh could go negative (a negative "chance" is nonsense).

(b) $σ (3.5) = \frac{1}{1 + e ^{- 3.5}} = \frac{1}{1 + 0.030197} = \frac{1}{1.030197} \approx 0.9707$ . → about $97%$ confident it is anomalous.

(c) For a true anomaly (label $1$ ), binary cross-entropy is $L = - ln (p) = - ln (0.9707) \approx 0.02975$ . Reading it: the loss is tiny because the network was confident and correct — exactly what we want. If it had predicted $0.03$ instead, the loss would balloon to $- ln (0.03) \approx 3.51$ .

Level 5 — Mastery

Problem 5.1 (Dead ReLU + a fix)

A neuron uses ReLU. Over the whole dataset its pre-activation is always $z \leq 0$ , so $y = 0$ every time. (a) Explain why gradient descent can never revive it (a "dead neuron"). (b) Replace ReLU with Leaky ReLU $f (z) = max (0.01 z, z)$ and compute $f (z)$ and $f^{'} (z)$ at $z = - 5$ . (c) Explain in one sentence why this rescues the neuron.

Recall Solution 5.1

(a) ReLU's slope for $z \leq 0$ is $0$ . During backprop, the weight update is proportional to that slope, so an update of $0$ means the weights never change — the neuron is frozen at "off" forever. See also Weight Initialization Strategies, since bad init is a common cause.

(b) At $z = - 5$ (negative branch, Leaky ReLU uses $0.01 z$ ): $f (- 5) = max (0.01 \cdot (- 5), - 5) = max (- 0.05, - 5) = - 0.05.$ $f^{'} (- 5) = 0.01 (slope of the leaky branch) .$ (c) Because the slope is now $0.01$ instead of $0$ , a small non-zero gradient flows back, so the weights can still be nudged and the neuron may recover.

Problem 5.2 (Vanishing-gradient budget)

You stack $L$ sigmoid layers, and the best-case product of derivatives is $0.2 5^{L}$ . Find the largest $L$ for which this product stays above $1 0^{- 3}$ (so learning is not yet hopeless). Solve with logarithms.

Recall Solution 5.2

WHY logs: we need to unstack an exponent, and the logarithm is the tool that turns "power of" into "multiply by." $0.2 5^{L} > 1 0^{- 3} ⟹ L ln (0.25) > ln (1 0^{- 3}) .$ Now $ln (0.25) = - 1.386294$ (negative), so dividing flips the inequality: $L < \frac{l n ( 1 0 ^{- 3} )}{l n ( 0.25 )} = \frac{- 6.907755}{- 1.386294} \approx 4.982.$ Since $L$ must be a whole number, the largest allowed is $L = 4$ . Reading it: even in the best case, only about four stacked sigmoids keep gradients above $1 0^{- 3}$ — hard proof that deep sigmoid stacks are doomed, and why we reach for ReLU or the techniques in Vanishing and Exploding Gradients. This same reasoning appears when designing Convolutional Neural Networks and controlling Overfitting and Regularization.

Recall Self-test checklist

Range of ReLU / sigmoid / tanh? ::: $[0, \infty)$ / $(0, 1)$ / $(- 1, 1)$ . Why does a stack of linear layers collapse? ::: Composing linear maps gives another linear map $W_{eff} x + b_{eff}$ . Maximum sigmoid slope and where? ::: $0.25$ , occurring at $z = 0$ (where $σ = 0.5$ ). Which activation for a signed, bounded control output? ::: tanh. Which for a probability output? ::: sigmoid. How does Leaky ReLU revive a dead neuron? ::: It gives a small non-zero slope ( $0.01$ ) for $z < 0$ so gradients still flow.