Exercises — Activation functions - sigmoid, tanh

What we check: the range (set of possible outputs) of each function. Read it off Figure 1: the blue $\sigma$ curve stays entirely above the dashed line at $0$; the yellow $\tanh$ curve dips below $0$ on its left half.

• $\sigma (z)\in (0,1)$ — always positive, never reaches $0$ or $1$.
• $\tanh (z)\in (-1,1)$ — can be negative for $z<0$. Answer: only $\tanh$ can be negative. Range of sigmoid is $(0,1)$; range of tanh is $(-1,1)$. Why the open brackets? Because $e^{-z}$ is never exactly $0$ (an exponential is always strictly positive), so $\sigma$ can approach but never touch $1$ — and similarly for the other bounds.

At $z=0$: $e^{-0}=e^0=1$.

• $\sigma (0)=\frac{1}{1+1}=\frac{1}{2}=0.5$.
• $\tanh (0)=\frac{1-1}{1+1}=\frac{0}{2}=0$. Check against Figure 1: the blue dot marks $(0,0.5)$ and the yellow dot marks $(0,0)$ — exactly the two values we just computed. Answer: $\sigma (0)=0.5$, $\tanh (0)=0$. These are exactly the midpoints of their ranges — the "meh, undecided" point of each dimmer switch.

Plug in: $\sigma (2)=\frac{1}{1+e^{-2}}=\frac{1}{1+0.135335}=\frac{1}{1.135335}$. Do the division by hand. We want $1÷1.135335$. Multiply a trial value: $1.135335\times 0.88=0.99909$, a touch under $1$; $1.135335\times 0.881=1.00023$, a touch over. So the answer sits between $0.880$ and $0.881$. Interpolating, $1-0.99909=0.00091$ of the gap $0.00114$ per thousandth, giving about $0.0008$ more: $0.8808$. Why round to 4 places? The inputs $e^{-2}=0.135335$ were given to 6 significant figures; carrying more than 4 decimals in the output would claim precision the inputs do not support, so 4 decimals ($0.8808$) is the honest stopping point. Locate it on Figure 1: at $z=2$ the blue curve is already high, hugging the upper dashed line near $1$ — consistent with $0.88$. Answer: $\sigma (2)\approx 0.8808$. The neuron "fires" at about 88% confidence.

Step 1 — pre-activation (weighted sum): $z=(0.5)(2)+(-1)(1)+(2)(0.5)+(-0.5)=1-1+1-0.5=\mathrm{0.5.}$ Why? The dot product $w\cdot x$ multiplies each weight by its matching input and adds them, then the bias shifts the total. Step 2 — apply tanh: $\tanh (0.5)=\frac{e^{0.5}-e^{-0.5}}{e^{0.5}+e^{-0.5}}=\frac{1.64872-0.606531}{1.64872+0.606531}=\frac{1.042189}{2.255251}=\mathrm{0.4621.}$ Answer: $z=0.5$, $\tanh (0.5)\approx 0.4621$.

Why this route? The identity says tanh is a stretched, shifted sigmoid; checking it two ways guards against arithmetic slips. Step 1: $2z=1$, so $\sigma (1)=\frac{1}{1+e^{-1}}=\frac{1}{1.367879}=0.731059$. Step 2: $2\sigma (1)-1=2(0.731059)-1=1.462117-1=0.4621$. Answer: $0.4621$ — matches Q2. The identity holds.

Use the self-referential derivative: ${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))$. ${\sigma }^{\prime }(4)=0.982014\times (1-0.982014)=0.982014\times 0.017986=\mathrm{0.017663.}$ Form the ratio $\frac{{\sigma }^{\prime }(0)}{{\sigma }^{\prime }(4)}=\frac{0.25}{0.017663}\approx 14.2$. Answer: ${\sigma }^{\prime }(4)\approx 0.01766$; it is approximately 14 times smaller than ${\sigma }^{\prime }(0)$. Confirm this visually on Figure 2 (the derivative-hump figure before Level 5): the green ${\sigma }^{\prime }$ curve peaks at $0.25$ over $z=0$ and has nearly flattened to the axis by $z=4$. Out on that flat tail a nudge in $z$ barely moves the output, so backprop passes almost no signal — the vanishing-gradient effect from Vanishing and Exploding Gradients.

Chain rule (from the recap): the loss depends on $z$ through the activation $a=\sigma (z)$, so multiply the incoming slope by the local slope: $\frac{\partial L}{\partial z}=\underset{\text{incoming}}{\underbrace{\frac{\partial L}{\partial a}}}\cdot \underset{\text{local}}{\underbrace{{\sigma }^{\prime }(z)}}=\frac{\partial L}{\partial a}\cdot a(1-a).$ $=1.5\times 0.017663=\mathrm{0.026494.}$ Answer: $\frac{\partial L}{\partial z}\approx 0.02649$. Even a healthy upstream gradient of $1.5$ is choked down to $\approx 0.026$ by the tiny local slope — one saturated layer already loses ~98% of the signal. See Backpropagation.

Compute: $\frac{\partial L}{\partial w_1}=\delta \cdot a_1=(-0.3)(0.7)=-0.21,\frac{\partial L}{\partial w_2}=\delta \cdot a_2=(-0.3)(0.9)=-\mathrm{0.27.}$ Both negative. Because $a_1,a_2$ are forced positive (sigmoid range $(0,1)$ — the entirely-above-zero blue curve in Figure 1), the sign of every weight gradient equals the sign of the single shared $\delta$. So all weights are pushed the same direction each step → the "zig-zag" inefficiency. Answer: $-0.21$ and $-0.27$; same sign. This is exactly why zero-centered $\tanh$ (which can output negatives) trains hidden layers more smoothly.

First, why we may swap variable from $z$ to $s$. We claim $\sigma$ is strictly increasing, so each $z$ maps to exactly one $s$ and vice versa. Justification: ${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))$; since $\sigma (z)\in (0,1)$ (from L1-Q1), both factors $\sigma (z)>0$ and $1-\sigma (z)>0$, so their product ${\sigma }^{\prime }(z)>0$ for every real $z$. A function with a strictly positive slope everywhere is strictly increasing — hence a genuine one-to-one correspondence between $z$ and $s$. Maximizing the slope over $z$ is therefore the same as maximizing it over $s$. Reframe: the slope is $g(s)=s(1-s)=s-s^2$ where $s$ ranges over $(0,1)$. Maximize $g$: $\frac{dg}{ds}=1-2s=0\Rightarrow s=\frac{1}{2}$. Second derivative $-2<0$, so it's a maximum. Translate back: $s=\frac{1}{2}$ means $\sigma (z)=0.5$, which happens at $z=0$. There $g=\frac{1}{2}\cdot \frac{1}{2}=0.25$. See it: this is exactly the peak of the green ${\sigma }^{\prime }$ hump in Figure 2, sitting at $(0,0.25)$. Answer: max slope $=0.25$ at $z=0$. ∎

Differentiate the identity using the chain rule from the recap. The outer function multiplies by $2$; the inner function is $2z$, whose derivative is $2$ — so that inner $2$ gets pulled out front (this is the $g^{\prime }(z)$ factor): ${\tanh }^{\prime }(z)=\frac{d}{dz}[2\sigma (2z)-1]=2\cdot \underset{f^{\prime }(g)}{\underbrace{{\sigma }^{\prime }(2z)}}\cdot \underset{g^{\prime }}{\underbrace{2}}=4{\sigma }^{\prime }(2z).$ Evaluate at $z=0$: ${\sigma }^{\prime }(0)=0.25$, so ${\tanh }^{\prime }(0)=4\times 0.25=\mathrm{1.0.}$ See it: in Figure 2 the red ${\tanh }^{\prime }$ hump peaks at $1.0$, exactly four times higher than the green ${\sigma }^{\prime }$ peak at $0.25$. Answer: ${\tanh }^{\prime }(0)=1.0$ — and the "$4\times$" factor is now explained: it's precisely the two chain-rule factors of 2 multiplying ${\sigma }^{\prime }$.

Invert sigmoid. From $\sigma (z)=\frac{1}{1+e^{-z}}=0.8$: $1+e^{-z}=\frac{1}{0.8}=1.25\Rightarrow e^{-z}=0.25\Rightarrow -z=\ln (0.25)=-\ln 4\Rightarrow z^{\star }=\ln 4=\mathrm{1.386294.}$ Slope check: ${\sigma }^{\prime }(z^{\star })=0.8(1-0.8)=0.8\times 0.2=\mathrm{0.16.}$ Answer: $z^{\star }\approx 1.3863$, slope $=0.16$. The function $z=\ln \frac{s}{1-s}$ that undoes sigmoid is called the logit — the backbone of Logistic Regression.

Start from the definition: $\sigma (-z)=\frac{1}{1+e^z}.$ Multiply top and bottom by $e^{-z}$ (a legal move, $e^{-z}\ne 0$): $=\frac{e^{-z}}{e^{-z}+1}=\frac{(1+e^{-z})-1}{1+e^{-z}}=1-\frac{1}{1+e^{-z}}=1-\sigma (z).■$ Interpretation (look at Figure 1): reflect the input across $0$ and the output reflects across $0.5$ — the blue curve has point symmetry about the marked dot $(0,0.5)$, confirming the L1 trap that it is not symmetric about the origin.

From the definition, replace $z$ by $-z$: $\tanh (-z)=\frac{e^{-z}-e^{-(-z)}}{e^{-z}+e^{-(-z)}}=\frac{e^{-z}-e^z}{e^{-z}+e^z}.$ Factor $-1$ out of the numerator and note the denominator is unchanged (addition is commutative, so $e^{-z}+e^z=e^z+e^{-z}$): $=\frac{-(e^z-e^{-z})}{e^z+e^{-z}}=-\frac{e^z-e^{-z}}{e^z+e^{-z}}=-\tanh (z).■$ Connection (see the yellow curve in Figure 1): oddness means the curve has point symmetry about the origin, so for symmetric inputs the outputs are balanced positive/negative — their average sits at $0$. That is exactly the "zero-centered" property that avoids the same-sign gradient zig-zag of L3-Q3.

Symbolic proof. Differentiate the identity $\sigma (-z)=1-\sigma (z)$. On the left, the chain rule (recap) applies to $\sigma (-z)$: outer slope ${\sigma }^{\prime }(-z)$ times inner slope $\frac{d}{dz}(-z)=-1$, giving $-{\sigma }^{\prime }(-z)$. On the right, $\frac{d}{dz}[1-\sigma (z)]=-{\sigma }^{\prime }(z)$. Equate: $-{\sigma }^{\prime }(-z)=-{\sigma }^{\prime }(z)\Rightarrow {\sigma }^{\prime }(-z)={\sigma }^{\prime }(z).■$ So the slope is an even function — a bell-shaped hump centered at $z=0$, its peak $0.25$. Numeric check. At $z=1.386294$, $\sigma (z)=0.8$ so ${\sigma }^{\prime }(z)=0.8\times 0.2=0.16$. By symmetry ${\sigma }^{\prime }(-1.386294)$ should also be $0.16$: indeed $\sigma (-1.386294)=1-0.8=0.2$, giving ${\sigma }^{\prime }=0.2\times 0.8=0.16$. ✓ Answer: both equal $0.16$ — the two green square markers at $z=\pm 1.386$ in Figure 2 sit at the same height $0.16$, the picture of this result.