Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh)

The Artificial Neuron

WHY each component?

Weights $w_i$ : Learning means adjusting these. High weight = "this input matters a lot."
Bias $b$ : Without it, output is zero when all inputs are zero. Bias lets the neuron activate even with no input—think of it as the neuron's "resting potential."
Activation $f$ : Converts the linear combo into a non-linear output. This is crucial—multiple linear functions compose to just another linear function, which can't learn XOR or any complex pattern.

Figure — Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh)

Formula Derivation: Why We Need Non-Linearity

Suppose we stack two linear layers without activation: $\text{Layer 1: } h = W_1 x + b_1$ $\text{Layer 2: } y = W_2 h + b_2 = W_2(W_1 x + b_1) + b_2$ $= (W_2 W_1)x + (W_2 b_1 + b_2) = W_{\text{eff}} x + b_{\text{eff}}$ This colapses to a single linear transformation. No matter how many layers, it's just $Ax + b$ . Can only learn linear decision boundaries (straight lines/planes).

The fix: Insert $f(z)$ between layers: $y = W_2 \cdot f(W_1 x + b_1) + b_2$ Now the composition is non-linear. The network can learn curves, circles, XOR gates, image features, flight envelopes—any non-linear pattern.

Activation Functions

1. ReLU (Rectified Linear Unit)

WHY ReLU?

Computationally cheap: Just a max operation, no exponentials.
Solves vanishing gradient: Gradient is1 for $z > 0$ , so backprop signals don't shrink.
Sparse activation: Negative inputs output zero—network learns to "switch off" irrelevant neurons.

WHEN to use: Default choice for hidden layers in deep networks (image recognition, control systems).

Worked example Worked Example: ReLU in Flight Control

Imagine a neural network predicting aircraft pitch angle from sensor inputs. One neuron computes: $z = 0.8 \cdot (\text{airspeed}) - 0.5 \cdot (\text{altitude}) + 2.0$ With airspeed = 150 m/s, altitude = 1000 m: $z = 0.8(150) - 0.5(1000) + 2.0 = 120 - 500 + 2 = -378$

Apply ReLU: $y = \max(0, -378) = 0$ WHY this step? The neuron decided this feature combo is not relevant for the current flight regime (negative signal). ReLU "turns it off," preventing it from interfering. If $z$ were positive, it would pass through unchanged, contributing proportionally.

2. Sigmoid

Definition Sigmoid

$\sigma(z) = \frac{1}{1 + e^{-z}}$ Derivative (derive it yourself!): $\sigma'(z) = \sigma(z)(1 - \sigma(z))$

Derivation: $\frac{d}{dz}\left(\frac{1}{1+e^{-z}}\right) = \frac{-1 \cdot (1+e^{-z})' }{(1+e^{-z})^2} = \frac{e^{-z}}{(1+e^{-z})^2}$ $= \frac{1}{1+e^{-z}} \cdot \frac{e^{-z}}{1+e^{-z}} = \sigma(z) \cdot \frac{e^{-z}}{1+e^{-z}}$ Note $\frac{e^{-z}}{1+e^{-z}} = 1 - \frac{1}{1+e^{-z}} = 1 - \sigma(z)$ . $\implies \sigma'(z) = \sigma(z)(1 - \sigma(z))$

Properties:

Range: $(0, 1)$ — naturally interprets as probability.
Smooth: Differentiable everywhere.
Saturates: For $|z| > 5$ , gradient $\approx 0$ (vanishing gradient problem).

WHEN to use: Binary classification output layer (e.g., "Is this sensor reading anomalous?").

3. Tanh (Hyperbolic Tangent)

Properties:

Range: $(-1, 1)$ — zero-centered (unlike sigmoid).
Stronger gradients near zero than sigmoid.
Still saturates at extremes.

WHEN to use: Hidden layers when you want zero-centered activations (helps with convergence), or outputs representing signed quantities (e.g., control surface deflection: -1 = full down, +1 = full up).

Comparison Table

Function	Range	Zero-Centered?	Gradient Issues	Best For
ReLU	$[0, \infty)$	No	Dead neurons (if $z < 0$ always)	Hidden layers, deep networks
Sigmoid	$(0, 1)$	No	Vanishing gradient	Binary output layer
Tanh	$(-1, 1)$	Yes	Vanishing gradient	Hidden layers (better than sigmoid), signed outputs

Common mistake Common Error: Using Sigmoid in Hidden Layers of Deep Networks

The mistake: "Sigmoid is smooth and bounded, so it's safe everywhere." Why it feels right: Early neural networks (1990s) used sigmoid everywhere. It's mathematically elegant.

The fix: In a10-layer network, gradients multiply: $\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial y} \cdot \prod_{i=1}^{10} \sigma'(z_i) \cdot \ldots$ Since $\sigma'(z) \leq 0.25$ always (maximum at $z=0$ ), we get $\leq 0.25^{10} \approx 10^{-6}$ —the gradient vanishes. Weight updates become negligible, learning stops.

ReLU fixes this: $f'(z) = 1$ for active neurons, so gradients don't shrink. Use ReLU in hidden layers, sigmoid/tanh only at output if needed.

Recall Explain to a 12-Year-Old

Imagine you're building a robot pilot. It needs to decide: "Should I pull the control stick up or push it down?"

The robot's brain is made of tiny decision-makers called neurons. Each neuron looks at sensor readings (airspeed, altitude), multiplies each by a number called a weight (like saying "airspeed is 3× more important than altitude"), adds them up, then adds a bias (a starting nudge).

But here's the trick: if we just add and multiply, the robot can only learn straight-line rules. Real flying is curvy and complex! So we use an activation function—a magic math rule that bends the straight line into curves.

ReLU is like a door: negative numbers become zero (door closed), positive numbers stay (door open). Fast and simple.
Sigmoid squishes everything between 0 and 1, like a confidence meter: "I'm 90% sure this is an emergency."
Tanh squishes between -1 and +1, perfect for "push stick forward (-1) or pull back (+1)."

Without these bending functions, the robot brain would be dumb—just fancy addition. With them, it learns to fly through storms, dodge obstacles, anything!

Connections

Backpropagation and Gradient Descent — activation derivatives are crucial for computing gradients
Vanishing and Exploding Gradients — why sigmoid/tanh are problematic in deep networks
Weight Initialization Strategies — He initialization for ReLU, Xavier for sigmoid/tanh
Binary Cross-Entropy Loss — pairs naturally with sigmoid output
Convolutional Neural Networks — use ReLU almost exclusively in hidden layers
Overfitting and Regularization — dropout and L2 regularization interact with activation choices
Sensor Fusion in Aerospace — neural networks combine IMU, GPS, radar data; activation choice affects robustness

#flashcards/coding

What are the three components of an artificial neuron's computation? :: Weighted sum of inputs ( $\sum w_i x_i$ ), bias ( $b$ ), and activation function ( $f$ ).

Why do neural networks need activation functions?

Without non-linearity, stacking layers colapses to a single linear transformation (

y = Ax + b

), incapable of learning complex patterns like XOR or curved decision boundaries.

What is the formula for ReLU?

f(z) = \max(0, z)

. Output is

z

if positive,

0

if negative.

What is the derivative of ReLU?

f'(z) = 1

z > 0

, else

0

. (Undefined at

z=0

, but treated as 0 or 1 in practice.)

What problem does ReLU solve compared to sigmoid?

Vanishing gradient problem. ReLU's gradient is 1 for positive inputs, preventing gradient shrinkage in deep networks.

What is the sigmoid function formula?

\sigma(z) = \frac{1}{1 + e^{-z}}

What is the range of sigmoid?

(0, 1)

—naturally interprets as probability.

Derive the derivative of sigmoid.

\sigma'(z) = \sigma(z)(1 - \sigma(z))

. Proof: apply chain rule to

\frac{1}{1+e^{-z}}

, factor into

\sigma(z) \cdot (1 - \sigma(z))

What is the tanh function formula?

\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}

or equivalently

2\sigma(2z) - 1

What is the range of tanh?

(-1, 1)

—zero-centered, useful for signed outputs.

What is the derivative of tanh? :: $\tanh'(z) = 1 - \tanh^2(z)$ .

When should you use ReLU?

Default for hidden layers in deep networks (fast, avoids vanishing gradient).

When should you use sigmoid?

Binary classification output layer where you need probabilities in

[0, 1]

When should you use tanh?

Hidden layers for zero-centered activations, or outputs representing signed quantities (e.g., control deflections).

What is the "dead ReLU" problem?

A neuron's weighted sum is always negative, so output is always 0. Gradient is 0, weights never update—neuron is permanently inactive.

How do you fix dead ReLU neurons?

Use Leaky ReLU (

f(z) = \max(0.01z, z)

), proper weight initialization (He), or lower learning rate.

Why is sigmoid bad for hidden layers in deep networks?

Vanishing gradient:

\sigma'(z) \leq 0.25

, so gradients multiply to near-zero across many layers, stopping learning.

What is the role of bias

b

in a neuron?

Shifts the activation threshold, allowing the neuron to activate even when inputs are zero.

How does ReLU introduce non-linearity if it looks linear for

z > 0

The "kink" at

z=0

is the non-linearity. The function is not differentiable everywhere, breaking the linear composition property.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Neural network ki basic building block hai neuron. Isko samjho biological brain cell ki tarah—inputs lete hai, unhe weights se multiply karte hai (importance decide karne ke liye), sab add karte hai, fir ek bias add karte hai (baseline adjustment), aur finally ek activation function se pass karte hai jo non-linearity lata hai. Ye non-linearity bahut zaroori hai, warna kitne bhi layers stack karo, bas ek fancy linear equation banega—complex patterns nahi seekh payega.

Aerospace mein neural networks ka use hota hai flight control, trajectory prediction, sensor fusion, aur anomaly detection mein. Teen main activation functions hain: ReLU (Rectified Linear Unit), jo negative values ko zero kar deta hai aur positive ko waise hi chodta hai—fast aur vanishing gradient problem solve karta hai, isliye deep networks mein default choice hai. Sigmoid output ko 0 se 1 ke bech squash karta hai, perfect for binary classification (jaise "sensor faulty hai ya nahi?"). Tanh -1 se +1 range deta hai, zero-centered, acha for control outputs jahan positive aur negative dono directions chahiye (jaise elevator up/down deflection).

Common mistake: sigmoid ko hidden layers mein use karna deep networks mein—gradient vanish ho jata hai because sigmoid ka derivative maximum 0.25 hai, toh 10 layers mein multiply karo toh gradient practically zero ho jaata hai. ReLU iska solution hai kyunki uska gradient 1 hai active neurons ke liye.Ek aur problem hai dead ReLU: agar neuron ka weighted sum hamesha negative ho, toh output hamesha 0, gradient0, weights kabhi update nahi—neuron permanently inactive. Fix: Leaky ReLU use karo ya proper initialization (He initialization) karo.

Activation functions neural networks ko power dete hain complex, non-linear patterns seekhne ki—jaise curved flight envelopes, image features, ya XOR logic gates. Bina inke, network sirf straight lines draw kar sakta hai, aur real-world problems rarely linear hote hain!

Neural network fundamentals — neuron, activation functions (ReLU, sigmoid, tanh)

The Artificial Neuron

Activation Functions

1. ReLU (Rectified Linear Unit)

2. Sigmoid

3. Tanh (Hyperbolic Tangent)

Comparison Table

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Machine Learning (Aerospace Applications)

Connections