Bias terms and their role — Hinglish notes

3.1.13 · AI-ML › Neural Network Fundamentals

Bias term KYA hota hai?

$x$ = input vector, $w$ = weight vector (ek weight per input).
$b$ kisi bhi input ko multiply nahi karta — yeh ek constant offset hai.
Har layer ke har neuron ka apna khud ka bias hota hai.

Yeh kyun zaroori hai? (First principles se derivation)

Sabse simple neuron aur linear activation $ϕ (z) = z$ se shuru karte hain. Bias ke bina: $z = w x .$ Yeh origin se guzarne wali line hai. $x = 0$ par, hum $z = 0$ par stuck ho jaate hain. Hum sirf $w$ change karke line ko rotate kar sakte hain; hum use kabhi upar ya neeche nahi le ja sakte.

Ab maano ki true relationship $y = 2 x + 5$ hai. Chahe hum koi bhi $w$ choose karein, $z = w x$ kabhi constant $+ 5$ produce nahi kar sakta. Model data fit karne mein fundamentally incapable hai.

Bias add karo: $z = w x + b .$ Ab $b$ exactly intercept provide karta hai. $w = 2, b = 5$ set karne se perfect fit ho jaata hai.

Absorbed-bias trick (yeh actually sirf ek aur weight kyun hai)

Input mein ek constant $1$ append karo: $\tilde{x} = [x_{1}, \dots, x_{n}, 1]$ , aur $\tilde{w} = [w_{1}, \dots, w_{n}, b]$ lo. Tab $\tilde{w}^{⊤} \tilde{x} = \sum_{i} w_{i} x_{i} + b \cdot 1 = z .$ Toh bias mathematically ek dummy input par weight hai jo hamesha 1 hoti hai. Yahi wajah hai ki isse same gradient descent rule se train kiya jaata hai jaise kisi bhi weight ko.

Bias kaise seekha jaata hai? (Gradient derivation)

Maano loss $L$ output $a = ϕ (z)$ par depend karta hai, $z = w^{⊤} x + b$ . Chain rule se: $\frac{\partial L}{\partial b} = \frac{\partial L}{\partial a} \cdot \frac{\partial a}{\partial z} \cdot \frac{\partial z}{\partial b} .$ Ab key piece: $\frac{\partial z}{\partial b} = \frac{\partial}{\partial b} (\sum_{i} w_{i} x_{i} + b) = 1.$

Update rule (learning rate $η$ ): $b \leftarrow b - η δ .$

$m$ examples ke poore minibatch ke liye, gradients sum/average kiye jaate hain: $\frac{\partial L}{\partial b} = \frac{1}{m} \sum_{k = 1}^{m} δ^{(k)} .$

Worked Examples

Common Mistakes (Steel-manned)

Recall Checks

Recall Bias-free neuron

$y = 3$ (constant) represent kyun nahi kar sakta? Kyunki $z = w^{⊤} x$ tab zero hota hai jab input zero ho aur constant offset add nahi kar sakta — har hyperplane jo woh define karta hai origin se guzarti hai.

Recall

$\partial z / \partial b$ kya hai aur yeh kyun matter karta hai? Yeh $1$ ke equal hota hai, isliye bias gradient simply neuron ka error $δ$ hota hai, input magnitude se independent.

Recall Sigmoid neuron ke liye

$- b / w$ kya represent karta hai? Woh input value jis par neuron half-activated hota hai — uska decision threshold.

Recall Feynman: ek 12-year-old ko explain karo

Ek light switch imagine karo jo tab on hota hai jab tum use kaafi hard push karo. Weights decide karte hain ki har push kitna count karta hai. Bias decide karta hai ki switch ko flip karna kitna mushkil hai — ek "loose" switch (bada bias) ek tiny push se on ho jaata hai; ek "stiff" switch ko bada shove chahiye. Learning ka matlab sirf us switch ko tight ya loose karna hai jab tak light exactly sahi moments par on na ho jaaye.

80/20 — the vital few

$z = w^{⊤} x + b$ ; bias ek added learnable offset hai per neuron.
Iske bina, decision boundaries origin se guzarti hain — flexibility ka bada loss.
$\partial L / \partial b = δ$ (kyunki $\partial z / \partial b = 1$ ), plain gradient descent se train hota hai.
Bias threshold set karta hai ( $- b / w$ ); yeh activations ko left/right shift karta hai.
0 par initialize karo, usually isse regularize mat karo.

Connections

Perceptron and the step function
Weighted sum and linear combinations
Activation functions (sigmoid, ReLU, tanh)
Backpropagation and the chain rule
Weight initialization strategies
Regularization (L1, L2)
Affine transformations

Neuron mein bias term kya hota hai?

Ek learnable scalar jo weighted sum mein activation se pehle add hota hai:

z = w^{⊤} x + b

; ek per neuron.

Bias kyun zaroori hai?

Yeh neuron ko apna output/threshold shift karne deta hai taaki decision boundary origin se guzarne ki zaroorat na ho, nonzero intercepts enable karta hai.

\partial z / \partial b

kya hai?

Exactly

1

, kyunki

b

add hota hai aur kisi bhi input ko multiply nahi karta.

Bias gradient

\partial L / \partial b

kya hai?

Neuron ka error signal

δ = \partial L / \partial z

(input-independent), weight gradients

δ x_{i}

ke unlike.

Bias as a weight — trick kya hai?

Ek constant input

1

append karo; bias us dummy input par weight ban jaata hai.

Sigmoid neuron ke liye

- b / w

ka kya matlab hai?

Woh input value jahan neuron half-activated hota hai — uska threshold.

Biases usually kaise initialize kiye jaate hain?

0 par (ReLU ke liye kabhi kabhi small positive constant); biases ke liye symmetry-breaking ki zaroorat nahi.

Kya biases ko L2-regularize karna chahiye?

Usually nahi — woh negligible capacity add karte hain aur inhe penalize karna sirf model ki output means set karne ki ability ko nuksan pahunchata hai.

Weights vs bias roles?

Weights boundary ka slope/orientation control karte hain; bias uska offset/threshold control karta hai.

H

neurons ki layer ke liye, bias ka shape kya hota hai?

R^{H}

mein ek vector — ek bias per neuron.