Implementing models from scratch — Hinglish notes

6.5.2 · AI-ML › Research Frontiers & Practice

Core Motivation

Jab tum torch.nn.Linear import karte ho, tum ek computational abstraction use kar rahe ho jo chhupata hai:

Gradients actually kaise propagate karte hain (backprop mechanics)
Memory layout aur computation order
Numerical stability ke trade-offs
Kyun kuch design choices exist karti hain

Scratch se build karna tumhe force karta hai jawab dene ke liye: Ye layer actually kya compute kar rahi hai, aur ye update rule kyun kaam karta hai?

The Three Pillars of From-Scratch Implementation

1. Forward Pass: Outputs Compute Karna

Ek Simple Dense Layer ke liye Derivation:

Fundamental question se shuru karo: Hum ek input vector ko output space mein kaise transform karein?

Linear transformation (sabse simple learnable operation): $z = Wx + b$

Ye form kyun?

$W$ (weight matrix): Input space ko rotate aur scale karta hai
$b$ (bias vector): Decision boundary ko translate karta hai
Ye sabse general affine transformation hai

Input $x \in R^{n}$ ke liye, output $z \in R^{m}$ :

$W \in R^{m \times n}$ (har row ek alag linear function hai)
$b \in R^{m}$

Phir activation function $σ$ apply karo: $a = σ (z)$

Activation kyun? Iske bina, layers stack karna sirf ek aur linear function deta hai: $W_{2} (W_{1} x + b_{1}) + b_{2} = W^{'} x + b^{'}$ . Activation non-linearity inject karta hai, universal function approximation enable karta hai.

Worked example Forward Pass Implementation

import numpy as np
class DenseLayer:
   def __init__(self, input_dim, output_dim):
       # Xavier initialization: scale by 1/sqrt(n)
       # WHY? Keeps variance stable across layers
       self.W = np.random.randn(output_dim, input_dim) / np.sqrt(input_dim)
       self.b = np.zeros((output_dim, 1))
       
   def forward(self, X):
       # X shape: (input_dim, batch_size)
       self.X = X  # Cache for backward pass
       self.Z = self.W @ X + self.b  # @ is matrix multiply
       return self.Z
 
# Why cache X? We'll need it for gradients!

3 samples ke batch ke liye step-by-step:

Input: $X \in R^{4 \times 3}$ (4 features, 3 samples)
Layer: 4 → 5 dimensions
$W \in R^{5 \times 4}$ , $b \in R^{5 \times 1}$
Output: $Z = WX + b \in R^{5 \times 3}$
Broadcasting $b$ ko har column mein add karta hai

2. Backward Pass: Gradients Compute Karna

Chain Rule se Derivation:

Loss $L$ diya hai, hum chahte hain $\frac{\partial L}{\partial W}$ aur $\frac{\partial L}{\partial b}$ .

Maano hum $\frac{\partial L}{\partial Z}$ agle layer se receive karte hain (ise $δ$ kaho).

Weights ke liye, chain rule use karte hue: $\frac{\partial L}{\partial W} = \frac{\partial L}{\partial Z} \frac{\partial Z}{\partial W}$

$\frac{\partial Z}{\partial W}$ compute karna:

$Z = WX + b$ se, element $Z_{i} = \sum_{j} W_{ij} X_{j} + b_{i}$ ke liye:

$\frac{\partial Z _{i}}{\partial W _{ij}} = X_{j}$

Matrix form mein: $\frac{\partial L}{\partial W} = δ X^{T}$

Transpose kyun? Dimensionality check:

$δ$ : $(m \times n_{ba t c h})$
$X^{T}$ : $(n_{ba t c h} \times n)$
Result: $(m \times n)$ ✓ ( $W$ ke same shape mein)

Bias ke liye: $\frac{\partial L}{\partial b} = δ 1$ jahan $1$ ones ka vector hai (batch dimension ke across sum karta hai).

Sum kyun? Bias saare samples mein shared hai—hum unke individual gradient contributions accumulate karte hain.

Previous layer ko gradient pass karne ke liye: $\frac{\partial L}{\partial X} = W^{T} δ$

Yahan transpose kyun? Hum forward multiplication ko "undo" kar rahe hain— $W$ ne $X \to Z$ transform kiya tha, toh $W^{T}$ gradients ko backwards $Z \to X$ transform karta hai.

Worked example Backward Pass Implementation

class DenseLayer:
   def backward(self, dL_dZ):
       # dL_dZ shape: (output_dim, batch_size)
       batch_size = dL_dZ.shape[1]
       
       # Gradient w.r.t. weights
       # WHY self.XT? Matches dimensions and chain rule
       self.dW = (dL_dZ @ self.X.T) / batch_size
       # Gradient w.r.t. bias
       # WHY sum and keepdims? Bias is per-neuron, shared across batch
       self.db = np.sum(dL_dZ, axis=1, keepdims=True) / batch_size
       
       # Gradient w.r.t. input (pass to previous layer)
       # WHY W.T? Reverse the forward transformation
       dL_dX = self.W.T @ dL_dZ
       
       return dL_dX

Worked Example: ReLU Activation Gradient Forward: $a = ReLU (z) = max (0, z)$ Derivative: $\frac{\partial a}{\partial z} = {10 if z > 0 if z \leq 0$

class ReLU:
   def forward(self, Z):
       self.Z = Z  # Cache for backward
       return np.maximum(0, Z)
   def backward(self, dL_dA):
       # Element-wise: gradient flows only where input was positive
       # WHY? ReLU "gates" the gradient—negative inputs had no effect
       return dL_dA * (self.Z > 0)

Agar $z = - 2.5$ , output $0$ hai, gradient $0$ hai (dead neuron). Agar $z = 3.7$ , output $3.7$ hai, gradient unchanged pass hota hai.

3. Parameter Update: Learning

Ye kaam kyun karta hai? $θ_{t}$ ke around Taylor expansion:

$L (θ_{t} - α \nabla L) \approx L (θ_{t}) - α ∥\nabla L ∥^{2}$

Doosra term negative hai, toh loss decrease hota hai (itne chote $α$ ke liye).

Stochastic Gradient Descent (SGD) Derivation:

Dataset $D$ par true gradient: $\nabla_{θ} L = \frac{1}{∣ D ∣} \sum_{(x, y) \in D} \nabla_{θ} L (x, y)$

Problem: Isko compute karne ke liye poore dataset ka full pass chahiye (expensive!).

Solution: Ek mini-batch $B \subset D$ use karke estimate karo: $\nabla_{θ} L \approx \frac{1}{∣ B ∣} \sum_{(x, y) \in B} \nabla_{θ} L (x, y)$

Ye kyun kaam karta hai: Law of large numbers ke by, sample average true expectation mein converge karta hai. Chote batches bhi (32-256) achhe estimates dete hain.

Worked example SGD with Momentum

Plain SGD noisy hota hai. Momentum past gradients ka exponential moving average add karta hai:

class SGDMomentum:
   def __init__(self, learning_rate=0.01, momentum=0.9):
       self.lr = learning_rate
       self.beta = momentum
       self.velocity = {}  # Per-parameter velocity
   def update(self, layer):
       # Initialize velocity for this layer if first update
       if id(layer) not in self.velocity:
           self.velocity[id(layer)] = {
               'W': np.zeros_like(layer.W),
               'b': np.zeros_like(layer.b)
           }
       
       v = self.velocity[id(layer)]
       
       # Momentum update: accumulate past direction
       # WHY beta? Smooths noisy gradients, accelerates in consistent directions
       v['W'] = self.beta * v['W'] + (1 - self.beta) * layer.dW
       v['b'] = self.beta * v['b'] + (1 - self.beta) * layer.db
       
       # Update parameters
       layer.W -= self.lr * v['W']
       layer.b -= self.lr * v['b']

Momentum kyun help karta hai?

Consistent gradient direction → velocity accumulate hoti hai → faster progress
Oscillating gradients → velocity cancel out ho jaati hai → smoother path
Mathematically: aise kaam karta hai jaise ek ball inertia ke saath hill se neeche roll kar rahi ho

Common Pitfalls

Common mistake Mistake 2: Backprop mein Shape Mismatches

Galat approach: Tab tak transpose karo jab tak shapes align ho jayein, bina samjhe kyun.

Kyun sahi lagta hai: "Matrix dimensions match karne chahiye, toh different combinations try karta hun."

Problem: Tumhe aisa shape mil sakta hai jo compile ho jaaye lekin galat gradient compute kare. Example: dL_dW = X @ dL_dZ.T ka shape sahi hai lekin semantics galat hain.

Rule: Gradient ka shape parameter ke shape se match karna chahiye. Chain rule se kaam karo:

Agar forward $Y = W X$ tha, backward hai $\frac{\partial L}{\partial W} = \frac{\partial L}{\partial Y} X^{T}$
Pehle symbolically derive karo, phir implement karo Fix: Computation graph dimensions ke saath draw karo. Verify karo:

assert dW.shape == W.shape, f"Gradient shape {dW.shape} != weight shape {W.shape}"

Common mistake Mistake 3: Activation Gradients Bhool Jana

Galat approach:

delta = prev_layer_delta  # Pass gradient through as-is

Kyun sahi lagta hai: "Gradient network ke through backwards flow karta hai."

Problem: Activations computation graph ka part hain! Unke derivatives matter karte hain.

$a = σ (z)$ ke liye, gradient hai: $\frac{\partial L}{\partial z} = \frac{\partial L}{\partial a} \cdot \frac{\partial a}{\partial z} = \frac{\partial L}{\partial a} \cdot σ^{'} (z)$

Example jahan ye fail karta hai: Sigmoid saturation. Agar $z = 10$ , toh $σ (10) \approx 1$ aur $σ^{'} (10) \approx 0$ . Gradient vanish ho jaata hai—tumhe iska account karna hi hoga!

Fix: Hamesha activation derivative se multiply karo:

delta = prev_delta * activation.derivative(self.Z)

Complete End-to-End Example

Worked example Scratch se 2-Layer Network Train Karna

Problem: Binary classification, 2D input, XOR function seekhna.

import numpy as np
# Data: XOR problem (not linearly separable!)
X_train = np.array([[0, 0], [0, 1], [1, 0], [1, 1]).T  # Shape: (2, 4)
y_train = np.array([[0, 1, 0]])  # Shape: (1, 4)
 
# Architecture: 2 -> 4 -> 1 with ReLU and Sigmoid
layer1 = DenseLayer(2, 4)
relu = ReLU()
layer2 = DenseLayer(4, 1)
def sigmoid(z):
   return 1 / (1 + np.exp(-z))
 
def sigmoid_derivative(a):
   return a * (1 - a)
# Binary cross-entropy loss
def compute_loss(ypred, y_true):
   # WHY clip? Avoid log(0) which is -inf
   y_pred = np.clip(y_pred, 1e-7, 1 - 1e-7)
   return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))
optimizer = SGDMomentum(learning_rate=0.1, momentum=0.9)
 
# Training loop
for epoch in range(1000):
   # === FORWARD PASS ===
   z1 = layer1.forward(X_train)
   a1 = relu.forward(z1)
   z2 = layer2.forward(a1)
   y_pred = sigmoid(z2)
   # Compute loss
   loss = compute_loss(y_pred, y_train)
   
   # === BACKWARD PASS ===
   # Start with loss gradient
   # WHY this form? Derivative of binary cross-entropy w.r.t. output
   dL_dy = (y_pred - y_train) / (y_pred * (1 - y_pred) + 1e-7)
   
   # Backprop through sigmoid
   dL_dz2 = dL_dy * sigmoid_derivative(y_pred)
   # Backprop through layer 2
   dL_da1 = layer2.backward(dL_dz2)
   
   # Backprop through ReLU
   dL_dz1 = relu.backward(dL_da1)
   
   # Backprop through layer 1
   dL_dx = layer1.backward(dL_dz1)
   # === UPDATE ===
   optimizer.update(layer1)
   optimizer.update(layer2)
   
   if epoch % 1000 == 0:
       print(f"Epoch {epoch}, Loss: {loss:.4f}")
# Test
predictions = (y_pred > 0.5).astype(int)
print(f"Predictions: {predictions}")
print(f"True labels: {y_train}")

Ye example kyun?

XOR linearly separable nahi hai—hidden layer chahiye (depth ki zarurat demonstrate karta hai)
Itna chota hai ki haath se verify ho sake
Saare components dikhata hai: forward, backward, update, loss computation Ek sample $[1, 0]$ ke liye step-by-step:
Forward: $z^{(1)} = W^{(1)} [1, 0]^{T} + b^{(1)}$ (4 values)
ReLU: $a^{(1)} = max (0, z^{(1)})$ (kuch neurons activate hote hain)
Forward: $z^{(2)} = W^{(2)} a^{(1)} + b^{(2)}$ (single value)
Sigmoid: $\overset{y}{^} = σ (z^{(2)})$ (probability)
Loss: $L = - [1 \cdot lo g (\overset{y}{^}) + 0 \cdot lo g (1 - \overset{y}{^})] = - l o g (\overset{y}{^})$
Backward: $\frac{\partial L}{\partial W ^{(2)}}$ , $\frac{\partial L}{\partial W ^{(1)}}$ , etc. compute karo
Update: $W \leftarrow W - α \nabla_{W} L$

Key Implementation Patterns

Initialization Strategy

Kyun matter karta hai: Buri initialization → vanishing/exploding gradients → network train nahi hoga.

Xavier/Glorot Initialization: $W_{ij} \sim N (0, \frac{1}{n _{in}})$

Derivation: $n$ inputs wale linear layer ke liye, agar variance 1 ho: $Var (z) = Var (\sum_{i} W i x_{i}) = \sum_{i} Var (W_{i}) Var (x_{i}) = n \cdot Var (W)$

$Var (z) = 1$ rakhne ke liye, humein $Var (W) = 1/ n$ chahiye.

He Initialization (ReLU ke liye): $W_{ij} \sim N (0, \frac{2}{n _{in}})$

Alag kyun? ReLU aadhe neurons ko kill karta hai, isliye compensate karne ke liye 2× variance chahiye.

Numerical Stability Tricks

# BAD: Direct softmax (can overflow)
def softmax_bad(z):
    return np.exp(z) / np.sum(np.exp(z))
 
# GOOD: Subtract max for stability
def softmax_stable(z):
    # WHY subtract max? exp(z - max) is bounded by exp(0) = 1
    # This prevents overflow while keeping ratios identical
    exp_z = np.exp(z - np.max(z, axis=0, keepdims=True))
    return exp_z / np.sum(exp_z, axis=0, keepdims=True)

Ye kyun kaam karta hai: $\frac{e ^{z_{i}}}{\sum _{j} e ^{z_{j}}} = \frac{e ^{z_{i} - c}}{\sum _{j} e ^{z_{j} - c}}$ kisi bhi constant $c$ ke liye. $c = max (z)$ choose karna overflow rokta hai.

Recall Ek 12-Saal ke Bacche ko Samjhao

Socho tum ek robot bana rahe ho jo cats pehchanna seekhta hai. Jab koi tumhe store se ready-made fancy robot kit use karna sikhata hai, tum use aasaani se use kar sakte ho—lekin tumhe samajh nahi aata kyun ye kaam karta hai.

Neural network scratch se banana aise hai jaise robot ko alag-alag parts se banana: motors, sensors, wires. Tumhe karna hoga:

Sahi wiring (forward pass): Sensors → brain → wheels connect karo taaki information flow kare
Galtiyon se seekhna (backward pass): Jab robot galti kare, pata lagao ki kaun si wires ne galat signals bheje aur unhe adjust karo
Dheere-dheere behtar hona (update): Har baar thode chote adjustments karo taaki overcorrect na ho

"From scratch" wala part matlab hai tum basic tools (jaise simple math operations) use kar rahe ho pre-built kits ki jagah. Mushkil hai, lekin ab tum exactly samajhte ho ki robot ke brain mein information kaise flow karta hai, ye errors se kaise seekhta hai, aur jab kuch toot jaaye toh kaise fix karein.

Key insight: Seekhna sirf patterns dhundhna hai cheezein try karke, maapna ki tum kitne galat ho, aur "knobs" (weights) ko thodi si us direction mein adjust karna jo galti kam kare. Ye haazaron baar karo, aur jaadu se, ye seekh leta hai!

Connections

6.1.01-Neural-Network-Fundamentals - Jo hum implement kar rahe hain uska foundation
6.2.03-Backpropagation-Algorithm - Gradient computation ki mathematical details
6.3.04-Optimization-Algorithms - SGD se aage advanced update rules
4.2.02-Gradient-Descent - Core optimization concept
6.4.01-PyTorch-Basics - Frameworks in details kaise abstract karte hain
6.5.01-Reading-Research-Papers - Novel architectures samajhna jo implement karni hain
2.3.01-Matrix-Operations - Saare operations ke liye linear algebra foundation
3.1.04-Chain-Rule - Backpropagation ke liye core calculus

#flashcards/ai-ml

Models scratch se implement karne ka primary purpose kya hai? :: Computational machinery—gradient flow, memory layout, numerical stability—ko samajhna jo frameworks abstract kar dete hain, taaki better debugging, optimization, aur innovation ho sake.

From-scratch implementation ke teen core components kya hain?

1) Forward pass (inputs se outputs compute karna), 2) Backward pass (backpropagation se gradients compute karna), 3) Parameter update (gradients apply karke seekhna).

Forward pass values cache kyun karni chahiye?

Kyunki backward pass ko chain rule se gradients compute karne ke liye intermediate activations aur pre-activation values chahiye; unhe recompute karna inefficient hai aur numerically unstable ho sakta hai.

Dense layer ke forward pass ka formula kya hai?

z = Wx + b

, jahan

W

input space ko rotate/scale karta hai aur

b

decision boundary translate karta hai.

Activation functions kyun chahiye?

Non-linear activations ke bina, layers stack karna sirf ek aur linear function deta hai:

W_{2} (W_{1} x) = W^{'} x

. Non-linearity universal function approximation enable karti hai.

Dense layer ka weights ke saath gradient derive karo.

Z = WX + b

aur chain rule se:

\frac{\partial L}{\partial W} = \frac{\partial L}{\partial Z} \frac{\partial Z}{\partial W} = δ X^{T}

, jahan

δ = \frac{\partial L}{\partial Z}

Weight gradients compute karte waqt X ko transpose kyun karte hain?

Dimensionality matching:

δ

(m \times n_{ba t c h})

hai,

X^{T}

(n_{ba t c h} \times n)

hai, result

(m \times n)

hota hai jo

W

ke shape se match karta hai.

Dense layer ka input ke saath gradient kya hai?

\frac{\partial L}{\partial X} = W^{T} δ

, jahan

W^{T}

forward transformation ko "reverse" karta hai taaki gradients backward flow karein.

Bias ke liye batch ke across gradients sum kyun karte hain?

Bias ek batch ke saare samples mein shared hota hai, isliye hum single shared bias parameter update karne ke liye har sample ke gradient contributions accumulate karte hain.

ReLU activation ka derivative kya hai?

\frac{\partial}{\partial z} ReLU (z) = {10 if z > 0 if z \leq 0

— gradient sirf wahan flow karta hai jahan input positive tha.

Gradient descent update rule derive karo.

Taylor expansion

L (θ - α \nabla L) \approx L (θ) - α ∥\nabla L ∥^{2}

se, doosra term negative hai, isliye

- \nabla L

direction mein move karna loss decrease karta hai. Update:

θ_{t + 1} = θ_{t} - α \nabla_{θ} L

Mini-batch SGD kya hai aur ye kyun kaam karta hai?

Full gradient

\nabla L = \frac{1}{∣ D ∣} \sum \nabla L_{i}

ko ek chote batch

\nabla L \approx \frac{1}{∣ B ∣} \sum_{i \in B} \nabla L_{i}

se approximate karna. Law of large numbers se kaam karta hai: sample average true expectation mein converge karta hai.

Optimization mein momentum kya hai aur ye kaise help karta hai?

Momentum gradients ka exponential moving average accumulate karta hai:

v_{t} = β v_{t - 1} + (1 - β) \nabla L

. Noisy gradients smooth karta hai aur consistent directions mein inertia add karke accelerate karta hai.

Xavier/Glorot initialization kya hai aur kyun?

W \sim N (0, 1/ n_{in})

. Variance stable rakhta hai:

Var (\sum W_{i} x_{i}) = n \cdot Var (W)

, isliye

Var (W) = 1/ n

set karne se

Var (z) = 1

maintain hota hai.

He initialization Xavier se alag kyun hai?

He initialization

N (0, 2/ n_{in})

use karta hai kyunki ReLU aadhe neurons kill kar deta hai (unhe 0 set karta hai), isliye reduced activations compensate karne ke liye 2× variance chahiye.

Softmax computation mein max kyun subtract karte hain?

Numerical stability:

\frac{e ^{z_{i}}}{\sum e ^{z_{j}}} = \frac{e ^{z_{i} - c}}{\sum e ^{z_{j} - c}}

kisi bhi

c

ke liye.

c = max (z)

choose karna

e^{z - c} \leq 1

bound karta hai, overflow rokta hai aur ratios identical rakhta hai.

Activation wale layer se complete backpropagation equation kya hai?

δ_{ℓ - 1} = (W^{(ℓ)})^{T} δ_{ℓ} ⊙ σ^{'} (Z^{(ℓ - 1)})

, jahan

⊙

element-wise multiplication hai aur

σ^{'}

activation derivative hai.

Backprop mein shapes match karne ke liye randomly transpose kyun nahi karna chahiye?

Sahi shape galat gradient guarantee nahi karta. Pehle chain rule se symbolically derive karo: galat transposes mathematically invalid gradients compute karte hain jo dimensionally sahi lagte hain.

Agar activation derivatives bhool jayein toh kya hoga?

Gradients activation ke information flow par effect ko account nahi karte. Example: saturated sigmoid (

z = 10

σ^{'} (10) \approx 0

) vanishing gradients cause karta hai—

σ^{'}

ignore karna is critical signal death ko miss karta hai.

XOR ke liye hidden layer kyun chahiye?

XOR linearly separable nahi hai—koi single line classes ko separate nahi kar sakti. Hidden layer non-linear activation ke saath ek naya feature space create karta hai jahan classes separable ho jaati hain.