3.3.1 · AI-ML › Deep Learning Frameworks

Tensor Kya Hai?

WHY multi-dimensional? Kyunki neural network data inherently structured hota hai: images mein spatial dimensions hain, text mein sequence length hai, batches mein ek batch dimension hai. Sab kuch 1D mein flatten karne se structural information kho jaati aur operations awkward ho jaate.

Ek tensor $\mathbf{T}$ ke liye jiska shape $(d_1,d_2,\dots ,d_n)$ hai: $\text{rank}(\mathbf{T})=n$ $\text{numel}(\mathbf{T})={\prod }_{i=1}^n{d}_i$

DERIVATION: Yeh sirf counting hai. Shape $(m,n)$ wali ek 2D matrix mein $m\times n$ elements hote hain. $n$ dimensions tak extend karo: har axis count ko multiply karta hai.

Tensors Banana

Scratch Se

WHY multiple creation methods? Neural networks ke liye alag-alag initialization strategies matter karti hain. Random initialization symmetry todta hai, zeros/ones kabhi kabhi masks ya biases ke liye chahiye hote hain, aur specific values debugging ke liye.

import torch
 
# Python list se - WHAT: explicit values, HOW: torch.tensor()
x = torch.tensor([1, 2, 3])  # 1D tensor, shape (3,)
y = torch.tensor([[1, 2], [3, 4]])  # 2D, shape (2, 2)
 
# Zeros aur ones - WHAT: constant tensors, WHY: common initialization
z = torch.zeros(3, 4)  # shape (3, 4), sab 0.0
o = torch.ones(2, 3, dtype=torch.int64)  # integers
 
# Random - WHAT: stochastic initialization, WHY: NN mein symmetry todta hai
r = torch.rand(2, 3)  # uniform [0, 1), shape (2, 3)
n = torch.randn(2, 3)  # standard normal N(0,1)
 
# Arange aur linspace - WHAT: evenly spaced values
a = torch.arange(0, 10, 2)  # [0, 2, 4, 6, 8]
l = torch.linspace(0, 1, 5)  # [0.0, 0.25, 0.5, 0.75, 1.0]
 
# Like operations - WHAT: kisi doosre tensor jaisi shape
x_zeros = torch.zeros_like(x)
x_rand = torch.randn_like(y, dtype=torch.float32)

WHY torch.tensor() vs torch.Tensor()? torch.tensor() data se dtype infer karta hai aur memory share nahi karta. torch.Tensor() torch.FloatTensor() ka alias hai aur kam commonly use hota hai.

NumPy Se

import numpy as np
 
# NumPy → PyTorch
np_array = np.array([[1, 2], [3, 4]])
torch_tensor = torch.from_numpy(np_array)  # memory share karta hai!
 
# PyTorch → NumPy
back_to_numpy = torch_tensor.numpy()  # yeh bhi memory share karta hai

WHY memory share karo? Efficiency. Badi arrays copy karna expensive hai. CAVEAT: ek mein change doosre ko bhi affect karta hai. Sharing todne ke liye .clone() ya .copy() use karo.

Core Tensor Operations

Element-wise Operations

WHAT: Har element par independently operation apply karo. WHY: Activation functions (ReLU, sigmoid), scaling, pixel normalization sab element-wise ops use karte hain.

x = torch.tensor([1.0, 2.0, 3.0])
y = torch.tensor([4.0, 5.0, 6.0])
 
# Arithmetic - broadcasting apply hoti hai agar shapes alag hain
z = x + y      # [5., 7., 9.]
z = x * 2      # [2., 4., 6.]
z = x ** 2     # [1., 4., 9.]
 
# Functions
z = torch.sqrt(x)     # element-wise square root
z = torch.exp(x)      # har element ke liye e^x
z = torch.log(x)      # natural log

DERIVATION of broadcasting: Jab shapes alag hoti hain, PyTorch dimensions automatically expand karta hai. Rules:

1. Agar ranks alag hain, toh chhote rank mein 1s prepend karo: (3,) becomes (1, 3) taaki (2, 3) se match ho
2. Dimensions compatible hain agar equal hain ya ek 1 hai
3. Size 1 wali dimension doosre se match karne ke liye stretch ho jaati hai

Example: (2, 1) + (3,) → (2, 1) + (1, 3) → (2, 3) (dono expand ho gaye)

Reduction Operations

WHAT: Ek ya zyada dimensions ko aggregating karke collapse karo. WHY: Loss functions batches par average karte hain, pooling layers spatial dimensions reduce karti hain.

x = torch.tensor([[1., 2., 3.],
                  [4., 5., 6.]])
 
# Sum - WHY: total error, gradient accumulation
total = torch.sum(x)           # 21.0 (scalar)
col_sum = torch.sum(x, dim=0)  # [5., 7., 9.] (rows ka sum)
row_sum = torch.sum(x, dim=1)  # [6., 15] (columns ka sum)
 
# Mean - WHY: average loss
avg = torch.mean(x)            # 3.5
 
# Max/Min - WHY: max pooling, best prediction dhundhna
max_val = torch.max(x)         # 6.0
max_per_row, indices = torch.max(x, dim=1)  # returns (values, indices)
 
# Keepdim - WHAT: dimension ko size 1 ke roop mein preserve karo
mean_keep = torch.mean(x, dim=1, keepdim=True)  # shape (2, 1)

WHY keepdim? Broadcasting. Agar tum ek dimension par mean compute karo aur phir use subtract karo, toh shapes align karni padti hain. keepdim=True se (2, 3) - (2, 1) broadcasting ke zariye kaam karta hai.

4 samples ka batch, har ek mein 3 features

batch = torch.randn(4, 3)

Batch ke across har feature ko normalize karo

mean = batch.mean(dim=0, keepdim=True) # shape (1, 3) std = batch.std(dim=0, keepdim=True) normalized = (batch - mean) / (std + 1e-5) # broadcasting kaam karti hai!

**Yeh step kyun?** Hum chahte hain ki batch ke across har feature ka mean 0, std 1 ho. `keepdim` ensure karta hai ki `(4, 3) - (1, 3)` sahi tarah broadcast ho.

### Matrix Operations

**WHAT**: Linear algebra operations. **WHY**: Neural network layers matrix multiplications hain: $\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{b}$.

```python
# Matrix multiplication
A = torch.randn(2, 3)
B = torch.randn(3, 4)
C = torch.matmul(A, B)  # ya A @ B, shape (2, 4)

# DERIVATION: (2,3) × (3,4) → (2,4)
# C[i,j] = Σ_k A[i,k] * B[k,j] ke liye, k=0..2
# Inner dimension (3) match karni chahiye, outer dimensions (2,4) result define karte hain

# Batch matrix multiplication
# WHAT: matrices ke batches parallel mein multiply karo
batched_A = torch.randn(10, 2, 3)  # 10 matrices of (2,3)
batched_B = torch.randn(10, 3, 4)  # 10 matrices of (3,4)
batched_C = torch.bmm(batched_A, batched_B)  # (10, 2, 4)

# Dot product (sirf 1D ke liye)
v1 = torch.tensor([1., 2., 3.])
v2 = torch.tensor([4., 5., 6.])
dot = torch.dot(v1, v2)  # 1*4 + 2*5 + 3*6 = 32

# Transpose
A_T = A.T  # ya A.transpose(0, 1)

WHY bmm vs matmul? bmm exactly 3D tensors require karta hai aur strict batch matrix multiplication karta hai. matmul broadcasting ke saath zyada flexible hai: yeh 1D, 2D, batched handle karta hai aur batch dimensions broadcast karta hai. By default matmul (@) use karo.

Input: 32 samples ka batch, 784 features (28×28 MNIST)

x = torch.randn(32, 784)

Weight: 784 inputs → 128 outputs

W = torch.randn(784, 128) b = torch.randn(128)

Forward pass: y = xW + b

y = x @ W + b # shape (32, 128)

WHY yeh step?

x ki har row (ek sample) W se multiply hoti hai → 128-dim output

(32, 128) tak broadcast hoti hai, har sample mein same bias add hoti hai

**W transpose kyun?** Convention. Kuch frameworks $\mathbf{y} = \mathbf{W}\mathbf{x}$ use karte hain (W as (128, 784)), PyTorch $\mathbf{y} = \mathbf{x}\mathbf{W}$ use karta hai (W as (784, 128)) taaki batching natural ho: `(batch, in) @ (in, out) = (batch, out)`.

### Reshaping Operations

**WHAT**: Data change kiye bina tensor shape badlo. **WHY**: Networks specific input shapes expect karte hain. CNNs ko 4D chahiye (batch, channels, H, W), fully-connected layers ko 2D chahiye (batch, features).

```python
x = torch.randn(2, 3, 4)  # 24 elements

# View - WHAT: reshape, memory share karta hai
y = x.view(2, 12)       # (2, 12), same data
y = x.view(-1, 4)       # (-1 dimension infer karta hai: 24/4 = 6 → (6, 4))

# Reshape - WHAT: view jaisa lekin copy kar sakta hai
y = x.reshape(3, 8)

# Squeeze/Unsqueeze - WHAT: size 1 wali dimensions hatao/jodo
x = torch.randn(1, 3, 1, 4)
y = x.squeeze()         # (3, 4), saari size-1 dims hata di
y = x.squeeze(2)        # (1, 3, 4), sirf dim 2 hata di
z = x.unsqueeze(0)      # (1, 1, 3, 1, 4), position 0 par dim add ki

# Flatten - WHAT: 1D ya 2D mein collapse karo
x = torch.randn(10, 3, 28, 28)  # images ka batch
flat = x.flatten(start_dim=1)   # (10, 3*28*28) = (10, 2352)

WHY view vs reshape? view contiguous memory require karta hai aur ek view return karta hai (no copy). reshape non-contiguous hone par copy kar sakta hai lekin hamesha succeed karta hai. Jab tak contiguity specifically verify karni ho, reshape use karo.

DERIVATION of -1 inference: Total elements conserve hone chahiye. Agar shape (2, -1, 4) hai aur tensor mein 24 elements hain, toh -1 = 24 / (2*4) = 3.

Convolution ke baad: (batch=32, channels=64, H=7, W=7)

conv_output = torch.randn(32, 64, 7, 7)

Fully-connected layer ke liye spatial dimensions flatten karo

fc_input = conv_output.flatten(start_dim=1) # (32, 6477) = (32, 3136)

WHY flatten? FC layers ko 2D chahiye: (batch, features)

WHY start_dim=1? Batch dimension alag rakhne ke liye

### Indexing aur Slicing

**WHAT**: Tensors ke subsets extract karo. **WHY**: Specific samples, channels, ya regions select karo; attention masks implement karo.

```python
x = torch.tensor([[1, 2, 3],
                  [4, 5, 6],
                  [7, 8, 9]])

# Basic slicing (NumPy jaisa)
row = x[0]           # [1, 2, 3]
col = x[:, 1]        # [2, 5, 8]
submat = x[0:2, 1:3] # [[2, 3], [5, 6]]

# Boolean masking - WHAT: condition meet karne wale elements select karo
mask = x > 5         # [[False, False, False],
                     #  [False, False, True],
                     #  [True, True, True]]
selected = x[mask]   # [6, 7, 8, 9] (flattened)

# Advanced indexing - WHAT: indices ke tensor se index karo
indices = torch.tensor([0, 2])
rows = x[indices]    # [[1, 2, 3], [7, 8, 9]]

# Where - WHAT: conditional selection
result = torch.where(x > 5, x, torch.zeros_like(x))
# Agar x[i] > 5, toh x[i] use karo, warna 0

ReLU: max(0, x)

relu_manual = torch.where(x > 0, x, torch.zeros_like(x))

Ya simpler:

relu = torch.clamp(x, min=0) # 0 se neeche values ko 0 par clamp karo

Ya built-in:

relu = torch.relu(x)

## Device Management (CPU vs GPU)

**WHY** devices ki parwah karo? GPUs mein massively parallel cores hote hain jo tensor ops ke liye optimized hain. Ek badi matrix multiplication GPU par 100× faster ho sakti hai. Lekin CPU/GPU ke beech data transfer ka overhead hota hai.

```python
# CUDA availability check karo
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Seedha GPU par tensor create karo
x_gpu = torch.randn(1000, 1000, device='cuda')

# Tensor ko GPU par move karo
x_cpu = torch.randn(1000, 1000)
x_gpu = x_cpu.to('cuda')  # ya .cuda()

# CPU par wapas laao
x_back = x_gpu.to('cpu')  # ya .cpu()

# Operations sabhi tensors ko same device par require karte hain
y_gpu = torch.randn(1000, 1000, device='cuda')
z = x_gpu @ y_gpu  # kaam karta hai, dono GPU par hain
# z = x_gpu @ x_cpu  # ERROR: operands alag devices par hain

Rule of thumb: Data ko GPU par ek baar move karo, saari computation wahan karo, results ek baar wapas laao. Baar baar CPU↔GPU transfers se bacho.

Automatic Differentiation Setup

WHAT: requires_grad flag PyTorch ko batata hai ki gradients compute karne ke liye operations track karo. WHY: Backpropagation ko computational graph chahiye.

# Tensor jise gradients chahiye (e.g., network weights)
w = torch.randn(2, 3, requires_grad=True)
x = torch.randn(3, 4)  # input, grad ki zarurat nahi
 
y = w @ x  # y.requires_grad = True (inherited)
 
loss = y.sum()
loss.backward()  # gradients compute karo
 
print(w.grad)  # dL/dw, w jaisi hi shape

WHY sirf kuch tensors ko grad chahiye? Inputs diye hue hain, hum unhe optimize nahi karte. Parameters (weights, biases) woh hain jo hum adjust karte hain, isliye sirf unhe gradients chahiye.

Yeh sahi kyun lagta hai: In-place ops (+=, *=, .relu_()) memory save karte hain kyunki nayi tensor create nahi hoti.

Problem yeh hai: Agar x computational graph ka hissa hai (requires_grad=True), toh in-place modification backprop ke liye zaruri information destroy kar deti hai. PyTorch RuntimeError: a leaf Variable that requires grad has been used in an in-place operation raise karega.

Fix yeh hai: Out-of-place operations use karo: x += 1 ki jagah x = x + 1. Ya .detach() use karo agar tum sure ho ki operation graph mein nahi honi chahiye.

x = torch.tensor([1., 2., 3.], requires_grad=True)
# x += 1  # ERROR agar baad mein backward mein use kiya
x = x + 1  # OK, nayi tensor create karta hai

Yeh sahi kyun lagta hai: Matrix multiplication rules kehte hain ki inner dimensions match karni chahiye: (10, 5) × (10?) confusing hai lekin dono mein 10 common hai.

Problem yeh hai: A @ B ke liye, A ki last dimension, B ki first dimension se match karni chahiye. (10, 5) @ (32, 10) (5 vs 32) try karta hai, jo fail hota hai.

Fix yeh hai: Batch operations ke liye, input shape (batch, features) hai, weight shape (features, outputs) hai. Sahi order: batch @ W → (32, 10) @ (10, 5) = (32, 5). Ya transpose karo: W.T @ batch.T.

"Matrix Mult: Inner dims match, Outer dims stay"

• (A, B) @ (B, C) = (A, C) - B cancel hota hai, A aur C rehte hain

"View shape badalta hai, Transpose data order badalta hai"

• .view() indices reinterpret karta hai, .transpose() actually dimensions swap karta hai

Ek regular Python list mein, bricks ek line mein hoti hain: [brick1, brick2, brick3, ...]. Lekin agar tum Lego ka kila bana rahe ho? Tumhare paas layers (height), rows (depth), aur columns (width) mein rakhi bricks hain. Ek 3D tensor waise hi hai: (layers, rows, columns).

PyTorch tensors special hain kyunki:

1. Yeh super fast hain - Jaise 1000 helpers ki ek team ho jo sab ek saath bricks sort kar sakti hai (GPU parallelism).
2. Yeh yaad rakhte hain ki tumne cheezein kaise banayi - Agar tum neeli bricks laal bricks ke upar rakhte ho, PyTorch steps yaad rakhta hai taaki agar tum kuch undo karna chahte ho (jaise neural network ko backward train karna), toh use pata ho exactly kya reverse karna hai.

Jab tum ek neural network train karte ho, tum essentially alag-alag Lego building instructions test kar rahe ho dekhne ke liye kaunsa structure sabse mazboot hai. Tensors saari bricks (data) hold karti hain, aur PyTorch operations unhe snap karne ke rules hain.

Connections

• Neural Network Fundamentals - Kyun matrix operations layers implement karti hain
• Backpropagation - Kaise requires_grad automatic differentiation enable karta hai
• Building Models in PyTorch - Layers define karne ke liye tensors ka use
• NumPy Fundamentals - PyTorch tensors NumPy arrays jaisi hain lekin gradients ke saath
• Batch Normalization - Batch dimension par reduction operations
• Convolutional Layers - Image data ke liye 4D tensor shapes
• GPU Acceleration - Training speed ke liye device management kyun matter karta hai

#flashcards/ai-ml

PyTorch mein ek tensor kya hota hai? :: Ek multi-dimensional array jisme uniform dtype, shape, device location, aur optional gradient tracking hoti hai. Scalars (0D) → vectors (1D) → matrices (2D) → nD arrays ko generalise karta hai.

requires_grad=True wale tensors par in-place operations se kaunsi error aati hai? :: RuntimeError: in-place operations backpropagation ke liye zaruri information destroy kar deti hain. Uski jagah out-of-place operations use karo (e.g., x += 1 nahi balki x = x + 1).

Concept Map