What is a Tensor?

WHY multi-dimensional? Because neural network data is inherently structured: images have spatial dimensions, text has sequence length, batches have a batch dimension. Flattening everything into 1D would lose structural information and make operations awkward.

For a tensor $\mathbf{T}$ with shape $(d_1, d_2, \ldots, d_n)$: $\text{rank}(\mathbf{T}) = n$ $\text{numel}(\mathbf{T}) = \prod_{i=1}^{n} d_i$

DERIVATION: This is just counting. A2D matrix of shape $(m, n)$ has $m \times n$ elements. Extend to $n$ dimensions: each axis multiplies the count.

Creating Tensors

From Scratch

WHY multiple creation methods? Different initialization strategies matter for neural networks. Random initialization breaks symmetry, zeros/ones are sometimes needed for masks or biases, specific values for debugging.

import torch
 
# From Python list - WHAT: explicit values, HOW: torch.tensor()
x = torch.tensor([1, 2, 3])  # 1D tensor, shape (3,)
y = torch.tensor([[1, 2], [3, 4]])  # 2D, shape (2, 2)
 
# Zeros and ones - WHAT: constant tensors, WHY: common initialization
z = torch.zeros(3, 4)  # shape (3, 4), all 0.0
o = torch.ones(2, 3, dtype=torch.int64)  # integers
 
# Random - WHAT: stochastic initialization, WHY: breaks symmetry inNN
r = torch.rand(2, 3)  # uniform [0, 1), shape (2, 3)
n = torch.randn(2, 3)  # standard normal N(0,1)
 
# Arange and linspace - WHAT: evenly spaced values
a = torch.arange(0, 10, 2)  # [0, 2, 4, 6, 8]
l = torch.linspace(0, 1, 5)  # [0.0, 0.25, 0.5, 0.75, 1.0]
 
# Like operations - WHAT: same shape as another tensor
x_zeros = torch.zeros_like(x)
x_rand = torch.randn_like(y, dtype=torch.float32)

WHY torch.tensor() vs torch.Tensor()? torch.tensor() infers dtype from data and doesn't share memory. torch.Tensor() is an alias for torch.FloatTensor() and is less commonly used.

From NumPy

import numpy as np
 
# NumPy → PyTorch
np_array = np.array([[1, 2], [3, 4]])
torch_tensor = torch.from_numpy(np_array)  # shares memory!
 
# PyTorch → NumPy
back_to_numpy = torch_tensor.numpy()  # also shares memory

WHY share memory? Efficiency. Copying large arrays is expensive. CAVEAT: changes to one affect the other. Use .clone() or .copy() to break sharing.

Core Tensor Operations

Element-wise Operations

WHAT: Apply operation independently to each element. WHY: Activation functions (ReLU, sigmoid), scaling, pixel normalization all use element-wise ops.

x = torch.tensor([1.0, 2.0, 3.0])
y = torch.tensor([4.0, 5.0, 6.0])
 
# Arithmetic - broadcasting applies if shapes differ
z = x + y      # [5., 7., 9.]
z = x *2      # [2., 4., 6.]
z = x ** 2     # [1., 4., 9.]
 
# Functions
z = torch.sqrt(x)     # element-wise square root
z = torch.exp(x)      # e^x for each element
z = torch.log(x)      # natural log

DERIVATION of broadcasting: When shapes differ, PyTorch expands dimensions automatically. Rules:

1. If ranks differ, prepend 1s to the smaller rank: (3,) becomes (1, 3) to match (2, 3)
2. Dimensions are compatible if equal or one is 1
3. The dimension of size 1 is stretched to match the other

Example: (2, 1) + (3,) → (2, 1) + (1, 3) → (2, 3) (both expanded)

Reduction Operations

WHAT: Collapse one or more dimensions by aggregating. WHY: Loss functions average over batches, pooling layers reduce spatial dimensions.

x = torch.tensor([[1., 2., 3.],
                  [4., 5., 6.]])
 
# Sum - WHY: total error, gradient accumulation
total = torch.sum(x)           # 21.0 (scalar)
col_sum = torch.sum(x, dim=0)  # [5., 7., 9.] (sum rows)
row_sum = torch.sum(x, dim=1)  # [6., 15] (sum columns)
 
# Mean - WHY: average loss
avg = torch.mean(x)            # 3.5
 
# Max/Min - WHY: max pooling, finding best prediction
max_val = torch.max(x)         # 6.0
max_per_row, indices = torch.max(x, dim=1)  # returns (values, indices)
 
# Keepdim - WHAT: preserve dimension as size 1
mean_keep = torch.mean(x, dim=1, keepdim=True)  # shape (2, 1)

WHY keepdim? Broadcasting. If you compute mean across one dimension and then subtract it, you need shapes to align. keepdim=True makes (2, 3) - (2, 1) work via broadcasting.

Batch of 4 samples, 3 features each

batch = torch.randn(4, 3)

Normalize each feature across the batch

mean = batch.mean(dim=0, keepdim=True) # shape (1, 3) std = batch.std(dim=0, keepdim=True) normalized = (batch - mean) / (std + 1e-5) # broadcasting works!

**Why this step?** We want each feature to have mean 0, std 1 across the batch. `keepdim` ensures `(4, 3) - (1, 3)` broadcasts correctly.

### Matrix Operations

**WHAT**: Linear algebra operations. **WHY**: Neural network layers are matrix multiplications: $\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{b}$.

```python
# Matrix multiplication
A = torch.randn(2, 3)
B = torch.randn(3, 4)
C = torch.matmul(A, B)  # or A @ B, shape (2, 4)

# DERIVATION: (2,3) × (3,4) → (2,4)
# For C[i,j] = Σ_k A[i,k] * B[k,j], k=0..2
# Inner dimension (3) must match, outer dimensions (2,4) define result

# Batch matrix multiplication
# WHAT: multiply batches of matrices in parallel
batched_A = torch.randn(10, 2, 3)  # 10 matrices of (2,3)
batched_B = torch.randn(10, 3, 4)  # 10 matrices of (3,4)
batched_C = torch.bm(batched_A, batched_B)  # (10, 2, 4)

# Dot product (1D only)
v1 = torch.tensor([1., 2., 3.])
v2 = torch.tensor([4., 5., 6.])
dot = torch.dot(v1, v2)  # 1*4 + 2*5 + 3*6 = 32

# Transpose
A_T = A.T  # or A.transpose(0, 1)

WHY bmm vs matmul? bmm requires exactly 3D tensors and does strict batch matrix multiplication. matmul is more flexible with broadcasting: it handles 1D, 2D, batched, and broadcasts batch dimensions. Use matmul (@) by default.

Input: batch of 32 samples, 784 features (28×28 MNIST)

x = torch.randn(32, 784)

Weight: 784 inputs → 128 outputs

W = torch.randn(784, 128) b = torch.randn(128)

Forward pass: y = xW + b

y = x @ W + b # shape (32, 128)

WHY this step?

Each row of x (one sample) is multiplied by W → 128-dim output

broadcasts to (32, 128), adding same bias to each sample

**Why transpose W?** Convention. Some frameworks use $\mathbf{y} = \mathbf{W}\mathbf{x}$ (W as (128, 784)), PyTorch uses $\mathbf{y} = \mathbf{x}\mathbf{W}$ (W as (784, 128)) so batching is natural: `(batch, in) @ (in, out) = (batch, out)`.

### Reshaping Operations

**WHAT**: Change tensor shape without changing data. **WHY**: Networks expect specific input shapes. CNs need 4D (batch, channels, H, W), fully-connected layers need 2D (batch, features).

```python
x = torch.randn(2, 3, 4)  # 24 elements

# View - WHAT: reshape, shares memory
y = x.view(2, 12)       # (2, 12), same data
y = x.view(-1, 4)       # (-1 infers dimension: 24/4 = 6 → (6, 4))

# Reshape - WHAT: like view but may copy
y = x.reshape(3, 8)

# Squeeze/Unsqueeze - WHAT: remove/add dimensions of size 1
x = torch.randn(1, 3, 1, 4)
y = x.squeeze()         # (3, 4), removed all size-1 dims
y = x.squeeze(2)        # (1, 3, 4), removed only dim 2
z = x.unsqueeze(0)      # (1, 1, 3, 1, 4), added dim at position 0

# Flatten - WHAT: collapse to 1D or 2D
x = torch.randn(10, 3, 28, 28)  # batch of images
flat = x.flatten(start_dim=1)   # (10, 3*28*28) = (10, 2352)

WHY view vs reshape? view requires contiguous memory and returns a view (no copy). reshape may copy if non-contiguous but always succeds. Use reshape unless you specifically need to verify contiguity.

DERIVATION of -1 inference: Total elements must be conserved. If shape is (2, -1, 4) and tensor has 24 elements, then -1 = 24 / (2*4) = 3.

After convolution: (batch=32, channels=64, H=7, W=7)

conv_output = torch.randn(32, 64, 7, 7)

Flatten spatial dimensions for fully-connected layer

fc_input = conv_output.flatten(start_dim=1) # (32, 6477) = (32, 3136)

WHY flatten? FC layers expect 2D: (batch, features)

WHY start_dim=1? Keep batch dimension separate

### Indexing and Slicing

**WHAT**: Extract subsets of tensors. **WHY**: Select specific samples, channels, or regions; implement attention masks.

```python
x = torch.tensor([[1, 2, 3],
                [4, 5, 6],
                  [7, 8, 9]])

# Basic slicing (like NumPy)
row = x[0]           # [1, 2, 3]
col = x[:, 1]        # [2, 5, 8]
submat = x[0:2, 1:3] # [[2, 3], [5, 6]]

# Boolean masking - WHAT: select elements meeting condition
mask = x > 5         # [[False, False, False],
                #  [False, False, True],
                     #  [True, True, True]]
selected = x[mask]   # [6, 7, 8, 9] (flattened)

# Advanced indexing - WHAT: index with tensor of indices
indices = torch.tensor([0, 2])
rows = x[indices]    # [[1, 2, 3], [7, 8, 9]]

# Where - WHAT: conditional selection
result = torch.where(x > 5, x, torch.zeros_like(x))
# If x[i] > 5, use x[i], else 0

ReLU: max(0, x)

relu_manual = torch.where(x > 0, x, torch.zeros_like(x))

Or simpler:

relu = torch.clamp(x, min=0) # clamp values below 0 to 0

Or built-in:

relu = torch.relu(x)

## Device Management (CPU vs GPU)

**WHY care about devices?** GPUs have massively parallel cores optimized for tensor ops. A large matrix multiplication can be100× faster on GPU. But data transfer between CPU/GPU has overhead.

```python
# Check CUDA availability
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Create tensor on GPU directly
x_gpu = torch.randn(1000, 1000, device='cuda')

# Move tensor to GPU
x_cpu = torch.randn(1000, 1000)
x_gpu = x_cpu.to('cuda')  # or .cuda()

# Move back to CPU
x_back = x_gpu.to('cpu')  # or .cpu()

# Operations must be on same device
y_gpu = torch.randn(1000, 1000, device='cuda')
z = x_gpu @ y_gpu  # works, both on GPU
# z = x_gpu @ x_cpu  # ERROR: operands on different devices

Rule of thumb: Move data to GPU once, do all computation there, move results back once. Avoid repeated CPU↔GPU transfers.

Automatic Differentiation Setup

WHAT: requires_grad flag tells PyTorch to track operations for computing gradients. WHY: Backpropagation needs the computational graph.

# Tensor that needs gradients (e.g., network weights)
w = torch.randn(2, 3, requires_grad=True)
x = torch.randn(3, 4)  # input, no grad needed
 
y = w @ x  # y.requires_grad = True (inherited)
 
loss = y.sum()
loss.backward()  # compute gradients
 
print(w.grad)  # dL/dw, same shape as w

WHY only some tensors need grad? Inputs are given, we don't optimize them. Parameters (weights, biases) are what we adjust, so only they need gradients.

Why it feels right: In-place ops (+=, *=, .relu_()) save memory by not creating a new tensor.

The problem: If x is part of the computational graph (requires_grad=True), in-place modification destroys information needed for backprop. PyTorch will raise RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.

The fix: Use out-of-place operations: x = x + 1 instead of x += 1. Or use .detach() if you're sure the operation shouldn't be in the graph.

x = torch.tensor([1., 2., 3.], requires_grad=True)
# x += 1  # ERROR if used in backward later
x = x + 1  # OK, creates new tensor

Why it feels right: Matrix multiplication rules say inner dimensions must match: (10, 5) × (10?) is confusing but has10s in common.

The problem: For A @ B, the last dimension of A must match the first dimension of B. (10, 5) @ (32, 10) tries to do (5 vs 32), which fails.

The fix: For batch operations, input shape is (batch, features), weight shape is (features, outputs). Correct order: batch @ W → (32, 10) @ (10, 5) = (32, 5). Or transpose: W.T @ batch.T.

"Matrix Mult: Inner dims match, Outer dims stay"

• (A, B) @ (B, C) = (A, C) - B cancels, A and C remain

"View changes shape, Transpose changes data order"

• .view() reinterprets indices, .transpose() actually swaps dimensions

In a regular Python list, you just have bricks in a line: [brick1, brick2, brick3, ...]. But what if you're building a Lego castle? You have bricks arranged in layers (height), rows (depth), and columns (width). A 3D tensor is like that: (layers, rows, columns).

PyTorch tensors are special because:

1. They're super fast - Like having a team of 1000 helpers who can all sort bricks at the same time (GPU parallelism).
2. They remember how you built things - If you stack blue bricks on red bricks, PyTorch remembers the steps so if you want to undo something (like training a neural network backward), it knows exactly what to reverse.

When you train a neural network, you're essentially testing different Lego building instructions to see which structure is strongest. Tensors hold all the bricks (data), and PyTorch operations are the rules for snapping them together.

Connections

• Neural Network Fundamentals - Why matrix operations implement layers
• Backpropagation - How requires_grad enables automatic differentiation
• Building Models in PyTorch - Using tensors to define layers
• NumPy Fundamentals - PyTorch tensors are like NumPy arrays with gradients
• Batch Normalization - Reduction operations over batch dimension
• Convolutional Layers - 4D tensor shapes for image data
• GPU Acceleration - Why device management matters for training speed

#flashcards/ai-ml

What is a tensor in PyTorch? :: A multi-dimensional array with uniform dtype, shape, device location, and optional gradient tracking. Generalizes scalars (0D) → vectors (1D) → matrices (2D) → nD arrays.

What error occurs with in-place operations on tensors with requires_grad=True? :: RuntimeError: in-place operations destroy information needed for backpropagation. Use out-of-place operations instead (e.g., x = x + 1 not x += 1).

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)