PyTorch tensors and operations: Question bank

Before starting, three words you must not confuse:

Prerequisite links: 2.4.01-numpy-fundamentals, 3.1.01-neural-network-fundamentals, 3.2.03-backpropagation, 4.2.01-gpu-acceleration.

True or false — justify

torch.from_numpy(arr) gives you an independent copy of the data.

False. It shares memory with the NumPy array — edit one and the other changes. Use .clone() (torch) or .copy() (numpy) to break the link.

A tensor with requires_grad=True on the GPU tracks gradients just like on CPU.

True. requires_grad is orthogonal to device; autodiff bookkeeping travels with the tensor to CUDA. See 4.2.01-gpu-acceleration.

x.view(-1, 4) always works on any tensor with a multiple-of-4 element count.

False. view needs the tensor to be contiguous. After some ops (like .transpose()) memory is scrambled; view then raises an error and you must use .reshape() or .contiguous().view().

torch.zeros(3, 4) and torch.zeros(3, 4, dtype=torch.int64) differ only in speed.

False. They differ in dtype. The default is float32; forcing int64 changes what values fit and how downstream ops behave (gradients need floats, indices need ints).

Summing over dim=0 on a shape

(2, 3)

tensor gives a scalar.

False. It collapses only axis 0, leaving shape

(3,)

— one value per column. To get a scalar you sum with no dim, collapsing everything.

A @ B and torch.bmm(A, B) are interchangeable for 3D tensors.

Mostly true but not identical. bmm demands exactly 3D and does strict batched matmul; @/matmul also handles 1D/2D and broadcasts the batch dimension. Prefer matmul for flexibility.

Broadcasting (2, 1) + (3,) fails because the shapes are unequal.

False. It succeeds: the (3,) is padded to (1, 3), then both size-1 axes stretch, giving

(2, 3)

. Size-1 axes are the whole point of broadcasting.

x.squeeze() on shape

(1, 3, 1, 4)

always returns

(3, 4)

True here, but the trap is generality: bare squeeze() removes every size-1 axis. If a batch dimension is legitimately 1, it gets deleted too — pass an explicit dim to be safe.

torch.mean(x, dim=1) and torch.mean(x, dim=1, keepdim=True) hold the same numbers.

True — same values. They differ in shape: without keepdim you get

(2,)

, with it you get

(2, 1)

, which is what broadcasting a subtraction back onto

(2, 3)

needs.

The @ operator on a

(2, 3)

and a

(3, 4)

tensor picks either

(2, 4)

(4, 2)

depending on order.

False. Order is fixed by the inner dimension rule: the two 3's must touch, giving

(2, 4)

. Writing B @ A would try to match 4 against 2 and fail.

Spot the error

C = torch.bm(batched_A, batched_B) on two

(10, 2, 3)

and

(10, 3, 4)

tensors.

The function is torch.bmm (two m's), not bm. The math is right — result

(10, 2, 4)

— but the name is a typo that raises AttributeError.

y = x.view(2, 12) right after x = x.transpose(1, 2).

After transpose the tensor is non-contiguous, so view throws. Fix with x.reshape(2, 12) or x.contiguous().view(2, 12).

normalized = (batch - mean) / std where mean = batch.mean(dim=0) (no keepdim), batch is

(4, 3)

mean is shape

(3,)

here so it broadcasts fine, BUT dividing by raw std risks division by zero. The parent note adds 1e-5: (std + 1e-5). Missing that epsilon is the real bug in fragile normalizers — see 3.4.05-batch-normalization.

dot = torch.dot(A, B) where A is

(2, 3)

and B is

(3, 4)

torch.dot works on 1D vectors only. For matrices use matmul/@. This raises a dimension error.

Editing torch_tensor[0,0] = 99 and expecting the original np_array to stay untouched after torch.from_numpy.

They share memory, so np_array[0,0] also becomes 99. The expectation is the error, not the code.

y = x @ W + b with W of shape

(128, 784)

for input x of shape

(32, 784)

Inner dims don't match:

784

(of x) vs

128

(of W). PyTorch convention is W shape

(784, 128)

so (32,784)@(784,128)=(32,128). See 3.3.02-building-modelspytorch.

Why questions

Why does PyTorch let from_numpy share memory instead of always copying?

Copying large arrays wastes time and RAM. Sharing is a deliberate speed choice; the cost is the caveat that mutations leak across the boundary.

Why does a reduction need the keepdim option at all?

So the reduced result can be broadcast back against the original. Subtracting a

(2, 1)

mean from a

(2, 3)

tensor aligns automatically; a

(2,)

mean would not.

Why does PyTorch use

y = xW

with W shaped (in, out) rather than the textbook

Wx

So batching is natural: (batch, in) @ (in, out) = (batch, out). Each row (one sample) flows through without transposing per call.

Why must the inner dimensions match in matrix multiplication?

Because

C [i, j] = \sum_{k} A [i, k] B [k, j]

sums over the shared index

k

. If A's column count and B's row count differ, the sum has no consistent range to run over.

Why prefer matmul/@ over bmm as a default?

matmul covers 1D, 2D, batched, and broadcasts batch dims, so one operator handles almost every case; bmm is a rigid 3D-only special case.

Why does adding a bias vector b of shape

(128,)

to a

(32, 128)

output "just work"?

Broadcasting:

b

is padded to

(1, 128)

and stretched across all 32 rows, so every sample gets the same bias — exactly the intended behaviour.

Why does breaking symmetry with random init matter?

If all weights start equal, every neuron computes the same thing and receives the same gradient, so they never differentiate — the network can't learn. Randomness (via torch.randn) breaks this tie. See 3.2.03-backpropagation.

Edge cases

What is numel of a tensor with shape

(0, 5)

Zero. The product of dimensions includes the 0, so the tensor is empty — valid but holds no elements. Many ops silently produce empty results.

What does torch.max(x, dim=1) return — one thing or two?

Two things: a tuple (values, indices). Forgetting the second slot (the argmax positions) and unpacking only one value is a classic bug in prediction code.

What is the rank of torch.tensor(5)?

Rank 0 — a scalar tensor, shape (), numel 1. It has no axes to index, which surprises people expecting at least 1D.

x.unsqueeze(0) versus x.unsqueeze(-1) on shape

(3, 4)

— what shapes?

unsqueeze(0) →

(1, 3, 4)

(new axis at front, e.g. adding a batch dim); unsqueeze(-1) →

(3, 4, 1)

(new axis at the end). Position matters.

Dividing by std when a feature is constant across the batch — what happens?

std is 0, so you divide by zero → inf/nan. This is why the epsilon + 1e-5 in the normalization formula is mandatory, not decorative.

What does torch.arange(0, 10, 2) include at the top end?

[0, 2, 4, 6, 8] — the stop value 10 is excluded (half-open interval), unlike linspace which includes both endpoints. Mixing them up gives off-by-one bugs.

torch.rand(2,3) vs torch.randn(2,3) — what ranges?

rand draws uniform in

[0, 1)

(bounded, all positive); randn draws standard normal

N (0, 1)

(unbounded, can be negative). Choosing wrong skews your initialization.

Recall Quick self-check

The single trap that catches most beginners on this topic? ::: Memory sharing after from_numpy/.numpy()/.view() — silent aliasing that mutates data you thought was safe. When in doubt, .clone().