Question bank — Flash attention and efficient attention

False. It computes the exact same softmax output — the tiling and online rescaling are just a different order of arithmetic, not an approximation like Linformer/Performer.

False. It does slightly more FLOPs (≈1.2× in the backward pass, from recomputing $P$). The win is fewer HBM reads/writes, not fewer multiplications.

False. Compute is still $O(N^{2}d)$; only the memory footprint drops from $O(N^2)$ to $O(N)$. Wall-clock speedup comes from avoiding memory round-trips, not from a lower arithmetic order.

False. The $\sqrt{d_k}$ divisor is untouched — it stabilizes score variance and is part of the definition of attention. Flash Attention only changes how the softmax is computed, not what it computes.

False. It works in training precisely because it recomputes $S$ and $P$ in the backward pass instead of storing them — that recomputation is the whole point.

True. $S$ has $N^2$ entries, so $N\to 2N$ makes it $(2N{)}^2=4N^2$ — the quadratic term dominates for long sequences, which is exactly what Flash Attention removes.

True. The memory bottleneck is worst when each moved byte does little arithmetic (small $d$). When $d$ is huge the kernel is already compute-bound, so removing memory round-trips helps less.

True. The inputs and the final output must live in HBM ($O(Nd)$ traffic). What it avoids is the extra $O(N^2)$ traffic of writing and re-reading the intermediate $S$ and $P$.

Error: SRAM (~20 MB) is far too small to hold a large $N\times N$ matrix. Flash Attention never materializes the full matrix anywhere — it holds only small $B\times B$ tiles in SRAM at a time.

Error: The two blocks used different max-references ($m^{(1)},m^{(2)}$). You must rescale first: ${\ell }^{\text{new}}=e^{m^{(1)}-m^{\text{new}}}{\ell }^{(1)}+e^{m^{(2)}-m^{\text{new}}}{\ell }^{(2)}$, else you're summing incompatible baselines.

Error: Only the max entry becomes $e^0=1$; the other becomes $e^{x_j-m}<1$. In the parent's example the denominator was $1+0.493=1.493$, not $2$.

Error: Earlier unnormalized outputs must be rescaled by $e^{m^{(1)}-m^{\text{new}}}$ whenever a later block raises the running max. That correction is essential for exactness.

Error: They're categorically different. Linear-attention methods change the math (approximate softmax) to get $O(N)$ compute; Flash Attention keeps softmax exact and only reorganizes memory access. See 4.1.3-Multi-head-attention — Flash Attention slots in per head unchanged.

Error: Flash Attention's backward pass is a form of 5.27-Gradient-checkpointing specialized to attention: it drops the stored activations ($S$, $P$) and recomputes them on demand, trading compute for memory.

Error: The max and the sum are coupled — every exponential is $e^{x_j-m}$, so changing $m$ rescales the whole sum. That coupling is exactly why merging blocks needs the $e^{m^{(b)}-m^{\text{new}}}$ factors.

Moving the $N^2$ score/probability matrices between slow HBM (~1.5 TB/s) and the chip costs on the order of the compute time, and standard attention reads/writes them multiple times, so the kernel spends most wall-clock waiting on memory.

Raw $e^x$ overflows for large scores. Subtracting the max makes the largest exponent exactly $e^0=1$ while leaving the ratios (and hence the softmax) unchanged — it's numerical safety, not a change of answer.

All exponentials share one reference $m$. When $m$ grows, every previously accumulated term was computed against a smaller reference and is now too large by a factor $e^{m^{\text{old}}-m^{\text{new}}}$, so we multiply them down to the common baseline.

Caching them costs $O(N^2)$ memory — the exact thing we're trying to avoid. Recomputing them from the stored $Q,K,V$ costs only ~1.2× FLOPs but keeps memory at $O(N)$, and memory is the binding constraint.

Because $N^2$ grows explosively — doubling $N$ quadruples the attention-matrix memory. Removing that term is what lets models handle long contexts (relevant to 4.13-Scaling-laws-for-transformers and long-context 4.2.3-GPT-architecture variants).

Because the output is a weighted sum of value vectors, and sums can be built up block by block — as long as we carry the running normalizer $\ell$ and rescale partial outputs when the max updates, the running estimate converges to the exact result.

Merging is associative: merging block 1 with block 2, then that result with block 3, gives the same $m,\ell ,O$ as any other grouping — each merge just re-references everything to the new global max.

Then there's no merge step — $m^{\text{new}}=m^{(1)}$, all rescale factors are $e^0=1$, and Flash Attention reduces exactly to ordinary safe-softmax attention.

Both rescale factors are $e^{m-m}=1$, so the merge simplifies to plain addition of $\ell$ and of $\tilde{O}$. Rescaling costs nothing here but is still logically present.

Every exponential is $0$, so $\ell =0$ and the softmax is undefined (0/0). Implementations guard this — a query with no valid keys should never occur; causal masking (see 4.1.8-Positional-encoding neighbours) always leaves at least the token itself.

Blocks entirely in the "future" of a query are skipped (they'd be fully masked), so causal Flash Attention does less work, not more — a big reason it's efficient for 4.2.3-GPT-architecture decoders.

The softmax is uniform ($1/N$ each); the block sums still combine correctly because equal scores share the same max and every exponential is $1$. The output is just the average of the value vectors.

Not exactly bit-for-bit — floating-point addition isn't perfectly associative, so a different summation order gives tiny (≈$10^{-6}$) differences. Mathematically it's exact; numerically it's within normal float tolerance.

Flash Attention applies unchanged — every query attends to every key, so no blocks are skipped, but the tiling and online softmax still avoid materializing the $N\times N$ matrix.