Simultaneous multithreading (SMT - hyperthreading)

Overview

Simultaneous Multithreading (SMT), commercially branded as hyperthreading by Intel, is a microarchitectural technique that allows a single physical processor core to execute instructions from multiple hardware threads in the same clock cycle. This maximizes the utilization of execution units that would otherwise sit idle due to pipeline stalls, cache misses, or instruction dependencies.

Think of it like carpooling: one car (core) carries multiple passengers (threads) to utilize empty seats. The car doesn't go faster, but more people reach their destination per trip.

The Problem: Underutilized Execution Resources

Derivation from First Principles

A typical out-of-order superscalar processor has:

Frontend: instruction fetch, decode (shared)
Execution units: 4-6 ALUs, 2-3 FPUs, 2load/store ports (shared)
Backend: register renaming, reorder buffer (ROB), commit logic (shared)

For a single thread, ideal instruction-level parallelism (ILP) extraction means:

\text{IPC}_{\text{ideal}} = \min\left(\text{Dispatch Width}, \text{Available ILP}\right)

But real programs have:

Data dependencies: instruction $I_2$ waits for $I_1$ 's result
Cache misses: 50-200 cycle stalls waiting for DRAM
Branch mispredictions: pipeline flushes

During a cache miss, all execution units are idle because the thread has no ready instructions. Utilization:

\text{Utilization}_{\text{single}} = \frac{\text{Active cycles}}{\text{Total cycles}} \approx 20\text{-}40\%

Key insight: The hardware exists but sits unused. Can we fill those bubles with work from another thread?

How SMT Works

Architecture Changes (What's Duplicated vs Shared)

Duplicated per thread (cheap):

Program Counter (PC)
Architectural register file (32-64 registers × 2 threads = 128 total)
Thread state (privilege level, exception vectors)

Cost: ~5% die area increase for 2-way SMT.

Execution units (ALUs, FPUs, load/store ports)
Caches (L1, L2, L3)
Branch predictor
TLB (translation lookaside buffer)

The Scheduling Mechanism

Each cycle, the instruction scheduler (part of the out-of-order engine) maintains a ready queue of instructions from all threads. Selection policy:

\text{Select} = \arg\max_{\text{thread}} \left(\text{Priority} \times \text{Ready Instructions}\right)

Round-robin with stall priority: If thread A stalls (cache miss), thread B monopolizes execution units until A unblocks.

Derivation: Throughput Gain

Single-Thread Baseline

Assume:

Core has $N = 4$ execution units
Thread A achieves IPC = 1.5 (uses 1.5 units/cycle on average due to stalls)
Utilization = $\frac{1.5}{4} = 37.5\%$

Time for 1000 instructions:

T_{\text{single}} = \frac{1000}{1.5} = 667 \text{ cycles}

Two-Way SMT

With thread B running simultaneously:

Thread A: IPC = 1.2 (slight slowdown from resource contention)
Thread B: IPC = 1.2
Combined throughput: $1.2 + 1.2 = 2.4$ IPC
Utilization: $\frac{2.4}{4} = 60\%$

Time for 1000 instructions per thread:

T_{\text{SMT}} = \frac{1000}{1.2} = 833 \text{ cycles per thread}

But: Both threads finish in the same 833 cycles, so total work = 2000 instructions in 833 cycles.

Throughput gain:

\text{Speedup} = \frac{2000 / 833}{1000 / 667} = \frac{2.4}{1.5} = 1.6\times

Why not 2×? Threads compete for:

Cache capacity (more mises)
TLB entries (more page faults)
Reorder buffer slots (limits out-of-order window)

Worked Examples

Without SMT:

Thread A: IPC = 2.0, time = 500 cycles
Thread B: IPC = 0.5, time = 2000 cycles
Total time: 500 + 2000 = 2500 cycles (sequential)

With SMT:

While B stalls, A uses execution units
Thread A: IPC = 1.8(slight slowdown), time = 556 cycles
Thread B: IPC = 0.6 (slight speedup), time = 1667 cycles
Both finish in: max(556, 1667) = 1667 cycles

Why this step? We overlap B's stalls with A's computation. The long pole is B's memory latency, but A "fills in" 70% of those cycles.

Result: 2500 → 1667 cycles = 1.5× speedup.

Without SMT:

Each: IPC = 2.5, time = 400 cycles
Total: 800 cycles sequential

With SMT:

Resource contention reduces each to IPC = 1.4
Time per thread: 714 cycles
Both finish in: 714 cycles

Why this step? Limited execution units split50/50. No idle resources to exploit.

Result: 800 → 714 cycles = 1.12× speedup (minimal gain).

Lesson: SMT helps most when threads have complementary resource needs.

Performance Modeling

\text{Speedup} = \frac{1}{(1 - f) + \frac{f}{2}}

Derivation:

CPU-bound fraction: $(1 - f)$ → no benefit (saturated units)
Memory-bound fraction: $f$ → second thread fills50% of bubles (conservative)

Why this formula? In memory-bound cycles, the second thread can use idle units. In CPU-bound cycles, threads compete. This is a simplified model; real speedup depends on cache behavior.

Example: If $f = 0.6$ (60% memory-bound):

\text{Speedup} = \frac{1}{0.4 + 0.3} = 1.43\times

Typical range: 1.2× to 1.4× for general workloads.

Common Mistakes

Steel-man: If execution units are 100% idle during stalls, and the second thread has zero contention, this would be true. In practice:

Shared caches → more misses → longer stalls for both
Limited reorder buffer → restricts out-of-order window
Branch predictor thrashing → more mispredictions

Reality: 20-40% throughput gain is typical. Single-thread latency often increases (more time per task, but more tasks per second).

Steel-man: This would be true for a coarse-grained time-sliced system. But SMT is fine-grained: instructions are selected per cycle based on readiness. A thread stalled on memory can use0% of execution units while the other uses 100%.

Fix: Think "opportunistic sharing," not "static partitioning."

Steel-man: With infinite execution resources, this holds. But:

More threads → more cache thrashing
More threads → larger architectural state (PC, registers) overhead
Diminishing returns beyond 2-4 threads per core

Reality: Most commercial CPUs stop at 2-way (Intel) or 4-way (IBM POWER) SMT.

Connections Out-of-Order Execution: SMT builds on OoO by expanding the instruction pool across threads

Superscalar Processors: Multiple execution units enable SMT's paralelism
Cache Coherence: Shared L1/L2 caches require coherence protocols when threads access shared data
Thread-Level Parallelism (TLP): SMT is TLP at the microarchitecture level (vs OS-level multithreading)
Pipeline Hazards: SMT mitigates structural hazards by borrowing idle units
Branch Prediction: Shared predictor can suffer from inter-thread interference
Memory Hierarchy: Cache miss latency is the primary opportunity for SMT

Recall Explain to a 12-Year-Old

Imagine you're doing homework and you have a pencil, eraser, calculator, and ruler. Sometimes you use the pencil to write, but the calculator just sits there. Your friend also has homework. Instead of taking turns using the desk, you BOTH sit at the desk at the same time! When you're writing (using the pencil), your friend uses the calculator. When you're stuck thinking, your friend uses the pencil. You share the tools, but you each do your own homework. The desk doesn't get bigger, but more homework gets done because the tools aren't sitting around unused. That's hyperthreading—one "desk" (CPU core) helps two "students" (programs) by letting them share tools (execution units) smartly.

Flashcards

What is Simultaneous Multithreading (SMT)? :: A microarchitectural technique where a single physical core maintains multiple hardware thread contexts (PC, registers) and can execute instructions from multiple threads in the same clock cycle, utilizing execution units that would otherwise be idle during stalls.

What hardware resources are DUPLICATED per thread in SMT?

Program Counter (PC), architectural register file, thread state (privilege level, exception vectors). Cost: ~5% die area for 2-way SMT.

What hardware resources are SHARED across threads in SMT?

Execution units (ALUs, FPUs), caches (L1/L2/L3), branch predictor, TLB, reorder buffer. These are too expensive to duplicate.

Why doesn't SMT provide 2× performance with 2 threads?

Because threads compete for shared resources: cache capacity (more misses), TLB entries (more page faults), reorder buffer slots (limits OoO window), and branch predictor accuracy (interference).

When does SMT provide the MOST benefit?

When threads have complementary resource usage: one thread is memory-bound (many stalls) while the other is CPU-bound (can use idle execution units). Typical gain: 1.2-1.4× throughput.

What is the typical throughput speedup from 2-way SMT?

1.2× to 1.4× for general workloads. Can reach 1.6× with ideal complementary threads (memory-bound + CPU-bound). Minimal (<1.1×) when both threads are CPU-bound.

How does SMT affect single-thread latency?

Single-thread latency often INCREASES because the thread faces more cache misses and resource contention. SMT improves THROUGHPUT (work per second) not LATENCY (time per task).

What is the instruction selection policy in SMT?

Each cycle, the scheduler picks instructions from ANY ready thread to fill execution slots. Typically round-robin with stall priority: if one thread stalls (cache miss), the other monopolizes units until it unblocks.

Using Amdahl's Law, if60% of cycles are memory-bound, what is the SMT speedup?

Speedup = 1 / ((1 - 0.6) + 0.6/2) = 1 / 0.7 = 1.43×. The formula is 1/((1-f) + f/2) where f is the memory-bound fraction.

Why do most CPUs stop at 2-way or 4-way SMT instead of 8-way?

Diminishing returns: more threads cause excessive cache thrashing, TLB pressure, and branch predictor interference. Architectural state overhead grows. Beyond 4 threads, performance often degrades.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

SMT matlab Simultaneous Multithreading ek aisa technique hai jisme ek hi physical core do ya zyada hardware threads ko same time pe execute karta hai. Socho ki tumhare pas ek calculator hai, par tum sirf addition kar rahe ho—multiply aur divide buttons bekar pade hain. Abagar tumhara dost bhi tumare sath baith jaye aur wo multiply kare jab tum addition kar rahe ho, toh calculator ka full utilization ho gaya! Exactly yahi SMT karta hai: jab ek thread memory se data lane ke liye wait kar raha hota hai (cache miss), tab dosra thread us core ki idle execution units (ALU, FPU) ka use karke apna kaam karta hai. Result? Throughput (kitna kaam ho raha hai total) 1.3-1.4× badh jata hai, lekin har thread thoda slow ho sakta hai kyunki resources ab share ho rahe hain—cache, TLB, branch predictor sab contested hain.

Iska sabse bada benefit tab milta hai jab ek thread memory-bound ho (bahut sare stalls) aur dosra CPU-bound (continuous computation). Dono threads complementary hain toh overlap perfect hota hai. Paragar dono threads CPU-intensive hain, toh wo execution units ke liye compete karenge aur benefit minimal hoga (~10-15%). Intel ke hyperthreading mein ye same concept hai—ek physical core ko do logical cores ke rupe mein OS dekhta hai. Programmer ko lagta hai ki do cores hain, par hardware level pe ek hi core smartly share ho raha hai. Gaming ya single-threaded apps mein sometimes hyperthreading off karna better hota hai kyunki cache contention se performance degrade hoti hai. Par servers aur multi-threaded workloads ke liye SMT ek game-changer hai—cost almost same, performance meaningful jump!