Multicore vs manycore designs

Core Concept

Modern parallel processors fall into two fundamental architectural philosophies: multicore (few powerful cores) and manycore (many simpler cores). This design choice represents a fundamental tradeoff between per-thread performance and throughput capacity.

The key insight: You can't have both. Transistor budget is fixed. Every dollar spent on complex out-of-order execution logic, large caches, and branch prediction is a dollar not spent on adding more cores.

Detailed Comparison

Multicore Architecture

Design Philosophy: Maximize single-thread performance. Each core can execute complex, irregular code with unpredictable branches and memory access patterns efficiently.

Why this architecture?

1. Legacy code optimization: Most software is written for single-threaded execution
2. Latency-critical tasks: Operating systems, databases, compilers benefit from fast sequential execution
3. Irregular paralelism: Tasks with unpredictable control flow (web servers, transaction processing)

$\text{Speedup} = \frac{1}{(1-P) + \frac{P}{N}}$

Where:

• $P$ = fraction of code that can be parallelized
• $N$ = number of cores
• $(1-P)$ = serial fraction (the killer)

Derivation from first principles: Total execution time has two components: $T_{\text{total}} = T_{\text{serial}} + T_{\text{parallel}}$

With $N$ cores, only parallel portion speeds up: $T_{\text{multicore}} = T_{\text{serial}} + \frac{T_{\text{parallel}}}{N}$

Normalizing by original time $T_{\text{orig}} = T_{\text{serial}} + T_{\text{parallel}}$: $\text{Speedup} = \frac{T_{\text{orig}}}{T_{\text{multicore}}} = \frac{T_{\text{serial}} + T_{\text{parallel}}}{T_{\text{serial}} + \frac{T_{\text{parallel}}}{N}}$

Dividing numerator and denominator by $T_{\text{orig}}$: $\text{Speedup} = \frac{1}{\frac{T_{\text{serial}}}{T_{\text{orig}}} + \frac{T_{\text{parallel}}}{N \cdot T_{\text{orig}}}} = \frac{1}{(1-P) + \frac{P}{N}}$

Critical insight: If even5% is serial ($P=0.95$), max speedup is $\frac{1}{0.05} = 20×$ regardless of core count. This is why multicore focuses on keeping each core fast.

Manycore Architecture

Design Philosophy: Maximize aggregate throughput through massive parallelism. Hide latency with thread-level parallelism rather than instruction-level tricks.

Why this architecture?

1. Data paralelism: Same operation on massive datasets (SIMT - Single Instruction Multiple Threads)
2. Throughput-oriented: Scientific computing, graphics, machine learning
3. Latency tolerance: Can switch to another thread during memory stalls (100+ cycles)

$\text{Throughput} = N_{\text{cores}} \times f_{\text{clock}} \times \text{IPC} \times \text{Utilization}$

Derivation:

• Each core completes $\text{IPC}$ (instructions per cycle) × $f_{\text{clock}}$ (cycles/sec) instructions/second
• With $N_{\text{cores}}$, raw capacity is $N \times f \times \text{IPC}$
• Utilization factor (0-1) accounts for thread divergence, memory bottlenecks

Key design choice: Reduce IPC and $f$ per core to increase $N$ dramatically.

Example calculation:

• Multicore: 8 cores × 4 GHz × 4 IPC × 0.9 util = 115.2 GIPS
• Manycore: 512 cores × 1.5 GHz × 1 IPC × 0.7 util = 537.6 GIPS (4.7× higher throughput)

But on a single thread, multicore executes $4 \times 4 = 16×$ faster per clock!

Example 1: Intel Core i9(Multicore)

• Architecture: 8-16 high-performance cores
• Per-core features:
  • 512KB L2 cache per core
  • 6-wide superscalar execution
  • 32MB shared L3 cache
  • Hyper-Threading (2 threads/core)
• Use case: Gaming, content creation, general computing
• Why it works: Games have complex AI, physics, rendering pipelines that benefit from fast sequential execution. Even with 8 threads, single-thread performance matters (one thread often bottlenecks).

Step-by-step performance analysis:

1. Game physics engine has 30% parallel, 70% serial code
2. With 8 cores: Speedup = $\frac{1}{0.7 + \frac{0.3}{8}} = \frac{1}{0.7375} = 1.36×$
3. But each core runs at 4.5 GHz with high IPC → absolute performance is high
4. Why this step? Even modest parallel speedup is acceptable when base performance is excellent.

Example 2: NVIDIA A100 GPU (Manycore)

• Architecture: 6912 CUDA cores (108 SMs × 64 cores each)
• Per-core features:
  • Very simple in-order pipeline
  • Tiny caches, massive thread count
  • Hardware thread scheduler (32-wide warps)
• Use case: Deep learning training, scientific simulation
• Why it works: Matrix multiplication is100% parallel. Training a neural network processes millions of weights independently.

Step-by-step performance analysis:

1. Matrix multiplication: $C = A \times B$ where $A, B$ are $1024 \times 1024$
2. Each of $1024^2 = 1,048,576$ output elements computed independently
3. Assign each element to a thread: 100% parallel ($P=1.0$)
4. Speedup = $\frac{1}{0 + \frac{1}{6912}} \approx 6912×$ (theoretical)
5. Why this step? With perfect parallelism, more cores = linear speedup.
6. Actual: ~4000× due to memory bandwidth limits, but still dominates multicore.

Example 3: Database Query (Multicore Wins)

• Task: Complex JOIN with 5tables, WHERE clauses, aggregations
• Why multicore?
  1. Query planning: sequential logic, unpredictable branches
  2. Index traversals: pointer chasing (terrible for manycore)
  3. Lock management: sequential critical sections
  4. Even with parallel scan, complexity favors fast cores

The Resource Tradeoff

A chip has fixed resources:

$A_{\text{total}} = N_{\text{cores}} \times (A_{\text{core}} + A_{\text{cache}} + A_{\text{control}}) + A_{\text{interconnect}}$

Where:

• $A_{\text{total}}$ = total die area (fixed by process node and cost)
• $A_{\text{core}}$ = area per core's execution units
• $A_{\text{cache}}$ = cache area per core
• $A_{\text{control}}$ = out-of-order logic, branch prediction, etc.
• $A_{\text{interconnect}}$ = network-on-chip for core communication

Derivation of the tradeoff: For multicore: $A_{\text{control}}$ and $A_{\text{cache}}$ are LARGE (50-60% of core area) For manycore: $A_{\text{control}} \approx 0$, $A_{\text{cache}}$ is minimal (10-15% of core area)

If a complex core occupies area $A_{\text{complex}} = 10\text{ mm}^2$, and simple core occupies $A_{\text{simple}} = 0.2 \text{ mm}^2$:

$N_{\text{simple}} = \frac{A_{\text{total}}}{A_{\text{simple}}} = 50 \times N_{\text{complex}}$

Why this matters: You trade1 complex core for ~50 simple cores. This works ONLY if your workload can use all 50 efficiently.

Memory Access Patterns

Multicore strategy: Hide latency with complex caching and out-of-order execution

• Large caches capture working set (temporal locality)
• Out-of-order execution finds independent instructions during memory stalls
• Hardware prefetchers predict access patterns
• Cost: Huge silicon area, power

Manycore strategy: hide latency with massive thread-level parallelism

• When thread1 waits for memory, instantly switch to thread 2
• With 100 threads per core, always find work
• Require SIMD/SIMT (all threads execute same instruction)
• Cost: Need thousands of parallel threads; doesn't help irregular code

Scenario: Load from DRAM takes 300 cycles

Multicore approach (Intel core):

1. Issue load instruction
2. Out-of-order engine finds 100+ independent instructions in window
3. Execute those while waiting
4. If independent work runs out, core stalls (bad!)
5. Why this step? Exploiting ILP within a single thread to hide latency.

Manycore approach (GPU):

1. Thread 1 issues load instruction → context switch (1 cycle)
2. Thread 2 executes → context switch
3. Thread 3 executes → ...
4. Thread 32 executes → back to thread 1, data likely arrived
5. Why this step? With enough threads, never stall; always someone ready to run.

Common Misconceptions

Why it's wrong: Speedup limited by serial fraction (Amdahl's Law) AND by communication overhead.

The fix: $\text{Real Speedup} = \frac{1}{(1-P) + \frac{P}{N} + C(N)}$

where $C(N)$ is communication/synchronization overhead that GROWS with $N$.

With1000 cores, if synchronization takes 1% of time per core: $C(N) = 0.01 \times N = 10$ → negative speedup!

Example: Sorting 1M numbers with 1000 cores:

• Partition data: $O(N)$ work, negligible
• Sort locally: $O(\frac{N}{P} \log \frac{N}{P})$ - perfect parallelism
• Merge results: $O(N \log P)$ - serial bottleneck
• For $P=1000$, merge time dominates unless $N$ is enormous

Why it's wrong: Only true for data-parallel workloads. GPUs are terrible at:

• Branching code (if-else causes "warp divergence" - wastes cycles)
• Pointer chasing (linked lists, trees)
• Small datasets (overhead of launching kernels)
• Irregular memory access

The fix: Match architecture to workload pattern.

Example - Tree traversal:

// Terrible on GPU
struct Node { int value; Node* left; Node* right; };
int search(Node* root, int target) {
    if (!root) return -1;
    if (root->value == target) return 1;
    return search(root->left, target) || search(root->right, target);
}

Why GPU fails:

1. Every thread follows different path (warp divergence)
2. Pointer chasing: 100% cache misses, can't hide latency
3. No data paralelism: each search is dependent

CPU with branch prediction + large cache wins by10×+.

Why it's wrong: The serial portion STILL runs at clock speed. By Amdahl's Law, serial portion determines ultimate speedup ceiling.

The fix: Balance matters. Extremely slow cores hurt even parallel code.

Example: Video encoding (90% parallel)

• Fast cores (4 GHz): Serial takes 10s, parallel takes 90s → with 8 cores: $10 + \frac{90}{8} = 21.25s$
• Slow cores (1 GHz): Serial takes 40s, parallel takes 360s → with 32 cores: $40 + \frac{360}{32} = 51.25s$

Despite 4× more cores, slow architecture is 2.4× SLOWER due to serial bottleneck.

Design Decision Framework

Use Multicore when: $\frac{T_{\text{serial}}}{T_{\text{total}}} > 0.05 \text{ OR branching/irregular memory}$

Use Manycore when: $P > 0.95 \text{ AND } N_{\text{available\_threads}} \gg N_{\text{cores}}$

Formal criteria:

1. Paralelism availability: Can you create $10× - 100×$ more threads than manycore cores?
2. SIMD/SIMT viability: Do threads execute same instruction stream?
3. Memory access pattern: Regular (strided, coalesced) or irregular (random)?
4. Latency tolerance: Can you afford 10-100× higher single-thread latency?

MANYcore: Maximize Agregate, No branches, Yield for latency

Or think: BIG·few vs little·many

Connections

• Amdahl's Law - quantifies the serial bottleneck limiting parallel speedup
• Thread-Level Parallelism - exploited differently by multi vs manycore
• Cache Coherence - critical for multicore shared memory, less relevant for manycore
• SIMD vs MIMD - manycore uses SIMT (GPU model), multicore is MIMD
• Memory Hierarchy - multicore emphasizes caches, manycore emphasizes bandwidth
• Power Consumption - manycore wins on performance-per-watt for parallel workloads
• Heterogeneous Computing - modern systems combine both (CPU+GPU)

Multicore is like having 8 really smart teachers. Each teacher can grade ANY test super fast because they're experts. They can handle weird questions, check work, give partial credit - all the complex stuff. If you only have 8 tests, great! But if you have 1000 tests that are all the same (just multiple choice), those 8 teachers are wasted talent.

Manycore is like having 500 fifth-graders with answer keys. Each kid can ONLY check if the answer matches the key. They're not smart individually, but there are SO MANY of them. If the test is all multiple choice (simple, repetitive), they'll crush those 1000 tests way faster than the 8 teachers. But if even ONE question needs explanation, all 500 kids are stuck.

The computer chip is like your school budget - you can hire 8 expert teachers OR 500 simple checkers, not both!

#flashcards/hardware

What is the key difference between multicore and manycore processors? :: Multicore has few (2-16) complex, high-performance cores optimized for single-thread speed; manycore has many (100s-1000s) simple cores optimized for aggregate throughput.

Concept Map

Hinglish (regional understanding)