Interconnects, Buses & SoC

Question 1

(a) Effective bandwidth (5)

• Raw rate per lane: $32\,\text{GT/s} = 32\times10^9$ transfers/s, 1 bit/transfer. (1)
• 128b/130b overhead factor $= 128/130 = 0.98462$. (1)
• Usable bits/lane/s $= 32\times10^9 \times \tfrac{128}{130} = 31.5077\times10^9$ bit/s. (1)
• ×16 lanes, ÷8 bits/byte: $\frac{32\times10^9 \times \frac{128}{130}\times 16}{8}$. (1)
• $= 63.015\times10^9$ B/s $\approx \mathbf{63.0\ GB/s}$ per direction. (1)

(b) Improvement factor (4)

• Gen3 effective/lane $= 8\times10^9 \times \tfrac{8}{10}/8 = 0.8$ GB/s. (1)
• Gen5 effective/lane $= 32\times10^9 \times \tfrac{128}{130}/8 = 3.938$ GB/s. (1)
• Ratio $= 3.938/0.8 = 4.923$. (1)
• Not 4 because Gen3 wastes 20% (8b/10b) while Gen5 wastes only ~1.5% (128b/130b); the encoding efficiency also improved, multiplying the raw 4× by $\frac{0.9846}{0.8}=1.23$. (1)

(c) Serial vs parallel (6)

• Reason 1: No inter-lane skew constraint — parallel bus requires all bits arrive within one UI; at multi-GHz the trace-length matching becomes physically impossible. Serial embeds clock in data (CDR). (1.5)
• Reason 2: Reduced crosstalk / EMI and fewer pins; a single differential pair has controlled impedance and better SI than 32 aggressor lines. (Also lower simultaneous-switching noise.) (1.5)
• Quantification: 32-bit word per 1 ns ⇒ per-bit UI on the parallel bus $=1\text{ns}$ (all bits parallel, one word/ns). $\pm5\%$ of UI $=0.05\times1\,\text{ns}=50\,\text{ps}$. Max skew $=\pm50$ ps. (3)

(d) CXL latency (5)

• CXL.mem/.cache use load/store (coherent) semantics rather than descriptor-based DMA. (1.5) No descriptor fetch/ring-buffer/doorbell round-trips. (1.5) The host memory controller / coherence engine handles the access as a native cacheable transaction, so latency approaches that of a memory access rather than an I/O transaction; and cache coherence avoids explicit software-managed copies. (2)

Question 2

(a) Topology metrics (8)

• Mesh $8\times8$: links $= 2\cdot n\cdot(n-1)$ per dimension summed $= 2\times8\times7 = 112$ bidirectional links. (2) Diameter $=(n-1)+(n-1)=7+7=\mathbf{14}$ hops. (1) Bisection: cut the $8\times8$ into two halves ⇒ $8$ links crossing (one column), each bidir ⇒ bisection BW $=8B$. (1)
• Crossbar (64-port): internally a full $64\times64$ switch ⇒ crosspoints $=64^2=4096$; "links" = 64 ports; effectively any-to-any. (2) Diameter $=1$ hop. (1) Bisection $=32B$ (32 ports on each side can all communicate) — full non-blocking. (1)

(b) Scaling proof (6)

• Average Manhattan distance in $n\times n$ mesh: mean of $|x_1-x_2|$ over $n$ points is $\frac{n^2-1}{3n}\approx n/3$; two dimensions ⇒ avg hops $\approx \frac{2(n^2-1)}{3n}=\Theta(n)$. (2)
• Links $=2n(n-1)=\Theta(n^2)$ (nodes $N=n^2$, so links $=\Theta(N)$ — linear in node count). (2)
• Crossbar: switch cost $\Theta(N^2)$ crosspoints, diameter $\Theta(1)$, bisection $\Theta(N)$. Trade-off: mesh has cheap $\Theta(N)$ cost but $\Theta(\sqrt N)$ latency and $\Theta(\sqrt N)$ bisection; crossbar gives constant latency & full bisection but $\Theta(N^2)$ area — infeasible at large $N$. (2)

(c) Arbitration (8)

• Round-robin, last granted M3 ⇒ priority order restarts after M3: M0,M1,M2,M3. Grant lowest-index requesting master after the last granted, cyclically. Grant sequence (one per cycle):

Grant sequence: M0, M1, M2, M1, M2, M3, M0, M1. (4)

• Worst-case wait, round-robin: a continuously-requesting master waits at most $N-1$ cycles (all others served once). (1.5)
• Fixed priority: a low-priority master can wait unbounded (∞) if higher-priority masters continuously request → starvation. (1)
• Starvation-free proof: after a master is passed over, the pointer advances past it only when it is granted or not requesting; a continuously-requesting master's position becomes highest-priority within $\le N-1$ grants, guaranteeing service — bounded wait ⇒ no starvation. (1.5)

Question 3

(a) Effective bandwidth model (7)

• Time per chunk $= t_o + C/B$ (overhead + payload transfer). (1)
• Effective BW $= \dfrac{C}{t_o + C/B}$. (2) $B_{\text{eff}}(C) = \frac{C}{t_o + C/B} = \frac{B}{1 + \frac{B\,t_o}{C}}.$ (1)
• Limit: as $C\to\infty$, $\frac{B t_o}{C}\to0$ ⇒ $B_{\text{eff}}\to B$. ✓ (1)
• Numeric: $B t_o = 16\times10^9 \times 200\times10^{-9} = 3200$ B. $B_{\text{eff}}=\frac{16\times10^9}{1+3200/4096}=\frac{16\times10^9}{1.78125}= \mathbf{8.982\times10^9\ B/s}\approx 8.98$ GB/s. (2)

(b) 90% target (5)

• Set $B_{\text{eff}}=0.9B$: $\frac{B}{1+Bt_o/C}=0.9B \Rightarrow 1+\frac{Bt_o}{C}=\frac{1}{0.9}$. (2)
• $\frac{Bt_o}{C}=\frac{1}{0.9}-1=\frac{1}{9}\Rightarrow C^\*=9\,B\,t_o$. (1)
• General: $C^\* = \dfrac{f}{1-f}\,B\,t_o$ for fraction $f$. (1)
• Numeric: $C^\* = 9\times3200 = \mathbf{28800\ B}$. (1)

(c) Code + overlap discussion (6)

import math
def plan_dma(S, C, to, B):
    n = math.ceil(S / C)
    payload_time = S / B                 # total bytes / bandwidth
    # non-last overheads overlap payload; only last overhead is exposed
    exposed_overhead = to                # one un-overlapped overhead
    total_time = payload_time + exposed_overhead
    beff = S / total_time
    return total_time, beff

(3)

• Discussion: In the non-overlapped model total time $= n(t_o + C/B)$, so overhead is paid $n$ times and $B_{\text{eff}}=\frac{B}{1+Bt_o/C}$. With overlap (descriptor prefetch while previous payload streams), only one overhead is exposed, giving $B_{\text{eff}}=\frac{S}{S/B + t_o}\to B$ much faster — near-full bandwidth even for small $C$, provided $t_o \le C/B$ (else overhead cannot be fully hidden). (3)

[
  {"claim":"Gen5 x16 effective BW ~63.0 GB/s","code":"beff=32e9*(128/130)*16/8; result = abs(beff-63.0e9) < 0.1e9"},
  {"claim":"Gen3->Gen5 per-lane improvement ~4.923x","code":"g3=8e9*(8/10)/8; g5=32e9*(128/130)/8; result = abs(g5/g3-4.923) < 0.01"},
  {"claim":"Parallel bus max skew is 50 ps for 1ns UI at 5%","code":"skew=0.05*1e-9; result = abs(skew-50e-12) < 1e-15"},
  {"claim":"B_eff at C=4096 is ~8.982 GB/s","code":"B=16e9; to=200e-9; C=4096; beff=B/(1+B*to/C); result = abs(beff-8.982e9) < 0.01e9"},
  {"claim":"C* for 90% is 28800 bytes","code":"B=16e9; to=200e-9; Cstar=9*B*to; result = abs(Cstar-28800) < 1e-6"},
  {"claim":"8x8 mesh diameter is 14","code":"n=8; result = (n-1)+(n-1)==14"},
  {"claim":"8x8 mesh bidirectional link count is 112","code":"n=8; result = 2*n*(n-1)==112"}
]