PCI Express (PCIe) architecture and generations

6.3.2 · Hardware › Interconnects, Buses & SoC

Overview

PCI Express (PCIe) ek high-speed serial point-to-point interconnect standard hai jo purane parallel PCI bus architecture ko replace kar chuka hai. Shared bus systems ke unlike, PCIe devices ke beech dedicated lanes create karta hai, jisse bina kisi contention ke simultaneous bidirectional communication possible hoti hai.

Core Architecture

Lane Structure

Differential pairs kyun? Single-ended signals electromagnetic interference (EMI) pick up karte hain. Agar noise dono wires pe equally hit kare, toh difference constant rehta hai. Receiver sirf difference ko amplify karta hai, jisse common-mode noise cancel ho jaata hai.

Full-duplex kyun? Dedicated TX aur RX paths ka matlab hai ki data bina kisi arbitration ya turn-around delays ke dono directions mein simultaneously flow karta hai.

(Note: x16 sabse wide standard link hai. x32 paper pe define hua tha lekin practice mein kabhi implement nahi hua.)

$Total Bandwidth = Lanes \times Per-Lane Rate \times 2 (bidirectional)$

First principles se derivation:

EK lane se shuru karo: 1 TX path + 1 RX path
Har path rate $R$ (bits/second) pe chalta hai
Unidirectional bandwidth per lane: $R$
Bidirectional: $R_{TX} + R_{RX} = 2 R$
$N$ lanes ke liye: Aggregate = $N \times 2 R$

Encoding overhead: Saare bits data nahi hote. PCIe clock recovery aur DC balance ke liye 8b/10b encoding (Gen 1-2) ya 128b/130b encoding (Gen 3+) use karta hai.

Gen 3+ ke liye 128b/130b ke saath effective bandwidth: $Effective BW = N \times R \times \frac{128}{130} \times 2$

Step 1: Per lane raw signaling rate

8 GT/s per direction per lane

Step 2: 128b/130b encoding account karo

Efficiency = 128/130 = 0.9846

Step 3: Per lane unidirectional bandwidth calculate karo $B W_{lane} = 8 GT/s \times 1 bit/transfer \times 0.9846 = 7.877 Gb/s$

Step 4: 16 lanes tak scale karo, bidirectional $B W_{total} = 16 lanes \times 7.877 Gb/s \times 2 = 252.4 Gb/s = 31.5 GB/s$

Ye step kyun? Har lane independent hai. Bidirectional ka matlab hai TX + RX simultaneously hote hain.

Packet Structure

Layered architecture (teen layers kyun?):

Transaction Layer: Packets assemble karta hai, flow control aur ordering handle karta hai
Data Link Layer: Sequence numbers aur LCRC (link CRC) add karta hai, ACK/NAK retry handle karta hai
Physical Layer: Packets ko bits mein serialize karta hai, 8b/10b encoding aur differential signaling

Ye split kyun? Separation of concerns. Transaction layer logical transfers pe focus karta hai. Data link ek hop ke upar reliable transmission ensure karta hai. Physical layer electrical/optical details handle karta hai.

CRC concept ki derivation:

Message $M (x)$ ko polynomial maana jaata hai
32 zeros append karo: $x^{32} M (x)$
Generator $G (x)$ se divide karo, remainder CRC hai
Transmitted: $T (x) = x^{32} M (x) - R (x)$ jahan $R (x)$ remainder hai
Receiver $T (x)$ ko $G (x)$ se divide karta hai; zero remainder matlab koi error nahi

32 bits kyun? Sabhi burst errors ≤32 bits detect karta hai, sabhi odd-bit errors detect karta hai, aur longer bursts ka >99.99% detect karta hai.

Generations Evolution

Bandwidth Progression

$B W_{effective} (n, L) = L \times R_{n} \times E_{n} \times 1 byte/8 bits$

Gen	Year	Rate (GT/s)	Encoding	Efficiency	x1 BW	x16 BW
1	2003	2.5	8b/10b	80%	250 MB/s	4 GB/s
2	2007	5.0	8b/10b	80%	500 MB/s	8 GB/s
3	2010	8.0	128b/130b	98.46%	985 MB/s	15.75 GB/s
4	2017	16.0	128b/130b	98.46%	1.97 GB/s	31.5 GB/s
5	2019	32.0	128b/130b	98.46%	3.94 GB/s	63 GB/s
6	2022	64.0	FLIT (242B/256B)	94.53%	15.13 GB/s	242 GB/s

(Saari bandwidths per direction / unidirectional hain. PCIe full-duplex hai, isliye bidirectional aggregate double hoga.)

Gen 3 solution (128b/130b):

Har 128 data bits ko 2 sync bits milte hain
Overhead gir ke 1.54% ho jaata hai

Calculation (do ALAG ratios — inhe confuse mat karo): $Overhead_{8b/10b} = \frac{10 - 8}{10} = 20%$ $Overhead_{128b/130b} = \frac{130 - 128}{130} = 1.54%$

Waste reduction (kitna wasted bandwidth shrink hota hai): $\frac{20%}{1.54%} \approx 13 \times less waste$

Usable-efficiency improvement (actually data tak kya pohonchta hai): $\frac{98.46%}{80%} \approx 1.23 \times more usable bandwidth$

Ye alag kyun hain: Overhead 20% → 1.54% jaana wasted fraction mein ~13× reduction hai, lekin kyunki 80% already usable tha, usable throughput sirf 98.46% tak badhta hai — lagbhag 1.23×. Beginners "13×" ko bandwidth boost bolte hain; ye galat hai. Sirf encoding change se real per-lane data gain ~1.23× hai.

Shuru se 128b/130b kyun nahi? Early generations (2003-2007) lower speeds pe chalte the jahan 8b/10b transitions robust clock recovery provide karte the. Jab tak speeds 8 GT/s tak nahi pahunchi, 128b/130b ki extra complexity ki zaroorat nahi thi.

Key Generational Improvements

Gen 1→2: Signaling rate double hua (2.5→5.0 GT/s) same 8b/10b encoding ke saath. Better signal integrity aur equalization ki zaroorat padi.

Gen 2→3: Naya encoding (128b/130b), EMI reduction ke liye scrambling, improved equalization.

Gen 3→4: Rate phir double hua (8→16 GT/s), tighter channel/loss budgets aur zyada aggressive equalization. (Abhi FEC nahi — FEC Gen 6 mein aata hai.)

Gen 4→5: 32 GT/s tak double hua, aur bhi sophisticated equalization aur crosstalk cancellation ki zaroorat padi.

Gen 5→6: PAM-4 signaling introduce ki (2 ki jagah 4 voltage levels), effective rate double hua. FLIT mode mein switch kiya jisme 256-byte blocks mein embedded CRC hai, aur — kyunki PAM-4 ke tight voltage margins error-prone hain — pehli baar mandatory forward error correction (FEC) add ki gayi.

Bits per symbol: $b = lo g_{2} (4) = 2 bits/symbol$

PAM-4 kyun? Baud rate (symbols/second) ko 32 GHz se aage double karna channel loss aur reflections ki wajah se impractical ho jaata hai. PAM-4 same baud rate pe bits/symbol double karta hai.

Tradeoff: Har voltage level ek doosre ke kareeb hota hai, jiske liye higher signal-to-noise ratio (SNR) chahiye. Interference ke liye zyada sensitive — aur exactly isliye Gen 6 mein FEC introduce hota hai.

SNR requirement ki derivation:

NRZ (2-level): Voltage separation = $2 V$
PAM-4 (4-level): Voltage separation = $\frac{2 V}{3}$
SNR penalty: $20 lo g_{10} (3) \approx 9.5 dB$ worse

Better equalization, FEC, aur shielding ki zaroorat hai.

Step 1: PAM-4 encoding — 64 GT/s already transfers count karta hai; har transfer raw bit stream carry karta hai.

Raw per-lane rate = 64 GT/s × 1 bit-equivalent per transfer = 64 Gb/s useful-line rate basis? Nahi — ise properly expand karo:
64 GT/s × 2 bits/symbol interpretation already "64 GT/s" figure mein fold ho chuki hai. Effective raw line data = 64 Gb/s per direction before FLIT overhead? Standard PCIe convention use karo: 64 GT/s ≈ 64 Gb/s raw per lane per direction already PAM-4 ko GT/s number mein account karte hue.

PCI-SIG ka cleaner standard convention:

Raw per-lane per-direction = 64 Gb/s
Overhead se pehle byte rate = 64 / 8 = 8 GB/s ... lekin ye PAM-4 doubling ignore karta hai.

Sahi PCI-SIG accounting:

Gen 6 ≈ 8 GB/s per lane per direction raw bytes deliver karta hai SIRF AGAR 64 GT/s = 64 Gb/s. Lekin PCI-SIG ≈ 8 GB/s/lane raw × 2 (PAM-4) quote karta hai → published number ≈ 15.13 GB/s per lane per direction hai FLIT efficiency ke baad.

Published, self-consistent figures use karo:

Step 2: FLIT overhead

Efficiency = 242/256 = 0.9453

Step 3: Per-lane effective bandwidth (per direction) $B W_{lane} = 16 GB/s (raw byte rate) \times 0.9453 = 15.13 GB/s$

Step 4: x16 tak scale karo (per direction) $B W_{total} = 16 \times 15.13 GB/s = 242 GB/s per direction$

Marketing figure: PCI-SIG ≈ 256 GB/s per direction raw (pre-FLIT-overhead) x16 number cite karta hai, aur ≈ 242 GB/s effective payload throughput ke roop mein. Bidirectional aggregate ≈ 484 GB/s.

FLIT mode kyun? 64 GT/s pe, traditional packet boundaries inefficiency cause karte hain. FLIT (Flow Control Unit) mode fixed 256-byte blocks with embedded CRC use karta hai, jo hardware simplify karta hai aur latency reduce karta hai.

Physical Implementation

Connector and Slot Configuration

x1 slot: 1 lane = 2 differential pairs (1 TX pair + 1 RX pair) plus power/ground. x16 slot: 16 lanes = 32 differential pairs (16 TX + 16 RX) plus power/ground.

Backward compatible kyun? Ek x1 card physically x16 slot mein fit ho jaata hai. PCIe training lane count negotiate karta hai. Protocol links ko width pe agree karna hota hai.

Training sequence bit pattern (TS1):

COM symbol (K28.5): 8b/10b control code
Lane number
Link/data rate identifiers
Lane alignment ke liye repeat kiya jaata hai

Auto-negotiation kyun? x16 slot mein x4 card 4 lanes use karta hai. Gen 3 slot mein Gen 5 device Gen 3 speed pe chalta hai. Bina manual configuration ke compatibility maximize hoti hai.

Switch Fabric Architecture

Root Complex (RC): CPU-attached PCIe controller, tree ka top Endpoint (EP): Final device (GPU, NIC, SSD) Switch: Multiple endpoints ko kam upstream lanes pe multiplex karta hai

        [Root Complex]
              |
        [PCIe Switch]
        /     |     \
     [GPU]  [NIC]  [SSD]

Switches kyun? CPU limited PCIe lanes provide karta hai (jaise 16-24). Switch 48+ endpoints allow karta hai. Electrical fanout impossible hai; switches packet routing provide karte hain.

Packet routing: Switch TLP header (bus/device/function address ya memory address) examine karta hai, correct downstream port ko route karta hai. Koi shared bus contention nahi.

Question: Kya saare chaar ports ek saath full x4 Gen 4 bandwidth pe chal sakte hain?

Analysis:

Har x4 Gen 4 port: 7.88 GB/s (ek direction)
Chaar ports × 7.88 GB/s = 31.5 GB/s total downstream→upstream
x16 Gen 4 upstream: 31.5 GB/s
Result: Haan, AGAR saara traffic upstream jaata hai. Mixed traffic mein, ye traffic pattern pe depend karta hai.

Ye kyun important hai? Switch balanced bidirectional traffic ke liye non-blocking hai lekin bottleneck ban jaata hai agar saare downstream devices simultaneously upstream push karein.

Real scenario: GPU (x16 Gen 4) + NVMe (x4 Gen 4) on x16 Gen 3 upstream (15.75 GB/s):

GPU uplink saturate karta hai → NVMe starved ho jaata hai
Switch bandwidth allocate karne ke liye traffic prioritization aur virtual channels use karta hai

Advanced Features

Quality of Service (QoS)

Flow control mechanism:

Receiver buffer credits (available space) advertise karta hai
Transmitter send karte waqt credits deduct karta hai
Receiver packets process karte waqt credits return karta hai
Jab credits exhaust ho jaate hain, transmitter stall ho jaata hai

Credit-based flow control ki derivation:

Problem: Agar sender bahut fast ho toh receiver buffer overflow
Solution: Sender receiver capacity jaanta hai
Math: $Sender_window \leq Receiver_buffer$
Credits har packet ke liye ACKs ke bina pipelining allow karte hain

Virtual channels kyun? Low-latency traffic (audio) bulk transfers (storage) se alag VC pe hota hai. Head-of-line blocking prevent hoti hai.

$Next_VC = ar g max_{i} {P_{i} ∣ Q_{i} > 0}$

Strict priority: highest-priority non-empty VC hamesha win karta hai. Same priority ke andar optional weighted round-robin.

Power Management

ASPM (Active State Power Management): Idle ke dauran automatically L0s/L1 enter karta hai. BIOS/OS policies se control hota hai.

L0s kyun? Individual lanes idle hone pe low-power enter karte hain. x4 active traffic wale x16 link mein 12 lanes power down ho jaate hain. Microsecond wake-up latency penalty prevent karta hai.

Mistake 1: "PCIe bas faster PCI hai" Ye sahi kyun lagta hai: Same naam, dono expansion cards connect karte hain. Steel-man: PCI ne shared parallel bus provide kiya tha jise bahut saare designs use karte the. PCIe software compatibility (config space) maintain karta hai jabki electrically sab kuch badal deta hai. Fix: PCIe fundamentally different hai — serial point-to-point vs. parallel shared bus. Protocols, signaling, topology sab badal gaye. Sirf configuration space software interface compatible raha.

Mistake 2: "x16 slot hamesha x16 bandwidth provide karta hai" Ye sahi kyun lagta hai: Slot physically x16 size ka hai. Steel-man: Bahut saare motherboards x16 slots ko sirf x8 ya x4 lanes ke saath wire karte hain (especially 2nd/3rd slots) cost aur PCIe lane budget bachane ke liye. Fix: Lane population check karo. Ek x16 slot electrically x4 ho sakta hai (NVMe adapters ke liye common). Actual wired lane count pe chalta hai, slot size pe nahi.

Mistake 3: "Higher gen hamesha better hai" Ye sahi kyun lagta hai: Spec mein Gen 4 > Gen 3. Steel-man: Higher gens ko better signal integrity chahiye. Long cables, riser cards, ya noisy environments higher speeds pe errors cause kar sakte hain. Fix: Marginal conditions mein x4 pe Gen 4, x8 pe Gen 3 se kam reliable ho sakta hai. Bandwidth = lanes × rate × reliability. Kabhi-kabhi lower gen pe zyada lanes better hota hai.

Mistake 4: "128b/130b 8b/10b se ~13× zyada bandwidth deta hai" Ye sahi kyun lagta hai: Overhead 20% se 1.54% tak girta hai, aur 20/1.54 ≈ 13. Steel-man: Wasted fraction sach mein ~13× shrink hoti hai, isliye "kitna throw away karte ho" wali improvement genuinely badi hai. Fix: Usable efficiency sirf 80% se 98.46% tak badhti hai — actual data throughput mein 1.23× gain. 13× overhead reduction pe apply hota hai, delivered bandwidth pe nahi. Gen 3 ki zyaatar speed clock-rate jump (5→8 GT/s) se aayi, encoding se nahi.

Mistake 5: "PCIe 6.0 bandwidth phir double karta hai" Ye sahi kyun lagta hai: Pehle har generation double hoti thi. Steel-man: PAM-4 same baud rate pe bits per symbol double karta hai, isliye paper pe raw rate double hota hai (32→64 GT/s effective). Fix: Raw roughly double hota hai, lekin FLIT overhead (242/256 = 94.5%) 128b/130b (98.46%) se heavier hai, isliye effective payload throughput slightly 2× se kam scale karta hai. Saath hi PAM-4 mandatory FEC aur stricter signal-integrity requirements force karta hai.

Gen 1: 2.5 GT/s (2003) Gen 2: 5.0 GT/s (double, 2007) Gen 3: 8.0 GT/s (poora double nahi