PCI Express (PCIe) architecture and generations

Overview

PCI Express (PCIe) is a high-speed serial point-to-point interconnect standard that replaced the older parallel PCI bus architecture. Unlike shared bus systems, PCIe creates dedicated lanes between devices, enabling simultaneous bidirectional communication without contention.

Core Architecture

Lane Structure

WHY differential pairs? Single-ended signals pick up electromagnetic interference (EMI). If noise hits both wires equally, the difference remains constant. The receiver amplifies the difference, canceling common-mode noise.

WHY full-duplex? Dedicated TX and RX paths mean data flows both directions simultaneously without arbitration or turn-around delays.

(Note: x16 is the widest standard link. x32 was defined on paper but never implemented in practice.)

$\text{Total Bandwidth} = \text{Lanes} \times \text{Per-Lane Rate} \times 2 \text{ (bidirectional)}$

Derivation from first principles:

• Start with ONE lane: 1 TX path + 1 RX path
• Each path runs at rate $R$ (bits/second)
• Unidirectional bandwidth per lane: $R$
• Bidirectional: $R_{\text{TX}} + R_{\text{RX}} = 2R$
• For $N$ lanes: Aggregate = $N \times 2R$

Encoding overhead: Not all bits are data. PCIe uses 8b/10b encoding (Gen 1-2) or 128b/130b encoding (Gen 3+) for clock recovery and DC balance.

Effective bandwidth for Gen 3+ with 128b/130b: $\text{Effective BW} = N \times R \times \frac{128}{130} \times 2$

Step 1: Raw signaling rate per lane

• 8 GT/s per direction per lane

Step 2: Account for 128b/130b encoding

• Efficiency = 128/130 = 0.9846

Step 3: Calculate unidirectional bandwidth per lane $BW_{\text{lane}} = 8 \text{ GT/s} \times 1 \text{ bit/transfer} \times 0.9846 = 7.877 \text{ Gb/s}$

Step 4: Scale to 16 lanes, bidirectional $BW_{\text{total}} = 16 \text{ lanes} \times 7.877 \text{ Gb/s} \times 2 = 252.4 \text{ Gb/s} = 31.5 \text{ GB/s}$

Why this step? Each lane is independent. Bidirectional means TX + RX happen simultaneously.

Packet Structure

Layered architecture (WHY three layers?):

1. Transaction Layer: Assembles packets, handles flow control, ordering
2. Data Link Layer: Adds sequence numbers and LCRC (link CRC), handles ACK/NAK retry
3. Physical Layer: Serializes packets to bits, 8b/10b encoding, differential signaling

WHY this split? Separation of concerns. Transaction layer focuses on logical transfers. Data link ensures reliable transmission over one hop. Physical layer handles electrical/optical details.

Derivation of CRC concept:

• Message $M(x)$ is treated as polynomial
• Append 32 zeros: $x^{32}M(x)$
• Divide by generator $G(x)$, remainder is CRC
• Transmitted: $T(x) = x^{32}M(x) - R(x)$ where $R(x)$ is remainder
• Receiver divides $T(x)$ by $G(x)$; zero remainder means no error

WHY 32 bits? Detects all burst errors ≤32 bits, all odd-bit errors, >99.99% of longer bursts.

Generations Evolution

Bandwidth Progression

$BW_{\text{effective}}(n, L) = L \times R_n \times E_n \times 1 \text{ byte/8 bits}$

Gen	Year	Rate (GT/s)	Encoding	Efficiency	x1 BW	x16 BW
1	2003	2.5	8b/10b	80%	250 MB/s	4 GB/s
2	2007	5.0	8b/10b	80%	500 MB/s	8 GB/s
3	2010	8.0	128b/130b	98.46%	985 MB/s	15.75 GB/s
4	2017	16.0	128b/130b	98.46%	1.97 GB/s	31.5 GB/s
5	2019	32.0	128b/130b	98.46%	3.94 GB/s	63 GB/s
6	2022	64.0	FLIT (242B/256B)	94.53%	15.13 GB/s	242 GB/s

(All bandwidths are per direction / unidirectional. PCIe is full-duplex, so bidirectional aggregate is double.)

Gen 3 solution (128b/130b):

• Every 128 data bits get 2 sync bits
• Overhead drops to 1.54%

Calculation (two DIFFERENT ratios — do not confuse them): $\text{Overhead}_{\text{8b/10b}} = \frac{10-8}{10} = 20\%$ $\text{Overhead}_{\text{128b/130b}} = \frac{130-128}{130} = 1.54\%$

Waste reduction (how much wasted bandwidth shrinks): $\frac{20\%}{1.54\%} \approx 13× \text{ less waste}$

Usable-efficiency improvement (what actually reaches the data): $\frac{98.46\%}{80\%} \approx 1.23× \text{ more usable bandwidth}$

WHY these differ: Overhead going from 20% → 1.54% is a ~13× reduction in the wasted fraction, but since 80% was already usable, the usable throughput only rises to 98.46% — about 1.23×. Beginners quote "13×" as a bandwidth boost; that is wrong. The real per-lane data gain from the encoding change alone is ~1.23×.

Why not 128b/130b from the start? Early generations (2003-2007) ran at lower speeds where 8b/10b transitions provided robust clock recovery. The extra complexity of 128b/130b wasn't needed until speeds reached 8 GT/s.

Key Generational Improvements

Gen 1→2: Doubled signaling rate (2.5→5.0 GT/s) using same 8b/10b encoding. Required better signal integrity and equalization.

Gen 2→3: New encoding (128b/130b), scrambling for EMI reduction, improved equalization.

Gen 3→4: Doubled rate again (8→16 GT/s), tighter channel/loss budgets and more aggressive equalization. (No FEC yet — FEC arrives in Gen 6.)

Gen 4→5: Doubled to 32 GT/s, required even more sophisticated equalization and crosstalk cancellation.

Gen 5→6: Introduced PAM-4 signaling (4 voltage levels instead of 2), doubled effective rate. Switched to FLIT mode with embedded CRC in 256-byte blocks, and — because PAM-4's tight voltage margins are error-prone — added mandatory forward error correction (FEC) for the first time.

Bits per symbol: $b = \log_2(4) = 2 \text{ bits/symbol}$

WHY PAM-4? Doubling the baud rate (symbols/second) beyond 32 GHz becomes impractical due to channel loss and reflections. PAM-4 doubles bits/symbol at the same baud rate.

Tradeoff: Each voltage level is closer together, requiring higher signal-to-noise ratio (SNR). More sensitive to interference — which is exactly why Gen 6 introduces FEC.

Derivation of SNR requirement:

• NRZ (2-level): Voltage separation = $2V$
• PAM-4 (4-level): Voltage separation = $\frac{2V}{3}$
• SNR penalty: $20\log_{10}(3) \approx 9.5 \text{ dB}$ worse

Requires better equalization, FEC, and shielding.

Step 1: PAM-4 encoding — 64 GT/s already counts transfers; each transfer carries the raw bit stream.

• Raw per-lane rate = 64 GT/s × 1 bit-equivalent per transfer = 64 Gb/s useful-line rate basis? No — expand it properly:
• 64 GT/s × 2 bits/symbol interpretation is folded into the "64 GT/s" figure already. Effective raw line data = 64 Gb/s per direction before FLIT overhead? Use the standard PCIe convention: 64 GT/s ≈ 64 Gb/s raw per lane per direction already accounting for PAM-4 in the GT/s number.

Cleaner standard convention used by PCI-SIG:

• Raw per-lane per-direction = 64 Gb/s
• Byte rate before overhead = 64 / 8 = 8 GB/s ... but this ignores PAM-4 doubling.

Correct PCI-SIG accounting:

• Gen 6 delivers ≈ 8 GB/s per lane per direction of raw bytes ONLY if 64 GT/s = 64 Gb/s. But PCI-SIG quotes ≈ 8 GB/s/lane raw × 2 (PAM-4) → the published number is ≈ 15.13 GB/s per lane per direction after FLIT efficiency.

Use the published, self-consistent figures:

Step 2: FLIT overhead

• Efficiency = 242/256 = 0.9453

Step 3: Per-lane effective bandwidth (per direction) $BW_{\text{lane}} = 16 \text{ GB/s (raw byte rate)} \times 0.9453 = 15.13 \text{ GB/s}$

Step 4: Scale to x16 (per direction) $BW_{\text{total}} = 16 \times 15.13 \text{ GB/s} = 242 \text{ GB/s per direction}$

Marketing figure: PCI-SIG cites ≈ 256 GB/s per direction as the raw (pre-FLIT-overhead) x16 number, and ≈ 242 GB/s as the effective payload throughput. Bidirectional aggregate ≈ 484 GB/s.

Why FLIT mode? At 64 GT/s, traditional packet boundaries cause inefficiency. FLIT (Flow Control Unit) mode uses fixed 256-byte blocks with embedded CRC, simplifying hardware and reducing latency.

Physical Implementation

Connector and Slot Configuration

x1 slot: 1 lane = 2 differential pairs (1 TX pair + 1 RX pair) plus power/ground. x16 slot: 16 lanes = 32 differential pairs (16 TX + 16 RX) plus power/ground.

WHY backward compatible? An x1 card physically fits in x16 slot. PCIe training negotiates lane count. The protocol links must agree on width.

Training sequence bit pattern (TS1):

• COM symbol (K28.5): 8b/10b control code
• Lane number
• Link/data rate identifiers
• Repeated for lane alignment

WHY auto-negotiation? x4 card in x16 slot uses 4 lanes. Gen 5 device in Gen 3 slot runs at Gen 3 speed. Maximizes compatibility without manual configuration.

Switch Fabric Architecture

Root Complex (RC): CPU-attached PCIe controller, top of tree Endpoint (EP): Final device (GPU, NIC, SSD) Switch: Multiplexes multiple endpoints onto fewer upstream lanes

        [Root Complex]
              |
        [PCIe Switch]
        /     |     \
     [GPU]  [NIC]  [SSD]

WHY switches? CPU provides limited PCIe lanes (e.g., 16-24). Switch allows 48+ endpoints. Electrical fanout is impossible; switches provide packet routing.

Packet routing: Switch examines TLP header (bus/device/function address or memory address), routes to correct downstream port. No shared bus contention.

Question: Can all four ports run at full x4 Gen 4 bandwidth simultaneously?

Analysis:

• Each x4 Gen 4 port: 7.88 GB/s (one direction)
• Four ports × 7.88 GB/s = 31.5 GB/s total downstream→upstream
• x16 Gen 4 upstream: 31.5 GB/s
• Result: Yes, IF all traffic goes upstream. If mixed, depends on traffic pattern.

Why this matters? Switch is non-blocking for balanced bidirectional traffic but becomes a bottleneck if all downstream devices simultaneously push to upstream.

Real scenario: GPU (x16 Gen 4) + NVMe (x4 Gen 4) on x16 Gen 3 upstream (15.75 GB/s):

• GPU saturates uplink → NVMe starved
• Switch uses traffic prioritization and virtual channels to allocate bandwidth

Advanced Features

Quality of Service (QoS)

Flow control mechanism:

1. Receiver advertises buffer credits (available space)
2. Transmitter deducts credits when sending
3. Receiver returns credits as it processes packets
4. Transmitter stalls when credits exhausted

Derivation of credit-based flow control:

• Problem: Receiver buffer overflow if sender too fast
• Solution: Sender knows receiver capacity
• Math: $\text{Sender\_window} \leq \text{Receiver\_buffer}$
• Credits allow pipelining without ACKs for every packet

WHY virtual channels? Low-latency traffic (audio) gets separate VC from bulk transfers (storage). Prevents head-of-line blocking.

$\text{Next\_VC} = \arg\max_{i} \{P_i \mid Q_i > 0\}$

Strict priority: highest-priority non-empty VC always wins. Optional weighted round-robin within same priority.

Power Management

ASPM (Active State Power Management): Automatically enters L0s/L1 during idle. Controlled by BIOS/OS policies.

WHY L0s? Individual lanes enter low-power when idle. x16 link with x4 active traffic powers down 12 lanes. Microsecond wake-up prevents latency penalty.

Mistake 1: "PCIe is just faster PCI" Why it feels right: Same name, both connect expansion cards. Steel-man: PCI provided shared parallel bus that many designs used. PCIe maintains software compatibility (config space) while changing everything electrically. The fix: PCIe is fundamentally different—serial point-to-point vs. parallel shared bus. Protocols, signaling, topology all changed. Only configuration space software interface remains compatible.

Mistake 2: "x16 slot always provides x16 bandwidth" Why it feels right: Slot is physically x16 size. Steel-man: Many motherboards wire x16 slots with only x8 or x4 lanes (especially 2nd/3rd slots) to save cost and PCIe lane budget. The fix: Check lane population. An x16 slot might be electrically x4 (common for NVMe adapters). Run at actual wired lane count, not slot size.

Mistake 3: "Higher gen is always better" Why it feels right: Gen 4 > Gen 3 in spec. Steel-man: Higher gens require better signal integrity. Long cables, riser cards, or noisy environments may cause errors at higher speeds. The fix: Gen 4 at x4 might be less reliable than Gen 3 at x8 in marginal conditions. Bandwidth = lanes × rate × reliability. Sometimes more lanes at lower gen wins.

Mistake 4: "128b/130b gives ~13× more bandwidth than 8b/10b" Why it feels right: Overhead drops from 20% to 1.54%, and 20/1.54 ≈ 13. Steel-man: The wasted fraction really does shrink ~13×, so the improvement in "how much you throw away" is genuinely large. The fix: Usable efficiency only rises from 80% to 98.46%, a 1.23× gain in actual data throughput. The 13× applies to overhead reduction, not delivered bandwidth. Most of Gen 3's speed came from the clock-rate jump (5→8 GT/s), not the encoding.

Mistake 5: "PCIe 6.0 doubles bandwidth again" Why it feels right: Every generation doubled before. Steel-man: PAM-4 doubles bits per symbol at the same baud rate, so on paper the raw rate doubles (32→64 GT/s effective). The fix: Raw does roughly double, but FLIT overhead (242/256 = 94.5%) is heavier than 128b/130b (98.46%), so effective payload throughput scales slightly under 2×. Also PAM-4 forces mandatory FEC and stricter signal-integrity requirements.

Gen 1: 2.5 GT/s (2003) Gen 2: 5.0 GT/s (double, 2007) Gen 3: 8.0 GT/s (not quite double

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, PCIe ka core idea bahut simple hai. Purane parallel bus mein 32 wires ek saath data bhejte the, but sabko ek hi speed pe synchronized rakhna padta tha. Jaise ek chaudi highway jahan sab cars ko exactly same speed pe chalna zaruri hai. Jab speed badhao, toh 32 wires ke beech timing skew aur electromagnetic interference itna problem create karta hai ki system kaam hi nahi karta. Isliye PCIe ne ulta rasta liya — ek serial lane, but bahut high speed pe. Ek lane mein do differential pairs hote hain: ek TX (transmit) ke liye, ek RX (receive) ke liye, matlab data dono directions mein simultaneously flow karta hai (full-duplex). Aur differential signaling ka magic ye hai ki agar noise dono wires pe barabar aata hai, toh unka difference constant rehta hai, so receiver clean signal nikaal leta hai.

Ab bandwidth ki baat karein toh formula seedha hai: Total BW = Lanes × Per-lane rate × 2 (kyunki bidirectional). Lekin ek catch hai — saare bits pure data nahi hote. PCIe ko clock recovery aur DC balance ke liye kuch extra bits chahiye, isliye encoding overhead lagta hai (Gen 3+ mein 128b/130b, matlab har 130 bits mein sirf 128 useful). Isiliye x16 Gen 3 ka effective bandwidth 31.5 GB/s aata hai, na ki theoretical maximum. Ye samajhna important hai kyunki real-world performance hamesha raw numbers se thoda kam hota hai, aur exams mein ye distinction pucha jaata hai.

Ye topic kyun matter karta hai? Aaj kal graphics cards, SSDs, network cards — sab PCIe pe hi chalte hain, so ye modern computer ka backbone hai. Aur PCIe ka layered design (Transaction, Data Link, Physical layer) tumhe ek beautiful engineering principle sikhata hai — "separation of concerns." Har layer apna kaam karti hai: upar wali logical transfers sambhalti hai, beech wali reliability (ACK/NAK retry, CRC check) dekhti hai, aur neeche wali electrical signals handle karti hai. Ye same concept networking (OSI model) aur bahut saare systems mein repeat hota hai, toh yahan samajh loge toh aage kaafi cheezein easy lagengi.