4.4.27 · D5Databases

Question bank — Distributed databases — sharding strategies, replication

1,959 words9 min readBack to topic

Symbols and terms used on this page (read first)

Everything below is used inside the questions. Nothing here is assumed — build these pictures once and the traps become easy.


True or false — justify

Sharding and replication solve the same problem.
False — sharding splits distinct data for scale, while replication copies the same data for survival; they are orthogonal and real systems use both. See Horizontal vs Vertical Partitioning.
Every row in a sharded table lives on exactly one node.
False as stated — a row has exactly one primary shard, but replication means copies of that row can exist on several nodes; "one shard owns it" ≠ "one physical machine has it."
Hash sharding eliminates hotspots for all query patterns.
False — it spreads point writes evenly, but range queries scatter across every shard, so a range-heavy workload still suffers even without a write hotspot.
Consistent hashing removes the need to move any data when you add a node.
False — it still moves about of keys (the new node's arc, i.e. its slice of the ring); it just avoids the near-total reshuffle of naive mod N. See Consistent Hashing.
With you are guaranteed to never read stale data.
Mostly true for synchronous quorum writes — the read set is forced to overlap the latest write set — but with sloppy quorums or in-flight concurrent writes you can still read an old-but-valid version; the guarantee is "you see a value at least as new as the last completed write."
Adding more replicas always makes the system safer.
False — raising forces you to grow and to keep , adding latency; safety is governed by the quorum relation, not raw replica count. See Quorum Consensus and Paxos/Raft.
Synchronous replication guarantees zero data loss on failover.
True for the acked writes — the leader waits for a follower's confirm before it acks (tells the client "done") — but it stalls the whole system whenever that follower is slow or down, trading availability for durability.
A single-leader setup has no write conflicts.
True — with exactly one writer (the leader) there is no way for two conflicting versions of a row to be accepted concurrently, which is precisely why it's the simplest topology.
id % N is a bad shard function because the hash is weak.
False — the hash quality isn't the issue; the problem is that % N ties every key's home to the value of , so scaling out reshuffles almost everything.
CAP says a distributed database is always either consistent or available.
False — CAP only forces the choice during a network partition; when the network is healthy you can have both. See CAP Theorem (and the triangle sketch below).

Spot the error

"We'll shard by created_at so recent data is easy to find."
The error is the hotspot: time-ordered keys all land in the newest range, so every incoming write hammers one shard while the others sit idle (see the hotspot figure below).
"We use , , for speed and still get consistent reads."
Wrong — , so the write set and read set can miss each other entirely and the reader may see a stale value.
"To reshard, just change hash(k) mod 4 to hash(k) mod 5 — cheap."
The error is cost: under naive mod about of keys change their remainder (residue class), so ~80% of the dataset must physically move.
"Multi-leader replication is strictly better because writes are fast everywhere."
It ignores the cost: two leaders can accept edits to the same row, producing write conflicts that need explicit resolution logic the single-leader design never faces.
"Asynchronous replication is fine, we never lose writes."
The error is replication lag: the leader acks (says "done") before followers copy, so a crash in that window loses the last un-replicated writes.
" gives us the safest writes."
It gives the safest durability but zero write fault tolerance — if any one of the replicas is down, no write can reach all and so no write can complete at all.
"Consistent hashing needs no virtual nodes; one point per server is enough."
With a single point per server, the arcs (each server's ring slice) are wildly uneven, so load and the data-to-move on failure are lopsided; virtual nodes exist to smooth the arcs. See Consistent Hashing.
"A follower serving reads always returns the same value the leader just wrote."
Not with asynchronous replication — the follower may lag behind, so it can return an older value than a just-completed write (the read-your-writes problem). See Replication Lag and Read-Your-Writes.

Why questions

Why does hash mod N remap almost every key when changes?
Because a key stays put only if ; since acts like a random number, landing back in the same one of the possible remainders happens with probability only about — so nearly all keys land in a new bucket. See Hash Functions.
Why does the quorum condition need strict inequality and not ?
The overlap size of the two groups is ; for a guaranteed shared replica () we need , i.e. . Equality allows the sets to be disjoint (touch zero common boxes).
Why is range sharding good for WHERE age BETWEEN 20 AND 30 but hash sharding bad for it?
Range sharding keeps contiguous keys together so the query touches few shards; hashing deliberately scatters neighbours across all shards, so a range scan must query every one.
Why can't you get both consistency and availability during a partition?
A partitioned node can't confirm with its peers, so it must either refuse to answer (keep C, lose A) or answer with possibly-stale data (keep A, lose C). See CAP Theorem.
Why does a leaderless system rely on version numbers when reading a quorum?
The read set may return several copies at different versions; version numbers let the reader identify the newest one among them, which the quorum overlap guarantees is present.
Why do we replicate each shard rather than replicate the whole database once?
Because sharding already split the data for scale; replicating each shard preserves that scale while adding survival — copying the whole thing again would defeat the sharding. See Load Balancing.

Edge cases

What happens under consistent hashing when and you add the 2nd node?
About of keys move — with only two nodes the ring splits into two arcs of roughly half each, so a large fraction moving is expected, not a bug.
What does require when (single replica)?
Any satisfies , but there is no fault tolerance and no read scaling — the "quorum" is trivial and the node is a single point of failure.
What happens to writes if a follower in a synchronous single-leader setup goes down?
The leader can no longer get its confirmation, so writes stall until the follower recovers or is dropped from the sync set — availability suffers precisely because durability was prioritised.
In leaderless with , what have you actually built?
A system that requires all replicas for both reads and writes — maximally consistent but with zero fault tolerance and the worst latency; correct but usually impractical.
If keys are perfectly time-ordered, does hash sharding still create a hotspot?
No — a good hash destroys the ordering, so sequential keys scatter uniformly; the hotspot is a property of range sharding on ordered keys, not of the data alone. See Hash Functions.
What is the fraction of keys moved under consistent hashing when you remove a node from nodes?
About — the departing node's single arc is handed to its clockwise neighbour, mirroring the add case.

Recall One-line summaries to lock in

Sharding ::: split distinct data for scale. Replication ::: copy same data for survival. Naive mod N reshard cost ::: about of keys move. Consistent hashing add-node cost ::: about of keys move. Quorum guarantee ::: forces read/write overlap.