Exercises — Logging and monitoring — structured logging, metrics, alerting

Answer: Logging.

• A log is a discrete record of one event with high detail (per-event fields).
• A metric throws away per-event detail and keeps only aggregate numbers over time — it literally cannot tell you this one payment's IP.
• Alerting is a decision layer ("wake a human"), not a store of event detail.

The tell-tale words are "exact single event." Logs are the only pillar that preserves individual events.

• (a) Counter — it only ever goes up (or resets to 0 on restart). "Total ... since boot" is the signature.
• (b) Gauge — "right now" and it can rise and fall as connections open/close.
• (c) Histogram — it ships bucket counts; percentiles are computed later, on the server, and — crucially — buckets from many instances can be summed before you read a percentile. Use this when you aggregate.
• (d) Summary — a fourth type (e.g. a Prometheus Summary) where each process computes its own quantiles (p50/p95/…) locally and exports them as ready-made numbers. Cheap to read, but you cannot meaningfully average quantiles across instances — so it's for per-instance views only.

Rule of thumb: only up → counter; up and down → gauge; shape you'll aggregate later → histogram; quantiles pre-computed per instance → summary.

Histogram vs Summary, in one line: histogram = "here are my raw buckets, you do the percentile math (and you can add my buckets to others')"; summary = "here is my p95, already cooked (but don't average it with someone else's)."

Normal case. $C(t)$ is the counter value; $\Delta t=t_2-t_1$ is the elapsed seconds. $\text{rate}=\frac{C(t_2)-C(t_1)}{t_2-t_1}=\frac{1,003,402-1,000,402}{30}=\frac{3000}{30}=100\text{req/s}$ Each increment is one event, so the difference is exactly the number of events; dividing by seconds gives events per second. The huge absolute value $1,000,402$ cancels — only the change matters.

Reset case. Naively $C(t_3)-C(t_2)=1500-1,003,402=-1,001,902$ — a negative rate, which is nonsense (a counter can never truly go down). The negative sign is the signal that a reset happened (process restart, or the counter wrapped past its storage limit). Two standard fixes:

• Reset handling (what Prometheus does): if $C(t_3)<C(t_2)$, assume it dropped to $0$ and climbed back, so the events since the reset are just $C(t_3)$ itself: $\text{events}\approx C(t_3)=1500,\text{rate}=\frac{1500}{t_3-t_2}=\frac{1500}{60}=25\text{req/s}$
• Wrap-around (fixed-width counters): if the counter is a $b$-bit integer that overflowed at $2^b$, recover the true delta with modulo arithmetic: $\text{events}=(C(t_3)-C(t_2))\mathrm{mod}{2}^b$.

The robust rule: compute $\Delta C=C(t_2)-C(t_1)$; if $\Delta C<0$, treat it as a reset (use $C(t_2)$) or as a wrap (add $2^b$) rather than reporting a negative rate. A monitoring system that skips this shows scary downward spikes on every deploy.

The nearest-rank rule: sort ascending, then the $p$-th percentile sits at rank $k=\lceil \frac{p}{100}\cdot n\rceil$, and $P_p=x_{(k)}$ (the $k$-th smallest). $\lceil \cdot \rceil$ means "round up."

p50: $k=\lceil 0.50\times 10\rceil =\lceil 5\rceil =5$ → the 5th value = 23 ms. p95: $k=\lceil 0.95\times 10\rceil =\lceil 9.5\rceil =10$ → the 10th value = 210 ms.

Why round up: we need at least $p\%$ of samples at or below the answer. With $n=10$ and $p=95$, exactly 9 samples is only $90\%$ — not enough — so we step to rank 10.

The error budget is the fraction allowed to fail, $1-\text{SLO}$, times the total. $\text{budget}=(1-0.999)\times 2,000,000=0.001\times 2,000,000=2000\text{failures}$ Why: SLO is the fraction that must succeed; whatever's left, $1-\text{SLO}$, is your spendable failure allowance.

Non-integer percentiles use the same rule. The rank formula $k=\lceil \frac{p}{100}\cdot n\rceil$ works for any $p$ — integer or not. You just plug the decimal in: $p_{99.1}:k=\lceil 0.991\times 1000\rceil =\lceil 991\rceil =991,p_{99.9}:k=\lceil 0.999\times 1000\rceil =\lceil 999\rceil =999.$

Mean = total time ÷ count: $\bar{x}=\frac{990\times 10+10\times 5000}{1000}=\frac{9900+50000}{1000}=\frac{59900}{1000}=59.9\text{ms}$ Percentiles. The sorted list is 990 copies of $10$ then 10 copies of $5000$ (ranks 991–1000 are the slow ones):

• $p_{99}$: rank $k=\lceil 0.99\times 1000\rceil =990$ → the 990th value = 10 ms (last fast sample).
• $p_{99.1}$: rank $991$ → the 991st value = 5000 ms (first slow sample).
• $p_{99.9}$: rank $999$ → 5000 ms.

Reading it: the mean is a fiction. The p99 sits right on the cliff edge — one rank later everything changes. To see the pain you need p99.9 ($=5000$ ms). General rule: your percentile must be deeper than your bad-request fraction. Here $1\%$ are slow, so anything up to p99 can look clean. See Percentiles and Distributions.

Scrapes per day: $\frac{24\times 60\times 60}{30}=\frac{86400}{30}=2880$ scrapes.

Single crossing: expected false pages $=0.05\times 2880=144$ per day. Unusable — pure alert fatigue.

10 consecutive (independence): probability a given window of 10 is all-noise $=0.05^{10}$. That's about $9.77\times 10^{-14}$ per window — effectively zero false pages ever.

Why the "for" clause is magic: independent noise crossings almost never line up 10 in a row, but a real sustained problem trivially does. You trade $\sim 5$ minutes of detection delay for wiping out ~144 false pages/day. This is exactly the parent's "for 5 minutes" idea. See Incident Response.

Allowed downtime $=(1-\text{SLO})\times \text{window}$.

• $99\%$: $(1-0.99)\times 43200=0.01\times 43200=432$ min $=7.2$ h
• $99.9\%$: $0.001\times 43200=43.2$ min
• $99.99\%$: $0.0001\times 43200=4.32$ min

Pattern: each added "nine" divides allowed downtime by 10. "Two nines" = 7.2 hours of slack (easy); "four nines" = 4.32 minutes (you likely can't even deploy safely inside that). Each nine is roughly $10\times$ the engineering cost. See SLI SLO SLA.

"Even" burn spends the whole budget across the window: $\frac{2000}{720}\approx 2.78$ failures/hour is the budgeted rate. $\text{burn rate}=\frac{\text{actual failures/hr}}{\text{budgeted failures/hr}}=\frac{700}{2000/720}=\frac{700\times 720}{2000}=\frac{504000}{2000}=252\times$ Interpretation: you are burning the monthly budget $252$ times too fast. Time to exhaust it: $\frac{2000}{700}\approx 2.86$ hours. Page immediately — the whole month's tolerance is gone before lunch. Why burn rate, not raw count: it converts "700/hr" into "how long until we've spent everything," which is directly actionable and self-scaling to any window.

If burn rate $b$ means "spending $b\times$ the even pace," then time-to-exhaust $=\frac{\text{window}}{b}$. Solve for $b$: $b=\frac{\text{window}}{\text{time-to-exhaust}}$ Window $=30$ days $=720$ h.

• Fast (2 h): $b=\frac{720}{2}=360\times$. for: 2m — a 2-minute confirmation window. At $360\times$ you're spending the whole month in 2 hours, so you cannot afford to wait long; 2 minutes still filters single noisy scrapes (see L3.2) without meaningful delay. Page the on-call now, high urgency.
• Slow (3 days $=72$ h): $b=\frac{720}{72}=10\times$. for: 1h — a 1-hour confirmation window. A $10\times$ creep isn't an emergency, so we can wait an hour to be sure it's sustained and not a transient bump; this kills false pages from short spikes. Ticket / working-hours page, not a wake-up.

Design defense:

• The two for durations are chosen from the response time each tier allows: the fast tier's budget dies in 2 h, so waiting more than a couple of minutes wastes precious response time; the slow tier's budget dies in 3 days, so a 1 h wait is negligible but hugely reduces noise.
• Two tiers avoid the parent's alert fatigue: tiny blips never sustain $10\times$ for a full hour, so they page no one.
• Both alert on a symptom (budget burn = user-visible failures), never on internal causes.
• This is exactly a symptom-based, budget-driven policy from the parent, made concrete.

First, two terms defined:

• A correlation ID (a.k.a. request ID) is a unique string generated once per incoming request — e.g. a random UUID like a1b2-... — and then attached as a field to every log line that request produces, in every service it touches. In practice the first service mints it and passes it downstream in an HTTP header (e.g. X-Request-Id / W3C traceparent); each service reads that header and includes it in its structured logs.
• A trace is the assembled story: all the log lines/spans that share one correlation ID, stitched in time order across services, so you see the request's full journey. Building traces automatically is Distributed Tracing.

Path:

1. Metrics told you that it's bad (symptom) — they can't tell you why (they discard per-event detail).
2. Filter structured logs for the failing events: event=payment_failed AND level=error around 14:03.
3. Group by an error field to spot the dominant failure, then pull one failing event's correlation ID.
4. Query all lines carrying that ID to reconstruct the trace — the one broken hop is your root cause.

Privacy pitfall: in step 2 you may be tempted to log full card numbers / passwords to "have everything." That logs PII into a searchable store — a compliance breach. Log a reference (last 4 digits, a hashed user id), never the secret. See PII and Data Privacy.

Query-performance pitfall (cardinality): cardinality is the number of distinct values a field can take. Indexing a high-cardinality field (a raw user_id, an email, a full URL with query string) makes the log store build an enormous index — queries and ingestion slow to a crawl or the index blows your memory budget. The correlation ID is itself high-cardinality, so index it as a lookup key but don't turn it into a metric label (labels multiply time series — one series per distinct value — and can crush a time-series database). Keep metric labels low-cardinality (status code, region); keep high-cardinality identifiers in logs, retrieved by exact-match lookup.

A counter stores values that only go ::: up (rate is derived as its slope) If the counter delta is negative it means ::: a reset or wrap-around happened — handle it, don't report a negative rate The nearest-rank p-th percentile rank is ::: $k=\lceil \frac{p}{100}\cdot n\rceil$ The metric type that pre-computes quantiles per instance is a ::: Summary Error budget equals ::: $(1-\text{SLO})\times \text{total}$ Burn rate to exhaust a window $W$ in time $T$ is ::: $b=W/T$ You alert on symptoms, not ::: causes The field that stitches one request across services is the ::: correlation / trace ID A field with many distinct values (bad as a metric label) is ::: high-cardinality