4.5.22 · D5Software Engineering

Question bank — Logging and monitoring — structured logging, metrics, alerting

1,563 words7 min readBack to topic

Prerequisites worth having open: Observability, Percentiles and Distributions, SLI SLO SLA, Distributed Tracing, Time Series Databases, Incident Response.


True or false — justify

More logs always make debugging easier
False. Beyond a point volume hides signal — a firehose of DEBUG lines buries the single ERROR that matters, and it costs storage/ingestion money. The right fields beat more lines.
A metric contains more detail about a single event than a log line
False. A metric deliberately throws away per-event detail to keep only cheap aggregatable numbers; a log keeps the rich per-event context. They trade detail for cheapness in opposite directions.
Average latency of 80 ms means almost all users have a good experience
False. The average hides the tail: 99% at 10 ms and 1% at 7 s still averages ~80 ms, yet 1 in 100 users suffers. See Percentiles and Distributions — track p95/p99, not the mean.
A counter that only ever increases is useless because the raw number 1,000,402 tells you nothing
False (the premise is right but the conclusion is wrong). The raw value is uninteresting, but its slope gives requests/sec, which is exactly what you want and survives restarts.
Alerting is just a fancier form of logging
False. Logging and metrics describe the system; alerting is a decision layer on top of metrics that answers one question — should a human act right now?
"99.9% availability" leaves essentially no room for failure
False. Over 30 days that "three nines" still permits minutes of downtime — a real, spendable budget, not perfection.
Symptom-based alerting means you alert on internal causes like "disk 80% full"
False. It is the opposite: alert on symptoms users feel (error rate, high p99). Causes are infinite; symptoms are few and actionable.
Adding a for 5 minutes duration clause to an alert makes it detect problems faster
False. It makes detection slightly slower on purpose, trading a few minutes of delay for a huge cut in false pages from noisy one-scrape spikes.
A correlation ID is only useful within a single service
False. Its whole point is to be attached to every log line a request touches across every service, so one ID reconstructs the entire distributed journey. This is the seed of Distributed Tracing.
Error budget is the fraction of requests that must succeed
False. That fraction is the SLO. The error budget is the allowed failure, — the part you are permitted to spend.

Spot the error

A dev logs card_number and cvv as structured fields "so incidents are easier to debug."
This logs PII / secrets — a security and compliance breach. Never log credentials or card data; redact or omit them regardless of debugging convenience. See PII and Data Privacy.
A team reports p95 latency computed from a sample of 5 requests as authoritative.
With tiny a single slow sample dominates the tail, so the percentile is statistically meaningless. Tail percentiles need lots of data to be trustworthy — Percentiles and Distributions.
A team uses a gauge to track "total requests served since boot."
A total-served count only goes up, so it should be a counter. A gauge is for values that rise and fall (memory in use, queue depth).
An alert pages on every single failed request "to catch everything."
This causes alert fatigue — constant pages get ignored, so the one real page is missed. Alert on sustained/high-enough failure (burn rate), not on each error.
A dashboard filters logs by matching the exact sentence "User 42 failed login…".
Free-form sentences break the moment someone reword the message; that is why you emit structured key–value events and query event=login_failed AND attempts>2 instead.
A rate is computed as C(t2) - C(t1) (raw difference) and labelled "requests per second."
That is a count of events over the interval, not a rate. You must divide by to get per-second units.
A team sets an SLO of "100% of requests succeed" and treats every error as a violation.
A 100% SLO leaves a zero error budget, so any single failure is a breach and every risky deploy is forbidden — an unworkable target. Real SLOs leave budget to spend. See SLI SLO SLA.
"Saturation" is measured by counting total errors.
Saturation measures how full a resource is (CPU, memory, queue depth), not errors. Errors are a separate golden signal.

Why questions

Why prefer a counter + derivative over a gauge for request volume?
A counter's rate (slope) is comparable across machines and survives process restarts, whereas a raw gauge snapshot of "current requests" is fragile and non-additive. Rates, not totals, are what you compare.
Why must every symbol/field you log ideally be something you might later filter, group, or alert on?
Logging has a cost (storage + ingestion). A field you will never query is pure overhead and noise, so the discipline is: if you'd never slice by it, reconsider logging it.
Why alert on burn rate rather than an absolute error count?
Burn rate tells you whether you are consuming the error budget fast enough to matter — it pages you for a 14×-too-fast burn, not for a slow trickle you can absorb, cutting false pages while still catching real outages.
Why is the p99 more operationally important than p50 for user-facing latency?
On a page that fires 100 backend requests, a p99 tail means almost every page hits at least one slow request, so the tail defines the felt experience even though it is "only 1%".
Why do the Four Golden Signals (Latency, Traffic, Errors, Saturation) give 80% of the value?
They are symptom-level signals users actually feel, and symptoms are few while internal causes are infinite — so a handful of signals covers almost every real problem without chasing every cause.
Why does a correlation ID turn "confetti" logs into a story?
Without a shared ID, each service's lines are unrelated scraps; the ID links every line one request produced across services into a single ordered narrative you can reconstruct.
Why store metrics in a time-series database rather than the log store?
Metrics are cheap numeric samples optimised for aggregation over time; a time-series DB compresses and queries these far more efficiently than a high-cardinality log store built for discrete events.

Edge cases

What does the counter-rate formula do when the counter resets (process restart) mid-interval?
The naive goes negative or wrong, because the derivation assumed no reset. Rate functions must detect the drop and treat the reset as a fresh start.
For samples and , why does the p95 land on the single worst value?
, the 10th rank — the maximum. With small the tail dominates, which is precisely why tail percentiles demand large samples.
What happens to detection if you set the for duration too long (say 2 hours)?
You suppress noise but also delay paging until the outage has hurt users for hours — the duration is a tunable trade-off between false pages and detection speed, not "bigger is always better".
If SLO = 100% of a 30-day window, what downtime does the budget allow?
minutes — zero tolerance, so the very first blip is a violation. This shows why sane SLOs are set below 100%.
What is the error budget when SLO = 0% (no requests need succeed)?
every request may fail — the degenerate opposite: nothing can ever violate it, so the SLO carries no meaning. Both 0% and 100% are useless extremes.
Is an alert that fires but is not actionable still useful?
No — an unactionable alert is noise that trains humans to ignore the alert channel, so it actively lowers reliability by masking real pages. If nobody can act, it shouldn't page.
Should you alert directly on "disk 80% full"?
Usually no — that is a cause, not a user symptom, and full disks may never hurt anyone. Prefer alerting on the symptom it might cause (errors/latency) and let the cause inform the investigation.

Recall One-line self-test before you close

Which pillar answers "should a human act right now"? ::: Alerting — the decision layer built on top of metrics, distinct from logging (what happened) and metrics (aggregate behaviour).