Logging and monitoring — structured logging, metrics, alerting: Visual walkthrough

The chain we are going to draw:

Each box below is one numbered step with one figure. Read the figures — the text only points at them.

Step 1 — A counter is just a staircase

WHAT. The most basic signal a service emits is a counter: a whole number that starts at zero and jumps up by one every time an event happens (a request served, a login failed). It never goes down (except a reset to zero when the process restarts). Call its value at time $t$ the symbol $C (t)$ .

$C$ — the running total (a plain integer like $0, 1, 2, 3, \dots$ ).
$t$ — the clock time when we read it, in seconds.
$C (t)$ — read aloud as " $C$ at time $t$ ": the total so far, measured at that instant.

WHY. We do not care about the raw total $1, 000, 402$ — that number depends on how long the process has been alive, so it is not comparable between two machines. But the shape of the staircase — how steeply it climbs — tells us how busy the service is. That steepness is the thing we actually want.

PICTURE. Look at the blue staircase: each vertical jump is one event. Where events come fast, the steps bunch together and the staircase climbs steeply. Where the service is idle, the line goes flat.

Figure — Logging and monitoring — structured logging, metrics, alerting

Step 2 — Slope turns the staircase into a rate

WHAT. To get "events per second" we measure how much the counter climbed between two clock readings and divide by how much time passed. Pick an earlier time $t_{1}$ and a later time $t_{2}$ .

$rate = events per second \frac{C ( t _{2} ) total now - C ( t _{1} ) total before}{seconds elapsed t _{2} - t _{1}}$

$C (t_{2}) - C (t_{1})$ — the number of events that happened in the window (each event added exactly $1$ , so the rise is the count).
$t_{2} - t_{1}$ — the width of the window in seconds. We write it $Δ t$ ("delta t" — the Greek letter delta means "the change in").
The whole ratio — rise over run — is the slope of the staircase.

WHY THIS TOOL — the slope? We want "how fast", and "how fast" is always a change in the thing divided by the change in time. That is exactly what slope means. We choose subtraction-then-division (not, say, the raw value) because subtraction cancels the meaningless starting height: whether the counter was at $5$ or at $5, 000, 000$ when the window opened, only the difference survives. That is also why a rate survives a restart — a fresh process starting at $0$ still has a sensible slope.

PICTURE. The green line is the straight ramp connecting the two dots $(t_{1}, C (t_{1}))$ and $(t_{2}, C (t_{2}))$ . Its steepness — vertical rise over horizontal run — is the rate. A steeper green line means more events per second.

Step 3 — Two counters give an error fraction

WHAT. Run two counters side by side:

$T$ = total requests (every request ticks it),
$E$ = failed requests (only failures tick it, and every failure also ticks $T$ ).

Take their rates over the same window and divide:

$f = \frac{rate ( E )}{rate ( T )} = \frac{\frac{E ( t _{2} ) - E ( t _{1} )}{Δ t}}{\frac{T ( t _{2} ) - T ( t _{1} )}{Δ t}} = \frac{E ( t _{2} ) - E ( t _{1} )}{T ( t _{2} ) - T ( t _{1} )}$

$f$ — the error fraction, a pure number between $0$ and $1$ (e.g. $0.02$ means $2%$ of requests failed).
The two $Δ t$ 's cancel because both counters were measured over the same window — that is why we deliberately used the same window.
Numerator = failures in the window; denominator = requests in the window.

WHY. A raw failure rate (" $5$ errors/sec") is not scary on its own — $5$ errors out of $50, 000$ requests is fine, $5$ out of $50$ is a fire. Dividing by traffic gives a number that means the same thing at 3 a.m. (low traffic) and at noon (high traffic). It is normalised to user pain.

PICTURE. The red staircase ( $E$ ) climbs gently under the taller blue staircase ( $T$ ). At each window we read both climbs; their ratio is the yellow fraction shown on the right axis.

Step 4 — The SLO draws a ceiling; the gap is the budget

WHAT. We now declare a promise. An SLO (Service Level Objective) is a target success fraction, e.g. " $99.9%$ of requests succeed", written $SLO = 0.999$ . Flip it around: the fraction we are allowed to fail is

$budget = all requests 1 - must succeed SLO$

$SLO$ — the floor on success (a fraction, here $0.999$ ).
$1 - SLO$ — the error budget: the ceiling on allowed failure ( $0.001$ , i.e. $0.1%$ ).

Multiply the budget fraction by total requests $N$ over a window to get an allowed count of failures, or by the window length to get allowed downtime:

$allowed failures = (1 - SLO) \cdot N, allowed downtime = (1 - SLO) \cdot window$

WHY. The budget converts a vague promise ("be reliable") into a hard, spendable currency. If you have not blown the budget, you can afford a risky deploy; if you have, you freeze and fix. It is a ceiling on the fraction $f$ from Step 3.

PICTURE. The green horizontal line is the SLO's mirror — the allowed-failure ceiling at $0.001$ . The yellow error fraction $f$ from Step 3 lives below it when healthy and pokes above it when we are in trouble. The shaded band between $f$ and the ceiling is budget remaining.

Step 5 — Burn rate: how fast are we spending the budget?

WHAT. Being over budget is not itself the emergency — the emergency is spending it fast. Define the burn rate as how many times faster than "sustainable" you are consuming the budget:

$burn = \frac{f}{1 - SLO} = \frac{current error fraction}{budget fraction}$

$f$ — the live error fraction from Step 3.
$1 - SLO$ — the budget fraction from Step 4.
$burn = 1$ means you are failing at exactly the allowed pace — the budget lasts the whole window and ends exactly at zero.
$burn = 14$ means you are spending $14 \times$ too fast: a whole month's budget gone in about $30/14 \approx 2$ days.

WHY THIS RATIO — why divide by the budget? Dividing $f$ by the budget fraction rescales everything so the meaningful threshold is always the plain number $1$ , no matter what the SLO is. A service at $99.9%$ and one at $99%$ both use "burn $> 1$ means trouble". It makes alerts portable.

PICTURE. The horizontal green line sits at $burn = 1$ (sustainable). The yellow curve is the live burn. Where it climbs above $1$ , the red shaded region marks budget being over-spent — the higher the spike, the faster the money burns.

Step 6 — The "for" clause kills the noise before it pages

WHAT. Raw metrics jitter: one scrape can spike to $burn = 20$ for a single second because two errors happened to land together. So the final alert rule is not " $burn > 14$ " but:

$page if burn > 14 held continuously for τ$

$τ$ (the Greek letter "tau") — the duration clause, e.g. $5$ minutes.
The condition must stay true across every scrape in that whole window, not just touch it once.

WHY. A single spike is almost always noise; a sustained burn is a real fire. Requiring persistence trades a few minutes of detection delay for a huge drop in false pages — the cure for alert fatigue, where humans learn to ignore a boy-who-cried-wolf alert and miss the real one. See Incident Response for what happens after the page fires.

PICTURE. The yellow burn curve crosses the threshold three times. Two crossings are thin spikes narrower than $τ$ — no page. The third stays above the line for the full $τ$ (grey window) — page fires, marked with the red bell.

The one-picture summary

Everything above is one climb: a staircase → its slope → a fraction of two slopes → compared to a ceiling → rescaled into a burn number → filtered by a duration. The final figure stacks all five plots so you can trace one incident from the first tick to the ringing bell.

Recall Feynman: tell the whole walkthrough to a 12-year-old

Imagine a turnstile that clicks up by one every time someone walks through — that's the counter (Step 1). You don't care what number the turnstile shows; you care how fast it's clicking, which is the click count divided by the seconds — that's the rate (Step 2). Now put two turnstiles side by side: one clicks for everyone, one clicks only when someone trips and falls. Trips divided by everyone gives the fraction of people having a bad time (Step 3). You promised the boss only one in a thousand may trip — that promise is a ceiling, and the space under it is your "trip budget" (Step 4). What matters isn't whether you've used the budget, but whether you're blowing through it fast — if a month's budget vanishes in two days, that's burning at fourteen times the safe speed (Step 5). Finally, you don't shout every time one person stumbles; you only ring the alarm if the fast-tripping keeps up for a solid five minutes, so you don't cry wolf (Step 6). That whole chain — click, speed, fraction, ceiling, burn, sustained — is how a computer decides to wake a human.

Recall Quick self-test

What does dividing $E (t_{2}) - E (t_{1})$ by $T (t_{2}) - T (t_{1})$ give, and why do the $Δ t$ 's vanish? ::: The error fraction $f$ ; both rates were measured over the same window, so the identical $Δ t$ in numerator and denominator cancels. Why divide the error fraction by $(1 - SLO)$ to get burn? ::: So the danger threshold is always the plain number $1$ regardless of the SLO, making alerts portable across services. Why add a "for $τ$ " clause instead of paging the instant $burn > 14$ ? ::: Single scrapes are noisy; a sustained breach is a real fire. The duration filter trades a little delay for far fewer false pages, avoiding alert fatigue.

See also: Observability · Percentiles and Distributions · Time Series Databases · Distributed Tracing · Rate Limiting · PII and Data Privacy