Intuition The one core ideaA running program is a sealed box you cannot pause or peek inside; so you make it spit out evidence about itself over time — words for single events, numbers for trends, and rules that shout when the numbers go bad. Everything in this topic is just three ways of turning that stream of evidence into something a tired human can act on at 3 a.m.
This page assumes you have seen nothing . Before you can read the parent note on Logging and Monitoring , you must be fluent in a handful of small ideas and symbols. We build each one from a picture, in an order where every idea leans only on the ones before it.
Definition Service (running program)A service is a program that stays alive and answers requests — small "please do this" messages sent to it over the network. A web server answering "give me this page" is a service.
Intuition Why we cannot just watch itOn your laptop you can stop the program and inspect it. In production the same program is answering thousands of requests per second across many machines. Stop it and users notice. So the only way to know what it's doing is to make it describe itself out loud while it runs . That description is the raw material of this whole topic.
Look at the figure: the box on the left is the service. Requests fly in, responses fly out, but the inside is dark. The three arrows leaving the top — words, numbers, a bell — are exactly logging, metrics, and alerting. Keep this picture; every later symbol lives on one of those three arrows.
An event is one thing that happened at one moment: "a login failed", "a payment was attempted". It is discrete — it either happened or it didn't.
Definition Field (key–value pair)A field is a labelled fact about an event, written key: value. user_id: 42 means "the field named user_id has the value 42". The key is the label; the value is the data.
Intuition Why key–value and not a sentence?A sentence like "user 42 failed login" mixes the label and the data into English glue. A machine has to guess where the number is. Key–value pulls them apart: the computer always knows user_id is where the user lives. This is the seed of structured logging — an event written as a bag of fields the parent note shows as JSON.
Definition JSON (the shape of a structured record)JSON is a plain-text way to write a bag of fields: curly braces around "key": value pairs separated by commas, e.g. {"event":"login_failed","attempts":3}. That is all you need to know — it is just a tidy, machine-readable way to write the fields above.
Cardinality is how many different values a field can take . level has low cardinality (only DEBUG/INFO/WARN/ERROR — 4 values). user_id has high cardinality (millions of possible users). request_id has near-infinite cardinality (a fresh one per request).
Intuition Why this word matters laterLogs happily hold high-cardinality fields — that is their superpower (you want to find one exact request). Metrics, which we meet next, choke on high cardinality because each distinct value becomes its own stored line of numbers. Knowing the word tells you which pillar a piece of data belongs in.
Metrics are numbers over time, so you need the picture of a number laid out.
Definition Number line and axisA number line is a straight line with a zero mark; positions to the right are bigger numbers, to the left smaller. An axis is a number line we draw on a graph to measure one quantity. The horizontal axis (called the x-axis ) usually measures time ; the vertical axis (the y-axis ) measures the value .
A time series is a list of (time, value) pairs — one measurement per moment — drawn as dots on the time-vs-value graph, often joined into a line. "Memory used, sampled every 10 seconds" is a time series.
In the figure the pale-yellow line only ever climbs — that is a counter . The blue line goes up and down — that is a gauge . Same axes, different shapes. The parent note's whole "Metrics" pillar lives on this picture.
The parent note turns a counter into a rate . To read that formula you need one idea: slope.
Definition Slope (rise over run)Take two dots on a line, at times t 1 t 2 C ( t 1 ) C ( t 2 ) rise is how much the value grew, C ( t 2 ) − C ( t 1 ) run is how much time passed, t 2 − t 1 slope is
slope = run rise = t 2 − t 1 C ( t 2 ) − C ( t 1 ) .
Intuition Why division, and why THIS division?We divide because we want "how much change per unit of time ". Division is the tool that answers "per" questions: change per second, cost per item. The parent note calls this slope the rate of a counter. A counter's raw value (1 , 000 , 402
The figure marks the two chosen points, the rise (vertical), and the run (horizontal). The steepness of that dashed line is the rate. A flatter line means fewer requests per second; a vertical jump means a burst.
Definition Fraction and percentageA fraction is a part out of a whole, like 2 1 percentage is a fraction with the whole fixed at 100: 99% = 100 99 = 0.999 99% = 0.99 99.9% = 0.999 0.999
Intuition Why "one minus the target" appears everywhereIf a fraction s succeed , then the fraction allowed to fail is "the rest of the whole", which is 1 − s 1 error budget = 1 − SLO
Worked example Sanity check the arithmeticTarget SLO = 0.999 = 1 − 0.999 = 0.001 43200 = 0.001 × 43200 = 43.2
Percentiles are the subtlest tool in the parent note, so we build every symbol.
Definition Sorted samples and rankCollect n Sort them from smallest to largest. The rank of a value is its position in that sorted line: rank 1 is the smallest, rank n x ( k ) k k sorted position , not the k
Definition The ceiling function⌈ x ⌉ ⌈ x ⌉ x round up to the next whole number . ⌈ 9.5 ⌉ = 10 ⌈ 7 ⌉ = 7 up , never down.
Intuition Why ceiling and not ordinary rounding?A percentile must guarantee "at least p % 9.5 9 up is the safe direction — it never under-counts. That is why the tool is ceiling, specifically.
The figure lays 10 sorted latency samples along a line. The p95 mark falls at rank ⌈ 0.95 × 10 ⌉ = 10 P 95 is the percentile — the picture makes obvious why you need many samples before p99 means anything.
Definition Percentile, in wordsThe p-th percentile P p p % median is just P 50
Definition Average (mean)The average of numbers is their sum divided by how many there are. It collapses a whole crowd into one point.
Intuition Why the topic prefers percentilesThe average is pulled around by extremes and hides a rare disaster. If 99 requests take 10 ms and 1 takes 7000 ms, the average is only ~80 ms — it makes the disaster invisible. A percentile keeps the tail in view . This is the "average vs percentile" argument in the parent note; you now own every symbol in it.
Structured Logging pillar
error budget and Alerting
Once these foundations are solid, the three pillars slot in: fields and cardinality feed structured logging , slope and time series feed metrics , and percentiles plus the fraction 1 − SLO alerting .
The umbrella property "infer inside state from outside signals" is Observability .
Stitching one request across services with a shared ID is Distributed Tracing .
Targets and budgets are formalised in SLI SLO SLA .
The tail-vs-average maths deepens in Percentiles and Distributions .
Storing time series efficiently is the job of Time Series Databases .
Acting on a fired alert is Incident Response .
Not logging secrets is PII and Data Privacy .
Shedding excess load ties into Rate Limiting .
Cover the right side and answer each. If any stumps you, reread its section above.
What is a "service" and why can't you just pause it to inspect it? A long-running program answering requests; pausing it in production breaks live users, so it must describe itself while running.
What is a field, and how does it differ from a plain sentence? A labelled key: value fact; unlike a sentence, the label and data are separated so a machine can query them.
What does high cardinality mean and which pillar tolerates it? Many possible distinct values; logging tolerates it, metrics do not.
On a time-vs-value graph, which shape is a counter and which is a gauge? Counter only rises (or resets to 0); gauge goes up and down.
Write the slope of a counter between t 1 t 2 t 2 − t 1 C ( t 2 ) − C ( t 1 )
What does Δ t A change in time, t 2 − t 1
What does C ( t ) "The value of counter C t
If SLO = 0.999 1 − SLO 0.001
Evaluate ⌈ 9.5 ⌉ ⌈ 7 ⌉ 10 7 p %
Define the p95 in one sentence. The value that 95% of samples are at or below.
Why does the topic distrust the average latency? It hides rare slow outliers (the tail), so a bad experience for a few users becomes invisible.
