Imagine your video game has a tiny robot inside it. The robot keeps a diary: "11:03 — player jumped, 11:04 — coin grabbed" (that's logging). It also keeps a scoreboard of numbers: coins per minute, how slow the game feels (that's metrics). And you tell the robot: "If the game gets super slow for 5 whole minutes, ring a bell and wake me up" (that's alerting). The diary tells you what happened, the scoreboard tells you how it's going, and the bell makes sure you only get woken when something actually matters — not every time a leaf moves.
Emitting log events as machine-parseable key–value records (e.g. JSON) instead of free-form text, so they can be queried, filtered, and aggregated.
Why attach a correlation/trace ID to logs?
So all log lines belonging to one request can be stitched together across multiple services into a single story.
Counter vs Gauge vs Histogram?
Counter only increases (e.g. total requests); Gauge goes up and down (e.g. current memory); Histogram buckets values to estimate a distribution (e.g. latency).
Why measure rate of a counter instead of its absolute value?
You care about events-per-second, which is comparable across machines and survives restarts; rate = ΔC / Δt.
What are the Four Golden Signals?
Latency, Traffic, Errors, Saturation.
Why use p99 latency instead of the average?
The average hides the tail; p99 reveals the slow 1% of requests that real users actually suffer.
How do you compute the p-th percentile of n sorted samples?
Rank k = ceil(p/100 · n), then take the k-th smallest value.
What is an error budget?
The allowed amount of failure, equal to (1 − SLO) × total requests (or window time).
How much monthly downtime does a 99.9% SLO allow?
(1 − 0.999) × 43200 min ≈ 43.2 minutes per month.
Why add a "for X minutes" duration to an alert?
To suppress noisy one-off spikes; the condition must be sustained, trading a little detection delay for far fewer false pages.
What is alert fatigue and how do you avoid it?
Humans ignoring alerts because too many fire; fix by alerting only on actionable, symptom/budget-burn conditions.
Symptom-based vs cause-based alerting — which is preferred and why?
Symptom-based (what users feel, e.g. error rate), because causes are infinite but user-facing symptoms are few.
One thing you must NEVER put in logs?
Secrets / PII like passwords or full card numbers.
Socho tumhara service ek black box hai jo production mein chal raha hai — tum debugger laga nahi sakte. Toh teen cheezein chahiye taaki andar dekh sako. Structured logging matlab har event ko JSON jaise key-value form mein likho ({"event":"login_failed","user_id":42}) na ki seedhi English sentence mein — kyunki JSON ko tum query kar sakte ho, filter kar sakte ho. Har request ko ek correlation ID do aur har log line mein daalo, taaki ek hi request ki poori kahani saare services mein jod sako.
Metrics numbers hote hain jo time ke saath track hote hain. Teen type: Counter (sirf badhta hai, jaise total requests), Gauge (upar-neeche, jaise memory), aur Histogram (distribution, jaise latency). Counter ki absolute value matter nahi karti — uska rate (ΔC / Δt = requests per second) matter karta hai. Aur sabse important: average latency par mat jao, kyunki average tail ko chhupa deta hai. Agar 1% requests 7 second leti hain toh average phir bhi chhota dikhega, par woh 1% users ko bahut bura experience milta hai. Isliye p95, p99 dekho — p99 woh value hai jisse 99% requests fast hain.
Alerting ka kaam hai: jab koi metric user-pain dikhaye (jaise error rate > 2%) aur kuch minute tak sustained rahe, tabhi human ko jagao. "For 5 minutes" wala clause noise hata deta hai. Har choti error par alert mat lagao — warna alert fatigue ho jaata hai aur asli emergency miss ho jaati hai. Best practice: error budget use karo. 99.9% SLO ka matlab mahine mein sirf ~43 minute downtime allowed hai — yeh number batata hai ki risky deploy karein ya budget bachayein. Yaad rakho: Logs batate hain KYA hua, Metrics batate hain KITNA, Alerts kehte hain ABHI ACTION lo.
