4.5.22
Intuition Ek-sentence mein poori pictureEk running service ek black box hai; logging, metrics, aur alerting woh teen holes hain jo tum drill karte ho taaki tum uske andar dekh sako , use measure kar sako , aur users ke notice karne se pehle jaag sako jab woh toot raha ho.
Intuition Bother kyun karein?Woh code jo "my machine pe kaam karta hai" production mein aisi tarahon se fail hota hai jo tum reproduce nahi kar sakte: ek node raat 3 baje memory khatam kar leta hai, ek customer ek rare race condition hit karta hai, latency har hafte 5 ms creep karti rehti hai. Tum ek live system pe debugger attach nahi kar sakte jo millions of requests serve kar raha ho. Toh tum system ko continuously khud ke baare mein evidence emit karne pe majboor karte ho. Teen pillars teen alag questions ka jawab dete hain:
Logging → Is ek event mein exactly kya hua? (discrete, high-cardinality)Metrics → System time ke saath aggregate mein kaisa behave kar raha hai? (numeric, cheap, time-series)Alerting → Kya ek insaan ko abhi act karna chahiye? (metrics ke upar decision layer)
Yeh teeno milke aksar observability kehlate hain — woh property ki tum external outputs se internal state infer kar sako.
Definition Structured loggingStructured logging ka matlab hai har log event ko ek machine-parseable record of key–value pairs (typically JSON) ke roop mein emit karna, na ki ek free-form human sentence ke roop mein.
FARQ kya hai?
Unstructured: "User 42 failed login from 10.0.0.3 after 3 tries" Structured: {"event":"login_failed","user_id":42,"ip":"10.0.0.3","attempts":3,"ts":"2024-...","level":"warn"} Structured ek sentence se kyun behtar hai:
Tum ise query kar sakte ho: event=login_failed AND attempts>2. Tum ek sentence ko reliably grep nahi kar sakte.
Yeh stable hai: ek message ki wording badalne se tumhare dashboards nahi tootenge.
Yeh context fields (request_id, user_id, trace_id) carry karta hai taaki tum ek request ki journey kai services mein stitch kar sako.
Intuition Killer feature: correlationHar incoming request ko ek unique correlation ID (a.k.a. trace ID) do aur use har log line se attach karo jo woh request touch karti hai, har service mein. Phir ek ID poori kahani reconstruct kar deta hai. Uske bina, logs confetti hain.
Worked example Worked example — context add karnalog.info( "payment_attempt" , request_id = req.id, # Kyun? humein is request ki SAARI lines dhundhne deta hai user_id = user.id, # Kyun? jab ek customer complain kare toh customer ke hisaab se slice karein amount_cents = 1999 , # Kyun? numeric → baad mein aggregate/sum kar sakte hain currency = "USD" , level = "info" ) Yeh step kyun? Har field kuch aisa hai jis par hum baad mein filter, group, ya alert kar sakte hain. Agar tum kabhi query nahi karoge, toh shayad log bhi mat karo (cost ki wajah se).
Common mistake Steel-man: "Sab kuch log karo, zyada safe hai."Kyun sahi lagta hai: jab incident aata hai, tum chaahte ho ki zyada data hota, toh zyada logging strictly safer lagti hai.
Kyun galat hai: logs paisa lagte hain (storage + ingestion), aur DEBUG noise ka firehose us ek ERROR line ko chhupa deta hai. Tum PII (passwords, card numbers) log karne ka risk bhi lete ho — yeh security/compliance breach hai.
Fix: sahi level par log karo (DEBUG/INFO/WARN/ERROR), zaroori cheezein structured fields banao, high-volume events ko sample karo, aur kabhi secrets log mat karo .
Ek metric ek numeric measurement hai system ki jo time ke saath sample ki jaati hai — ek time series. Ek log ke unlike, yeh per-event detail discard kar deta hai aur sirf woh numbers rakhta hai jo tum cheaply aggregate kar sako.
Teen classic metric types:
Type
Kya karta hai
Example
Counter sirf upar jaata hai (ya reset hota hai)
total requests served
Gauge upar aur neeche jaata hai
current memory in use
Histogram values ko bucket karta hai distribution estimate karne ke liye
request latency
Intuition "Requests" ke liye counter kyun, gauge kyun nahi?Tumhe actually absolute count 1,000,402 ki chinta nahi — tumhe rate ki chinta hai. Ek counter + derivative tumhe requests/second deta hai. Rates process restarts survive karte hain aur machines ke beech comparable hote hain.
Intuition 4 metrics se 80% valueGoogle ki SRE book: sirf Latency, Traffic, Errors, Saturation ("LTES") monitor karo aur tum almost sab kuch cover kar lete ho.
Latency — requests kitna time lete hain (tail dekho, mean nahi!)Traffic — demand (requests/sec)Errors — failed requests ki rateSaturation — tumhara resource kitna "full" hai (CPU, mem, queue depth)
Common mistake Steel-man: "Average latency 80 ms hai, hum theek hain."Kyun sahi lagta hai: average ek saaf number hai jo sabko summarize karta hai.
Kyun galat hai: average tail chhupa deta hai . Agar 99% requests 10 ms lete hain aur 1% 7 s lete hain, average phir bhi ~80 ms hai — lekin 100 mein se 1 user bura time khar raha hai, aur 100 requests wale page par almost har page slow hai.
Fix: percentiles track karo — p50 (median), p95, p99. p99 woh latency hai jisse 99% requests tez hain.
Worked example p95 compute karoLatencies (ms), sorted: [10,12,15,20,22,30,45,90,120,800] (n = 10 k = ⌈ 0.95 × 10 ⌉ = ⌈ 9.5 ⌉ = 10 P 95 = 800 Yeh step kyun? 95% requests ko answer ke barabar ya neeche hona chahiye; yahan sirf worst single sample (800) 95% mark ke upar baithta hai... actually yeh boundary hi hai — chhote n
Ek alert ek automated rule hai: jab ek metric itni der ke liye ek condition cross kare, ek insaan ko notify karo (ya page karo) .
Ek achha alert kaise banta hai:
Ek metric chunno jo user pain se judi ho (high error rate, high p99).
Ek threshold set karo.
Ek duration / "for" clause add karo taaki 1-second blip kisi ko page na kare.
Intuition "5 minutes ke liye" kyun?Raw metrics noisy hoti hain. Ek single scrape spike kar sakta hai. Duration clause ka matlab hai condition sustained rehni chahiye — yeh detection mein thodi minutes ki delay ke badle false pages ki massive reduction hai.
Intuition Symptom-based vs cause-based alertingSymptoms par alert karo jo users feel karte hain ("error rate > 2%"), na ki har internal cause par ("disk 80% full"). Causes infinite hain; symptoms kam hain. Yeh golden-signals 80/20 ka alerting form hai.
Definition SLO aur error budgetEk SLO (Service Level Objective) ek target hai jaise "99.9% requests succeed hoon." Error budget allowed failure hai: 1 − SLO
Worked example 99.9% kitna downtime buy karta hai?Window = 30 days = 30 × 24 × 60 = 43200 ( 1 − 0.999 ) × 43200 = 0.001 × 43200 = 43.2 min/month Yeh step kyun? "Three nines" bulletproof lagta hai lekin sirf ~43 min/month hai. Yeh number batata hai ki budget ko ek risky deploy par kharchein ya protect karein .
Common mistake Steel-man: "Har error par alert karo taaki sab kuch pakad sakein."Kyun sahi lagta hai: zero tolerance responsible lagta hai.
Kyun galat hai: alert fatigue — insaan un alerts ko ignore karte hain jo constantly fire hote hain, toh woh ek asli page miss ho jaata hai. Ek alert jo actionable nahi hai woh noise hai.
Fix: error-budget burn rate par alert karo (jaise "monthly budget 14× zyada tezi se burn ho raha hai"), taaki tum sirf tab page karo jab failure itni tez ho ki matter kare .
Recall Feynman: 12-saal ke bachche ko explain karoSocho tumhare video game ke andar ek chhota robot hai. Robot ek diary rakhta hai: "11:03 — player jumped, 11:04 — coin grabbed" (yeh hai logging ). Woh ek scoreboard bhi rakhta hai numbers ka: coins per minute, game kitna slow feel hota hai (yeh hai metrics ). Aur tum robot ko kehte ho: "Agar game poore 5 minutes ke liye super slow ho jaaye, toh ghanti bajao aur mujhe jagao" (yeh hai alerting ). Diary batati hai kya hua , scoreboard batata hai kaisa chal raha hai , aur ghanti ensure karti hai ki tumhe sirf tab jagaya jaaye jab kuch actually matter kare — har baar jab koi patta hiley nahi.
Mnemonic Pillars aur golden signals yaad rakhne ke liye"Logs kehte hain KYA, Metrics kehte hain KITNA, Alerts kehte hain ABHI ACT KAR."
Golden signals = LTES : L atency, T raffic, E rrors, S aturation → "Let The Engineers Sleep."
Structured logging kya hai? Har log event ko machine-parseable key–value records (jaise JSON) ke roop mein emit karna na ki free-form text, taaki unhe query, filter, aur aggregate kiya ja sake.
Logs se correlation/trace ID kyun attach karna chahiye? Taaki ek request se judi saari log lines multiple services mein ek single story mein stitched ki ja sakein.
Counter vs Gauge vs Histogram? Counter sirf badhta hai (jaise total requests); Gauge upar neeche jaata hai (jaise current memory); Histogram values ko bucket karta hai distribution estimate karne ke liye (jaise latency).
Counter ki rate measure kyun karein, uski absolute value kyun nahi? Tumhe events-per-second ki parwah hai, jo machines ke beech comparable hai aur restarts survive karta hai; rate = ΔC / Δt.
Four Golden Signals kya hain? Latency, Traffic, Errors, Saturation.
Average ki jagah p99 latency kyun use karein? Average tail chhupa deta hai; p99 us slow 1% requests ko reveal karta hai jo real users actually suffer karte hain.
n sorted samples ka p-th percentile kaise compute karte hain? Rank k = ceil(p/100 · n), phir k-th smallest value lo.
Error budget kya hai? Allowed failure ki maatra, jo (1 − SLO) × total requests (ya window time) ke barabar hai.
99.9% SLO monthly kitna downtime allow karta hai? (1 − 0.999) × 43200 min ≈ 43.2 minutes per month.
Alert mein "for X minutes" duration kyun add karte hain? Noisy one-off spikes suppress karne ke liye; condition sustained rehni chahiye, thodi detection delay ke badle bahut kam false pages.
Alert fatigue kya hai aur ise kaise avoid karein? Insaan alerts ko ignore karte hain kyunki bahut saare fire hote hain; fix: sirf actionable, symptom/budget-burn conditions par alert karo.
Symptom-based vs cause-based alerting — kaun sa preferred hai aur kyun? Symptom-based (jo users feel karte hain, jaise error rate), kyunki causes infinite hain lekin user-facing symptoms kam hain.
Ek cheez jo tumhe logs mein KABHI nahi daalni chahiye? Secrets / PII jaise passwords ya full card numbers.
Observability — woh umbrella property jo yeh teen pillars provide karte hainDistributed Tracing — correlation IDs ko full request spans mein extend karta haiSLI SLO SLA — service levels jinpar error budgets based hainPercentiles and Distributions — p95/p99 ke peeche statisticsTime Series Databases — jahan metrics store hoti hain (Prometheus, etc.)Incident Response — alert fire hone ke baad kya hota haiRate Limiting — same counter/rate machinery use karta haiPII and Data Privacy — isliye logs scrub karte hain
stitches request across services
Running service is a black box
Correlation ID / trace ID
Log levels DEBUG/INFO/WARN/ERROR
