Observability¶
Tier: 3 (Applied Domains)
Concepts: 14
Durability: Durable (Observability principles predate modern tooling; instrumenting systems for understanding traces back to control theory and early distributed systems research)
Fundamental Tensions¶
| Tension | Description |
|---|---|
| Coverage vs. Cost | Every measurement has storage, processing, and attention costs. Complete observability is economically impossible; you must choose what to instrument. This is fundamentally a resource allocation problem that cannot be eliminated. |
| Detail vs. Noise | More granular data enables deeper investigation but increases cognitive load and obscures signal. High cardinality provides precision but creates exponential storage growth and analysis difficulty. |
| Reactivity vs. Proactivity | Monitoring reacts to known failure modes; true observability requires instrumentation before you know what questions you will ask. Investment in observability competes with feature development. |
| Availability vs. Reliability | 100% reliability is infinitely expensive; error budgets formalize the trade-off between shipping velocity and stability. This tension is mathematical: diminishing returns on reliability investment. |
| Automation vs. Human Judgment | Automated responses provide speed but lack context. Human judgment provides nuance but doesn't scale. Novel failures require human reasoning; recurring failures demand automation. |
Concepts¶
- The Three Pillars: Complementary System State Representations
- Cardinality as Observability's Resource Constraint
- Service Level Hierarchy: SLI, SLO, SLA
- Error Budgets: Quantifying Acceptable Failure
- Monitoring vs. Observability: Known-Unknowns vs. Unknown-Unknowns
- Context Propagation: Causality Across Boundaries
- Sampling: Trading Completeness for Feasibility
- Structured Logging: Machine-Parseable Events
- Alert on Symptoms, Not Causes
- Alert Fatigue: The Attention Depletion Problem
- Blameless Post-Mortems: Learning Over Punishment
- Mitigation Before Root Cause
- Chaos Engineering: Proactive Failure Injection
- Runbooks: Codified Incident Response
Cross-Domain Transfer Summary¶
| This Domain Concept | Transfers To |
|---|---|
| Three Pillars | Scientific measurement, Business intelligence, Network analysis |
| Cardinality | Database indexing, Data warehousing, Machine learning |
| SLI/SLO/SLA | Quality assurance, Manufacturing, Contract law |
| Error Budgets | Financial risk, Project management, Safety engineering |
| Monitoring vs. Observability | Scientific method, Debugging, Security |
| Context Propagation | Supply chain, Audit systems, Distributed systems |
| Sampling | Statistics, Quality assurance, A/B testing |
| Structured Logging | Data engineering, Event sourcing, API design |
| Symptom-based Alerts | Medicine, Quality management, Business operations |
| Alert Fatigue | Healthcare, Security, Information management |
| Blameless Post-Mortems | Aviation, Medicine, Safety engineering |
| Mitigation Before Root Cause | Emergency medicine, Firefighting, Crisis management |
| Chaos Engineering | Security testing, Military exercises, Disaster preparedness |
| Runbooks | Aviation checklists, Clinical protocols, Emergency SOPs |
Canonical Sources¶
| Source | Coverage |
|---|---|
| Beyer et al., "Site Reliability Engineering" (2016) | SLIs/SLOs, error budgets, incident response, alerting philosophy, on-call practices |
| Sridharan, "Distributed Systems Observability" (2018) | Three pillars distinction, observability vs monitoring, structured logging |
| Majors, "Observability Engineering" (2022) | Modern observability practices, high cardinality, debugging with observability |
| Sigelman et al., "Dapper" paper (2010) | Distributed tracing foundations, context propagation, sampling |
| Rosenthal et al., "Chaos Engineering" (2017) | Chaos engineering principles, failure injection, resilience testing |
| Dekker, "The Field Guide to Understanding Human Error" (2006) | Blameless culture, systems thinking in incidents, just culture |
| Cook, "How Complex Systems Fail" (1998) | Systemic failure analysis, multiple contributing factors |
| Allspaw, "Blameless PostMortems" (2012) | Post-mortem practice, learning from incidents |
| Hidalgo, "Implementing Service Level Objectives" (2020) | Practical SLO implementation, error budgets |