Alert on Symptoms, Not Causes¶
Alerts should fire on user-visible symptoms (high latency, errors) not internal causes (CPU usage, queue depth) - symptoms indicate actual impact while causes often self-resolve or lack user impact.
Tension Addressed¶
Automation vs. Human Judgment - symptom alerts preserve human attention for real impact
Recognizable As¶
Alerts on SLO breach vs. resource thresholds; "users are affected" vs. "something looks wrong"; multi-signal alerting requiring impact evidence
Canonical Sources¶
Google SRE Book, Chapter 11 "Being On-Call" (2016); "My Philosophy on Alerting" (Rob Ewaschuk)
Transfers To¶
Medicine (symptom-based triage), Quality management (customer-facing metrics), Business operations (outcome metrics)
High CPU is not inherently a problem; slow user requests are. Alerting on causes creates noise and alert fatigue; alerting on symptoms ensures pages represent real user impact.