An alert should represent something a human must act on.
Everything else belongs in dashboards, logs, reports, or backlog work. Quiet alerting is not less observability. It is better routing of attention.
The failure mode is easy to recognise and hard to admit. A team wires up monitoring, sets thresholds that feel prudent, and within a month the on-call channel is producing forty notifications a day. Nobody reads them. When the real incident arrives, it lands in the same channel, styled the same way, and scrolls past.
The system did not fail to detect the problem. It failed to make the problem visible.
The One Question
Before anything becomes an alert, it has to survive a single question:
If this fires at 3am, is there something a human should do about it right now?
Not “would this be interesting to know.” Not “could this eventually matter.” Right now, at 3am, by a tired person who has to decide in thirty seconds whether to open a laptop.
If the honest answer is no, it is not an alert. It is telemetry.
That question eliminates most of what teams instinctively page on. CPU at 85%. A single 500. A pod restart. A queue depth spike that drains on its own. Each of these is worth recording. None is worth waking someone.
Four Destinations, Not One
The mistake is treating “we should know about this” as a binary that resolves to either page or nothing. There are four places a signal can go, and only one of them interrupts a person.
- Page. Wakes someone. Reserved for user-visible harm that is ongoing and needs a human decision.
- Ticket. Enters the backlog with a due date. For degradation that is real but not urgent — a certificate expiring in three weeks, a disk trending toward full.
- Dashboard. Available when someone looks. For capacity, cost, throughput, and everything you want during an investigation but not before one.
- Log. Recorded, queryable, silent. For the long tail you will only ever read after the fact.
Most noisy alerting is a routing failure. The signal was correct; the destination was wrong.
flowchart TD
A[Signal] --> B{User-visible harm,<br/>happening now?}
B -->|Yes| C{Can a human<br/>act on it?}
B -->|No| D{Will it become<br/>harm soon?}
C -->|Yes| E[PAGE]
C -->|No| F[Fix the automation,<br/>not the alert]
D -->|Yes| G[TICKET]
D -->|No| H{Useful during<br/>an investigation?}
H -->|Yes| I[DASHBOARD]
H -->|No| J[LOG]Alert On Symptoms, Not Causes
Cause-based alerts multiply. Symptom-based alerts do not.
There are a hundred reasons checkout might break: the database is slow, the payment provider is down, a deploy shipped a bad query, a cache is cold, a dependency is rate-limiting you. If you alert on each cause, you maintain a hundred rules, and you still miss the hundred and first.
Alert on checkout success rate has dropped below its threshold. One rule. It catches every cause, including the ones you have not thought of yet.
The causes still belong in your telemetry — that is how you diagnose once you are awake. But they are the investigation, not the interruption.
This is also why the best alerts map to things a customer would complain about. If no user would notice, no user is being harmed, and the 3am question answers itself.
Every Alert Needs An Owner And A Response
An alert with no documented response is a notification with extra anxiety.
We require two things before a rule goes live:
- A named owner. Not a team alias — a person who is accountable for whether this rule earns its place.
- A runbook entry. What this means, what to check first, what the likely fixes are, and what “resolved” looks like.
If nobody can write the runbook, the alert is not understood well enough to page on. That is a useful forcing function. It surfaces monitoring that was added defensively rather than deliberately.
The Alert Budget
Treat alerts like a budget with a hard ceiling.
A rough working number: an on-call shift should produce no more than one or two pages per week, and every one of them should have been worth it. If you are consistently above that, the system is not noisier than it should be — the rules are.
When a new alert is proposed and the budget is full, something has to be removed. That constraint does more for signal quality than any tooling decision. It forces the conversation about which failures actually matter.
Review What Fired
Every page gets a two-minute retrospective, even the ones that resolved themselves.
- Did a human need to act?
- Did the runbook match reality?
- Could this have been detected earlier, or prevented entirely?
- Should this rule be tightened, rerouted, or deleted?
That last option matters most and gets used least. Deleting an alert feels like reducing safety. Usually it increases it, because the remaining alerts get read.
We also treat repeatedly self-resolving as a defect rather than a comfort. If a condition reliably clears on its own, either the system should heal it without a human, or the threshold is wrong. A page that trains people to wait five minutes and see is a page that trains people to ignore.
Quiet Is Not The Same As Blind
The point is not fewer signals. It is fewer interruptions.
A quiet system usually has more instrumentation than a noisy one — richer dashboards, better traces, more structured logs, tighter SLOs. What it does not have is the assumption that every measurement deserves a human’s attention.
Observability answers questions. Alerting decides who gets woken. Conflating the two produces systems that are simultaneously over-monitored and under-observed.
The Bottom Line
A page that fires constantly is not a safety net. It is background noise with a pager attached.
Alert on user-visible harm. Route everything else to a ticket, a dashboard, or a log. Give every rule an owner and a runbook. Keep a hard budget and spend it deliberately.
The measure of good alerting is not how much you detect. It is whether the one page that mattered this quarter got read.