How to design alerts and notifications for dev teams

​

Overview

A notification is a mechanism that shortens the gap between “something happened” and “someone knows.” A failed deploy, a newly appearing production error, and an unread review request are all cheap to detect and expensive to miss, so the delay before anyone notices becomes the bottleneck. This page covers why noticing quickly is the goal, how to decide what deserves a notification, how to handle alerts and the results of automated runs, how to notify and remind about development workflow events, and how to keep a channel worth reading.

​

Notice fast, act fast

Driving defects and outages toward zero is worth continuous effort. Test coverage, review, and staged rollout all lower the rate at which failures reach production. But lowering a rate is not the same as reaching zero. There is exactly one way to reach zero reliably, and that is to write no code at all. A system that never changes produces no change-induced failures, and it delivers no value either. Zero failures is a target that can only be met in exchange for shipping nothing. So the goal is not to eliminate failures. It is to notice the ones that occur, and recover from them, as quickly as possible. Restoration cannot begin before someone knows there is something to restore. The interval from failure to awareness is mean time to detect (MTTD); the interval from failure to restored service is mean time to recovery (MTTR). Both are counted from the moment the failure occurs, which makes MTTD a segment inside MTTR. MTTD can exceed the time the fix itself takes. A defect fixed in ten minutes but noticed after two days caused two days of harm. This reframes what counts as the problem. The failure itself is an event with a bounded cost. The failure that nobody notices is a cost that accrues for as long as it goes unseen: transactions keep failing, a batch keeps writing bad data, users keep leaving. Thresholds and notifications exist to compress the interval during which that accrual is invisible. The same structure applies outside incidents. While a review request goes unnoticed, the lead time of the change keeps growing. Here too, the problem is not that a review is needed — it is that nobody yet knows a review is needed.

​

Deciding what to notify

Every notification spends the attention of everyone in the channel. That cost is paid whether or not the message turns out to matter, so routing an event is a design decision rather than a default. Three questions determine whether an event belongs in a channel. Events that survive all three questions are worth notifying. The rest — routine successes with no owner, informational state changes, anything that fires many times an hour — belong in a dashboard or a digest instead.

​

Alerts

This section covers the notifications a running system produces. The central case is incident response: wire monitoring tools so their alerts arrive in chat the moment a problem appears. Alongside alerts for abnormal states, this family also includes notifications that should be sent when nothing is wrong, such as the outcome of an automated run.

• Error tracking → chat. Application exceptions and newly appearing error types post to a dedicated channel, so whoever picks one up starts triaging instead of hunting for the stack trace.
• Infrastructure monitoring → chat. Monitors for load, latency, and resource pressure fire when the system is under stress, surfacing degradation before it cascades into an outage.
• Service level objective (SLO) monitoring → chat. The deviation the objective permits is the error budget. Tracking how fast it is being spent surfaces the trend while there is still room, rather than once the objective is missed.
• Cost and security checks → chat. A sudden change in cloud spend, or the result of a security scan, is another signal whose cost grows the longer it goes unseen.

​

Route by severity

Not every abnormal state needs someone now; some can wait until the next working day. Sending everything to one channel at one intensity buries the urgent ones. The test is how the failure affects continuity of service. If one process dies in a redundant setup, the remaining ones absorb the work and a replacement starts on its own, so recording a warning is enough. If a batch job with only one process exits abnormally, the work stops at that moment and someone has to act. Once severity is decided, the destination and the strength of the mention follow from it. Urgent cases call the owner directly; warnings go to a channel kept for the record.

​

Treat a false positive as a defect

An alert that fires on a condition the team has decided to ignore is worse than no alert, because it teaches people to skim past the channel. When a monitor produces false positives, the fix is to adjust the firing condition — the threshold, the time window it is evaluated over, or how related alerts are grouped — not to mute the channel. Tuning runs in both directions, and the two errors are not symmetric. A threshold loose enough to never produce a false positive will also miss real degradation, so the target is the level at which a firing monitor reliably means something is wrong, revisited as traffic and architecture change.

​

Report both outcomes

For anything that runs without a person driving it — a nightly batch, a backup, a data sync, a CI run, a deploy — report the outcome either way. The reason for notifying differs by target. CI results decide whether the team’s own work can proceed, so notify on the runs that matter to the team. Deploy results bear directly on users, so notify regardless of outcome. A deploy does not necessarily run from CI, so treat the two as separate notifications. Notifying only on failure looks economical, and it introduces an ambiguity that cannot be resolved from the channel. When no message arrives, you cannot tell whether every run succeeded, whether the job never ran, or whether the notification path itself is broken. The latter two are exactly the failures that otherwise go unnoticed for days — the situation the whole setup exists to prevent. A success message removes the ambiguity, because it is positive evidence that the job ran and that the path from the job to the channel works. Silence then carries information of its own: a success message missing at the expected time is itself worth investigating. For how this is configured in a pipeline, see CI/CD.

​

Development workflow notifications

Outside incidents, the events worth routing are the ones where someone is waiting on someone else. A review request, a question in an issue comment, and a requested change all block a change until a specific person responds. While that wait lasts, what sits on the critical path is the interval in which the reviewer does not yet know. A review request noticed half a day late extends the pull request’s lead time by half a day. No amount of faster implementation or faster CI shortens that interval. Off-the-shelf integrations cover this, but their default configuration usually notifies far more than the events above. When a channel is too noisy to read, the options are to narrow the integration’s event and repository filters, or to replace it with a custom integration that posts only the events that need a response, in a short form. A message that is acted on tends to contain four things:

• The person expected to act, as an actual mention rather than a name in text.
• What is being asked, in one line.
• A link that opens the exact place to act.
• Enough context to decide priority, such as the age of the request.

​

Remind, do not only announce

A single notification is a single chance to be seen. A review request that arrives while its reviewer is in a meeting is effectively never delivered, which is why the announcement needs a periodic counterpart: a scheduled job that lists what is still unhandled. The same pattern applies wherever an item has an owner and a deadline: list what is still unhandled, post it on a schedule, and post nothing when the list is empty. Silence then carries information too.

​

Keeping a channel worth reading

A notification setup decays: monitors accumulate, integrations are added and never removed, and the volume creeps up until the channel is skimmed rather than read. Three practices keep it usable.

• Give every channel an owner and a purpose. A channel that anyone may route anything into has neither, and it degrades fastest.
• Review what actually fired. Periodically look at which alerts fired and which were acted on. A monitor that never leads to action is a candidate for removal or retuning.
• Mention precisely. Broadcast mentions to an entire channel are noticed the first few times and ignored afterwards. Mention the person or group expected to act.

​

Related pages