Which events deserve a message, and which deserve two

Every team that has wired Kubernetes events into Slack has the same channel: the one that was useful for three days, then filled with BackOff and Unhealthy and FailedScheduling during a routine rollout, then got muted. It is still there. Nobody reads it. The failure was not that the events were wrong; every one of them was true. The failure was that the channel could not tell a fact from a judgement, and treated an app being unhealthy for four seconds during a deploy the same as an app being unhealthy for an hour.

A deploy platform is in a position to do better, because it knows what it just did. That knowledge is worth four design decisions.

Facts are on by default, judgements are opt-in

A deploy succeeded or it failed. A cluster's agent has been unreachable for two minutes or it has not. These are facts about something that happened, they are true the moment they are emitted, and they belong on by default, because a person who set up a destination and hears nothing when a deploy fails has been let down.

"The app is degraded" is a different kind of statement. It is a judgement about a duration, and the duration is a choice. Five minutes is a defensible choice for an app; a restart backoff, a node drain and a rolling update all produce a window shorter than that in which the app is genuinely, temporarily, unhealthy. A judgement that fires inside that window teaches people to mute the channel, and once the channel is muted the facts are lost with it. So health events default to off, and the person who turns them on has been told what the threshold is.

Hold, then fire once

The condition has to hold for its delay before anything is sent. Then it is sent once. An app that is still degraded an hour later produces nothing further, because the second message carries no information the first did not, and a channel that repeats itself every five minutes is a channel with one message in it, repeated.

Recovery is the other half, and it has a rule of its own: exactly one closing message, naming how long the degradation lasted, and only if the degradation was announced. An app that dipped for ninety seconds and came back never crossed the threshold, so it never produced an alert, so it must not produce a recovery either. A recovery message with no alert before it is noise wearing a green tick.

Two messages about one failure, on purpose

The one deliberate exception. A deploy fails and leaves the app unhealthy. The platform sends deploy.failed at once, because that is a fact and the person who pushed wants to know now. Five minutes later, if the app is still unhealthy, it sends app.degraded. Two messages about one incident looks like a violation of the previous rule, and is not: the first says the deploy did not work, the second says the app is still not working, and those are different facts with different audiences. The person who pushed fixes the first. Whoever is on call needs the second.

The cap has to leave a trace

Any system that sends messages needs a ceiling per destination, or one bad loop empties the Slack webhook's rate budget and takes every other notification with it. Sixty an hour is generous for a channel a human reads. The detail that matters is what happens above the ceiling: a message dropped by the cap has to be recorded as dropped, with the reason, where the person can see it. A cap that discards silently turns "nothing arrived" into an unanswerable question.

The second-order detail is that the dropped rows must not count toward the cap. If they did, one burst would keep the destination suppressed for an hour after the burst ended, and the message that mattered would arrive at minute sixty-one.

What a webhook receiver is owed

For a webhook destination, the receiver is code someone else wrote, and it deserves the same contract every well-behaved sender offers. A signature over the body and a timestamp, so a forged request can be rejected and a captured one cannot be replayed. A stable event id in the body that is the same across retries, so a receiver that was down and gets the event again can deduplicate. A retry schedule that backs off far enough to survive a real outage: one minute, five, thirty, two hours, six hours, about nine hours in all. And a rule about what not to retry: a 4xx other than 429 means the destination is wrong, and retrying a wrong URL for nine hours helps nobody.

One more, easy to get wrong: the delivery row is written before the first attempt, not after. Otherwise an empty log means either "nothing was sent" or "the process died mid-send", and the person reading it cannot tell which.

How this works in Anyport

A destination is a Slack channel, a webhook or an email address, added under Settings → Notifications. The four defaults are deploy.succeeded, deploy.failed, cluster.disconnected (two minutes unreachable) and bandwidth.spike; app.degraded and service.degraded are opt-in and fire after five minutes. Each fires once per episode, with one recovered or reconnected message naming the duration, sent only if the alert was. A destination can be narrowed to particular projects, and sends are capped at 60 per destination per hour, with a dropped delivery recorded as suppressed in the log.

Webhooks carry X-Anyport-Signature-256, an HMAC-SHA256 over timestamp + "." + body keyed with a secret shown once and rotatable, plus the event type, a per-attempt delivery id and the signing timestamp. Retries follow the schedule above; a 4xx other than 429 is not retried and the row says why. Settings → Notifications → Deliveries keeps thirty days of attempts with the receiver's response, and anyport notify log --destination team-slack reads the same log from a terminal.

The documentation has the payload and the headers. The decision this post is about comes first: of everything your platform can see, which of it is a fact, which is a judgement, and how long does a judgement have to hold before it is worth interrupting someone.