1.7 KiB
Notifications runbook
Configure
Create a filtered email or webhook subscription through /api/v2/notifications/subscriptions. Webhooks require a write-only signing secret. Configure SMTP separately at /api/v2/notifications/email-settings; only authenticated STARTTLS SMTP is accepted. Verify a channel with POST /subscriptions/{id}/test and inspect delivery/attempt history before relying on it.
Filters are a nonempty set of exact catalog IDs or family wildcards such as execution.*; they may be narrowed by job IDs, repository IDs, or severity. The public catalog is live-events-only: every listed type is emitted by a currently available operation. Deferred channels and source capabilities have no catalog entries.
Rotate and recover
Rotate webhook keys using the signing-key rotate endpoint with an idempotency key and an explicit bounded overlap. Receivers must accept both signatures during overlap, then remove the old key after expiry. A recovery bundle deliberately excludes subscriptions, SMTP settings, signing secrets, event history, and delivery attempts. Reconfigure notifications after a fresh-host recovery.
Failure handling
The worker claims due deliveries with a lease. Transient errors enter bounded exponential retry; interrupted leases recover as retryable work and may send an event again. Inspect response class and redacted diagnostics in history. A terminal failed delivery can be retried manually once the destination is corrected. To stop outbound traffic, disable/archive subscriptions or stop the worker; do not delete outbox history. Rollback consists of disabling subscriptions and worker dispatch while retaining audit/outbox records for investigation.