Incidents & alerts

Alert channels

Channels are where your team, or your agent, hears about problems. One workspace can have up to 50 channels, each with its own event filter.

#Channel types

{
  "name": "Ops agent",
  "type": "webhook",
  "url": "https://agent.example.com/hooks/upbutler",
  "headers": { "x-tenant": "acme" },
  "events": ["monitor.*", "incident.*"]
}

{
  "name": "#ops",
  "type": "slack",
  "url": "https://hooks.slack.com/services/T000/B000/XXXX"
}

{
  "name": "Discord #alerts",
  "type": "discord",
  "url": "https://discord.com/api/webhooks/1234567890/abcdef"
}

{
  "name": "On-call Telegram",
  "type": "telegram",
  "botToken": "7712345678:AAH-example-bot-token",
  "chatId": "-1001234567890"
}

{
  "name": "On-call email",
  "type": "email",
  "emails": ["[email protected]", "[email protected]"],
  "isDefault": false
}

#Create a channel

curl -X POST https://upbutler.com/api/v1/channels \
  -H "Authorization: Bearer $UPBUTLER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "Ops agent", "type": "webhook", "url": "https://agent.example.com/hooks/upbutler"}'

{
  "_id": "ch_0n3q8kz3j6d0q2b5ny8",
  "name": "Ops agent",
  "type": "webhook",
  "config": { "url": "https://agent.example.com/hooks/upbutler", "secret": "whsec_MfKQ…" },
  "events": [],
  "enabled": true,
  "isDefault": true,
  "stats": { "ok": 0, "failed": 0 },
  "signingSecret": "whsec_MfKQ9r8GKYqrTwjUPD8ILPZIo2LaLaSw"
}

Store signingSecret: it is only returned on creation, and when you rotate it with PATCH /channels/:id {"rotateSecret": true}. Elsewhere it is masked. Invalid URLs or missing fields for the chosen type return 422.

#Choosing events

A channel with no events list receives the default alert events:

monitor.down monitor.up monitor.degraded monitor.reminder incident.created incident.updated incident.resolved maintenance.scheduled maintenance.started maintenance.updated maintenance.completed incident.acknowledged

Or pass an explicit list. Wildcards work:

• "*": every event that is delivered to channels, including component.status_changed. (monitor.created and monitor.deleted are only recorded in the event stream.)
• "monitor.*", "incident.*", "maintenance.*": everything under that prefix.

All event types are described in Events.

#Which channels get what

• Monitor events (monitor.down, monitor.up, monitor.degraded, monitor.reminder) go only to the channels attached to that monitor (its channelIds), and only if they want the event.
• Incident and maintenance events go to every enabled channel that wants them. For automatic incidents, your team already got monitor.down and monitor.up, so the automatic incident.created and incident.resolved go to subscribers only, not to channels. Manual incidents, updates and maintenance go to channels.
• Component changes (component.status_changed) go to channels that subscribe to them explicitly (or "*").

#Default channels

isDefault is true unless you say otherwise. A default channel is attached to all existing monitors when it is created, and to every new monitor created without explicit channelIds. Set isDefault: false for channels you want to attach by hand, such as a VIP email list for a single critical monitor (PATCH /monitors/:id with channelIds).

enabled: false pauses a channel without detaching it. Deleting a channel detaches it from every monitor.

#Test a channel

curl -X POST https://upbutler.com/api/v1/channels/ch_0n3q8kz3j6d0q2b5ny8/test \
  -H "Authorization: Bearer $UPBUTLER_API_KEY"
# → { "ok": true, "status": "sent", "error": null, "httpStatus": 200 }

Sends a test.ping through the real delivery path and reports what happened, including the HTTP status your endpoint returned.

#Reminders

While a monitor stays down, a monitor.reminder goes out at each of its reminderMinutes after the incident started (default [30, 120, 480]), with downForMinutes in the payload. Configure it per monitor. [] turns reminders off.

#Acknowledging, on-call and escalation

Alerts that someone should own (monitor.down, monitor.reminder, incident.created) carry an Acknowledge link in email, Slack, Discord and Telegram. Acknowledging (on every plan, also over POST /incidents/:id/ack) stops the reminders for that incident and tells the alerted channels with incident.acknowledged.

On the Business plan, escalation policies page whoever is on call on a weekly rotation, then the next level after N minutes if nobody acknowledges. These pages go to people (email and Telegram) and to channels, and are sent as incident.escalated. See On-call & escalation.

#Quotas and delivery

• Email is metered. Each email sent counts toward the monthly quota, whether it goes to a team recipient or a status page subscriber. Once the quota is used up, further emails that month are dropped. Usage is shown in GET /workspace.
• Webhooks, Slack, Discord and Telegram are unmetered.
• Failed deliveries are retried with backoff (30 s, 2 min, 10 min, 30 min, 1 h, 3 h, 6 h). Client errors other than 408 and 429 count as permanent and aren't retried. See Webhooks → Retries.
• Each channel keeps stats: ok, failed, lastDeliveryAt, lastError.