Incidents & alerts

Incident room

UpButler is not a chat product. The incident room is where an incident is worked and recorded, and it stays in step with the thread your team already has open.

#What is in the room

Open an incident in the dashboard (/app/incidents/<id>). The page is the room:

• One timeline: public updates, internal notes, what agents did (claims, verify results, pull requests, rollbacks, stopped runs), checks flipping, deploys, the upstream provider's incident, and the room's own events (roles, tasks, approvals), merged by time, with filters. Every entry has a stable id and is a link.
• Live: the page follows a Server-Sent Events stream and updates as things happen. No websocket; a proxy that buffers streams falls back to a refresh every few seconds.
• Who is here: members who have the room open right now.
• Roles: incident commander, communications, operations, each a member or an agent responder.
• Tasks: the list items of the runbooks that apply to the incident, plus tasks you add. Agents tick them.
• Needs approval: an agent asks before it acts; a person approves or denies.
• Catch me up and Ask about this incident.

curl https://upbutler.com/api/v1/incidents/inc_…/room \
  -H "Authorization: Bearer $UPBUTLER_API_KEY"
# → { "incident": {…}, "timeline": [ { "id": "upd_…", "at": "…", "kind": "note", "text": "…",
#       "author": { "name": "Dana", "kind": "person" }, "internal": true, "mentions": […] }, … ],
#     "room": { "roles": { "commander": {…}, "comms": null, "ops": null }, "tasks": […], "approvals": […] },
#     "presence": […], "catchUp": {…} | null, "sig": "1760090000000.7.identified.12" }

# Wait for the next change (long poll, up to 50 s). Browsers use the SSE stream instead:
curl "https://upbutler.com/api/v1/incidents/inc_…/room?since=1760090000000.7.identified.12&wait=50" …
curl -N https://upbutler.com/api/v1/incidents/inc_…/room/stream …   # event: ready, then event: change

#Notes and mentions

An internal note in the composer is a room note. @handle mentions a member of the workspace: their email, the part before the @, their name without spaces, or their first name when it is unique. A mentioned member is paged the way they asked to be reached in their contact details: email, plus push and Telegram where set up.

curl -X POST https://upbutler.com/api/v1/incidents/inc_…/room/notes \
  -H "Authorization: Bearer $UPBUTLER_API_KEY" -H "Content-Type: application/json" \
  -d '{"message": "@dana the 502s start right after deploy 3f2c1aa. Rolling back.",
       "links": [{"url": "https://grafana.example.com/d/api", "label": "API dashboard"}],
       "share": true}'

• A handle that fits nobody, or more than one member, mentions nobody and stays text. An address of somebody outside the workspace cannot be reached through a note.
• Attachments are links (links, http or https). UpButler hosts no files.
• Two acknowledgements, on purpose: 👀 "seen, looking" and ✅ "done / agreed".
• share: true ("Share to chat") also posts the note to the chat channels that were alerted about the incident.

#Tasks from runbooks

When the room is first opened, every list item (- …, * …, 1. …, - [ ] …) of the runbooks attached to the incident's monitors and components, and of the workspace default, becomes a task. Editing a runbook later adds its new items; nothing ticked or added by hand is removed. Ticking, reopening and adding tasks are on the timeline with the name of who did it.

#Agents ask before they act

An agent responder can ask a human to decide: POST /incidents/:id/approvals with what it wants to do and why. The incident commander is paged with a link to the room; with no commander, whoever acknowledged; else the people on call; else the alert channels of the incident. In the room the request has Approve and Deny and room for a note. The agent waits on GET …/approvals/:id?wait=50.

# With the incident token of the page it was paged with (or its responder key):
GET   /api/v1/incidents/{id}/room                     the room; ?since=&wait= to wait for changes
POST  /api/v1/incidents/{id}/notes                    a note (mentions and the chat thread apply); MCP: incidents_note
PATCH /api/v1/incidents/{id}/tasks/{taskId}           {"done": true}
POST  /api/v1/incidents/{id}/approvals                {"title": "Roll back api to 1.4.1", "detail": "…"}
GET   /api/v1/incidents/{id}/approvals/{approvalId}?wait=50   → state: pending | approved | denied | expired

#Catch me up, and Ask

Catch me up writes what happened, what was tried, the current hypothesis and the open tasks, from the timeline and the task list, following the status page's AI guidelines. A new summary counts as one AI report of the month; the stored one is returned, free, until the timeline, a task or a decision changes. With AI off or the quota used up you get the same four parts built from the records, marked as such.

Ask about this incident answers a question from the incident's own context (timeline, evidence, runbooks, tasks) and cites the timeline entries the answer rests on; citations that name no real entry are dropped, and the model is told to say when the context does not hold the answer. One AI report per question, 20 questions per person and hour, team only.

#The chat thread

#Who wrote it

A Slack reply is attributed to the workspace member with the same email address as the Slack user. Anyone else is shown as Slack: name; their note is recorded, and a mention in it pages nobody. A note that came from the thread is not posted back to it.

#Connect Slack

1. Settings → Integrations → Slack → Connect Slack (owners and admins). Slack shows what the app asks for.
2. In Slack, invite the app to the channel for incidents: /invite @UpButler.
3. Back in Integrations, paste the channel id (channel name → About → at the bottom, C0123ABCDEF). UpButler posts a hello message there.

The app asks for chat:write, channels:history, groups:history, users:read, users:read.email: post and reply; receive the messages of channels it was invited to (that is how thread replies arrive); read the email address of a Slack user to match them to a member. It reads no channel it was not invited to and no direct messages. A Slack workspace connects to one UpButler workspace.

#Running your own UpButler: register the Slack app

On upbutler.com this is done. On your own installation the operator (an address in OPERATOR_EMAILS) registers the app once at /app/admin/slack-app:

1. Create on Slack opens Slack's "create an app from a manifest" screen with the manifest below filled in. Pick the workspace that should own the app and press Create.
2. On the app's Basic Information page copy App ID, Client ID, Client Secret and Signing Secret into the form. They are stored encrypted and never shown again.
3. If other Slack workspaces will install it: Manage Distribution → activate public distribution.

Or paste an app configuration token (api.slack.com/apps → Your App Configuration Tokens; valid 12 hours, not stored) and UpButler creates the app through Slack's manifest API and stores what Slack returns.

{
  "display_information": { "name": "UpButler", … },
  "features": { "bot_user": { "display_name": "upbutler", "always_online": true } },
  "oauth_config": {
    "redirect_urls": ["https://<your host>/app/settings/slack/callback"],
    "scopes": { "bot": ["chat:write","channels:history","groups:history","users:read","users:read.email"] }
  },
  "settings": {
    "event_subscriptions": { "request_url": "https://<your host>/hooks/slack", "bot_events": ["message.channels","message.groups"] },
    "socket_mode_enabled": false
  }
}

Events arrive at POST /hooks/slack. Each is verified with the signing secret (X-Slack-Signature: HMAC-SHA256 over v0:<timestamp>:<body>) before anything is parsed, a timestamp more than five minutes off is refused, an event id is handled once, and the UpButler workspace comes from our record of the Slack workspace, never from the payload.

#Viewers of an internal page

On an internal status page, the incident page is the room as far as the page's sections allow: public updates always; team notes, roles, tasks, checks and deploys when their section is on. Signed-in viewers can write notes and acknowledge (switch: "Viewers can write notes"); a viewer's mention pages nobody; shared links are read-only. Catch me up shows there once a member has written it; Ask, roles, tasks and approvals are for the team in the dashboard.

#After the incident

The room stays as the record. The postmortem draft is written from the same timeline: notes, who held which role, tasks done, and what was approved or denied are in it.