Monitoring

Spend guard

A runaway agent or a retry loop costs money for hours before anyone opens a billing page. Spend guard adds up what you spend per day, compares it with what is usual for you, and opens an incident that names who is spending.

#What it is

Two things, both in the dashboard under Spend and in the API:

• The Spend page (GET /spend): spend per source and UTC day, month to date, what is usual for each source, the days that were unusual, and the agent runs spending the most today. Readable on every plan.
• Spend monitors (monitor kind spend): a normal monitor that adds up the sources you give it and goes degraded or down on a daily cap, a monthly budget or unusual spend. Because it is a normal monitor, it opens incidents, alerts your channels, gets an AI report and can be handed to an agent responder.

Amounts are US dollars. Days are UTC days, because that is how every provider it reads from buckets its bill: a daily cap resets at 00:00 UTC, not at your local midnight.

#The three kinds of source

A day a source has no record for is treated differently per kind. For Agent Runs and balances, which UpButler records itself, a day without a record after tracking began means nothing was spent: it is $0. For a provider account, a day that was not read is unknown: charts leave a gap, and the baseline skips it.

#Sources can overlap

So the Spend page never shows a grand total across sources: numbers are per source. A spend monitor adds up exactly the sources in its sources list, and picking ones that do not overlap is up to you. With sources empty the monitor chooses automatically, in a way that cannot double count the usual case:

• when at least one provider account is connected (and readable on your plan): the provider accounts only;
• otherwise: Agent Runs and every USD balance monitor.

#Connecting a provider account

In the dashboard: Spend → Connect a provider account. Over the API: POST /spend/sources. Owners and admins only. The key is tried against the provider before anything is saved, so a source that exists has worked at least once; a key that is refused answers 409 with the provider's reason.

Every request UpButler makes with these keys is a GET. It reads cost reports and nothing else: no completions, no deployments, no changes.

#OpenAI

• Key: an organization Admin API key (sk-admin-…). Normal project keys (sk-proj-…) cannot read costs and are refused with 403.
• Where: platform.openai.com → Settings → Organization → Admin keys. Only an organization owner can create one.
• Endpoint: GET /v1/organization/costs with daily buckets, grouped by line_item and project_id. (OpenAI reference)
• Breakdown: line item (model and usage type), under its project.

#Anthropic

• Key: an Admin API key (sk-ant-admin01-…). Workspace API keys (sk-ant-api03-…) cannot read costs.
• Where: Claude Console → Settings → Admin keys, by a member with the admin role. Organization accounts only: an individual account has no Admin API, so there is nothing to connect.
• Endpoint: GET /v1/organizations/cost_report with daily buckets, grouped by workspace_id and description. The report states amounts in cents as decimal strings; UpButler converts them to dollars. (Anthropic reference)
• Breakdown: model (or cost description), under its workspace.

#OpenRouter

• Key: a management key (formerly "provisioning key"). It cannot make completions; it reads activity and credits. A normal sk-or-v1-… inference key is refused.
• Where: openrouter.ai → Settings → Management keys.
• Endpoints: GET /api/v1/activity for the last 30 finished UTC days per model, and GET /api/v1/credits for today: the account's total usage now, minus where it stood when the UTC day began. (OpenRouter reference)
• Breakdown: model, under the upstream provider, for finished days. Today is one number without a breakdown until the day is over.

#Vercel

• Key: an access token scoped to the team, plus the team id (team_…) or slug as teamId.
• Where: Vercel → Account Settings → Tokens, created by a member whose role can see billing (Owner, Member, Developer, Security, Billing or Viewer).
• Endpoint: GET /v1/billing/charges, which returns FOCUS v1.3 charges as JSONL. (Vercel reference)
• What counts: metered usage only (ChargeCategory: "Usage"). Plan fees and credit purchases are left out: they would look like a spike on the day they are billed.
• Finished days only: today stays empty and the source is judged on yesterday. A daily cap therefore cannot fire on Vercel spend during the day; a budget and the baseline can.
• Breakdown: service (the FOCUS ServiceName), under its project.

#How keys are kept

• Sealed with AES-GCM before they are stored, with the same mechanism as monitor secrets.
• Write-only: no endpoint, export or audit entry returns a key. The API shows keyHint (its last four characters) so you can tell keys apart.
• Replace a key with PATCH /spend/sources/:id {"key": "…"}: the new one is tried first and the old one stays if it fails.
• Disconnecting deletes the key and the spend history of that source. Connecting, replacing and disconnecting are in the audit log.

Polling: every 15 minutes, re-reading the last 3 days each time because providers restate recent days. The first read goes back about a month, so the baseline has history from the start. After a failed read the next try is in 30 minutes; a source that has not answered for 6 hours makes its monitors degraded with the provider's error, because a monitor that cannot see is not "up".

#What is not supported, and why

Supabase, Railway, Netlify and Fly.io cannot be connected. None of them has a documented endpoint that reports cost (Supabase's Management API, for example, has add-ons and request counts but no spend). UpButler does not scrape dashboards or guess prices from usage counters, because a wrong dollar figure is worse than none.

What works for them instead:

• a credit balance monitor on any JSON endpoint that returns a prepaid balance in dollars: it becomes a spend source by itself;
• Agent Runs budgets, when the cost comes from what your agents do there;
• the provider's own spend limit or budget alert, where it has one.

#The rules

A spend monitor is evaluated every 5 minutes. It needs at least one of: a daily cap, a monthly budget, anomaly detection switched on.

The monitor recovers by itself: a cap at the next UTC midnight, a budget at the start of the next month or when you raise it, unusual spend when the source is back in its range.

#The month-end projection

Month to date, plus the recent daily rate for the rest of today and the days left. The rate is the mean of the last 7 finished days, or of the month so far when fewer are known. A spike that cost money stays in the rate: it is a forecast of money, not of typical days. With fewer than 3 finished days nothing is projected and only the "used up" half of the budget rule can fire.

#Unusual spend

Each source is judged on its own, against its own history. No total is involved, so overlap does not matter here.

• Baseline: the median of the previous 14 days that have data (baselineDays, 7 to 30), and how much those days normally vary, measured with the median absolute deviation (MAD). Median and MAD are used instead of mean and standard deviation so that one earlier spike in the window neither hides a new one nor makes ordinary days look quiet.
• Needs 5 days: with fewer days of data the source is "learning" and nothing is flagged.
• Threshold: median + sensitivity × one step of normal variation. One step is the largest of: 1.4826 × MAD, 15% of the median (a perfectly flat history has no MAD, and $10.00 → $10.40 is not news), and half the distance from the median to the 90th percentile (so a workload with regular busy days is measured by its busy days).
• sensitivity (1.5 to 10, default 3.5): lower flags more, higher flags less.
• minUsd (default 5): the threshold is never closer to the median than this many dollars. A jump from $0.40 to $2.00 is five times the usual and still not worth an alert.
• Twice as far out (median + 2 × that distance) is severe: down instead of degraded.
• Same hour for hourly sources: Agent Runs and balances are compared mid-day with what earlier days had spent by the same UTC hour, so 09:00 is compared with other mornings and a normal morning is not "low" against whole days, nor a bad one hidden until evening. Provider accounts report days: today's running total is compared with whole earlier days, which means a provider anomaly fires once today has already passed an unusual full day.
• Steady growth is not flagged: a slow climb moves the median along with it. Spend guard catches jumps; the monthly budget is the rule for "we are simply spending more".

The Spend page shows each source's usual amount and its status (learning, normal, unusual), and marks the days that would have been flagged on the chart.

#Alerts, and stopping the spend

A spend incident is a normal incident. What its alerts add:

• The reason in dollars: "Daily cap reached: $52.10 spent today (cap $50)", or "Anthropic: $31.40 so far today, 3.5× the usual $9.10".
• Top contributors: the models, line items, projects or agents behind the day that tripped the rule, largest first, each with its source. The same list goes into the AI report and into the handoff packet for agent responders, along with the agent runs active today by spend.
• A Stop button, when agent runs are running and spending: it sets the kill switch on the runs of the agent monitor that is spending the most today. Like every alert action it opens a confirm page first; nothing is stopped until the button there is pressed. The runs are told to stop in the answer to their next ping.

#Automatic stop (opt-in)

With autoStopRuns: true, reaching the daily cap sets the kill switch on running agent runs without waiting for a person. It is off unless you switch it on. Which runs are stopped depends on what the monitor adds up:

With sources left empty the monitor picks provider accounts whenever one is connected, which is the second row. List "runs" explicitly if you want only the spending agents stopped. In both cases:

• only the daily cap triggers it: a line you drew yourself. A projected budget or unusual spend never stops anything;
• runs a person has resumed after a stop are left alone;
• only scheduled evaluations stop runs. Test now in the dashboard (POST /monitors/:id/check) and an incident's verify step return the same verdict and evidence but are read-only: they never stop a run and do not change the numbers the Spend page shows;
• the alert says how many runs were told to stop, and each run's timeline records why.

#API

#Connect a provider account

curl -X POST https://upbutler.com/api/v1/spend/sources \
  -H "Authorization: Bearer $UPBUTLER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"provider": "anthropic", "name": "Anthropic (production org)", "key": "sk-ant-admin01-…"}'
# 201 → { "_id": "sps_…", "key": "provider:sps_…", "provider": "anthropic", "name": "Anthropic (production org)",
#         "keyHint": "x9Qa", "hasKey": true, "enabled": true, "lastOkAt": "2026-10-10T09:20:51.204Z", … }
# 409 → the provider's reason, e.g. "Anthropic refused this key for cost data (HTTP 403). It needs an admin / management key, not a normal API key."
# 402 → plan_limit, with error.details.upgrade naming the plan that includes it

Vercel needs the team:

curl -X POST https://upbutler.com/api/v1/spend/sources \
  -H "Authorization: Bearer $UPBUTLER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"provider": "vercel", "key": "vcp_…", "teamId": "team_a1B2c3D4"}'

# Read the cost report now instead of waiting for the next poll
curl -X POST https://upbutler.com/api/v1/spend/sources/sps_0n3q8jz4k7m2p9x1c5v/sync -H "Authorization: Bearer $UPBUTLER_API_KEY"

# Rename, replace the key, or pause polling (the history stays)
curl -X PATCH https://upbutler.com/api/v1/spend/sources/sps_0n3q8jz4k7m2p9x1c5v \
  -H "Authorization: Bearer $UPBUTLER_API_KEY" -H "Content-Type: application/json" \
  -d '{"enabled": false}'

# Disconnect: deletes the key and this source's history, and takes it out of every spend monitor
curl -X DELETE https://upbutler.com/api/v1/spend/sources/sps_0n3q8jz4k7m2p9x1c5v -H "Authorization: Bearer $UPBUTLER_API_KEY"

#Create a spend monitor

curl -X POST https://upbutler.com/api/v1/monitors \
  -H "Authorization: Bearer $UPBUTLER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "AI spend",
    "kind": "spend",
    "spend": {
      "sources": ["provider:sps_0n3q8jz4k7m2p9x1c5v"],
      "dailyCapUsd": 50,
      "monthlyBudgetUsd": 800,
      "anomaly": {"enabled": true, "sensitivity": 3.5, "minUsd": 5},
      "autoStopRuns": false
    }
  }'

Fields of the spend object. The other monitor fields (name, channelIds, tags, …) are the same as for every monitor; PATCH /monitors/:id takes the same object, and null clears a cap or a budget.

#Read spend

curl "https://upbutler.com/api/v1/spend?days=30" -H "Authorization: Bearer $UPBUTLER_API_KEY"

{
  "currency": "USD", "timezone": "UTC", "today": "2026-10-10",
  "days": ["2026-09-11", "…", "2026-10-10"],
  "sources": [{
    "key": "provider:sps_0n3q8jz4k7m2p9x1c5v", "label": "Anthropic (production org)", "kind": "provider", "provider": "anthropic",
    "resolution": "daily", "locked": false, "since": "2026-09-06",
    "todayUsd": 31.4, "mtdUsd": 212.75,
    "series": [null, null, 18.2, 19.05, "…", 31.4],
    "baseline": { "status": "anomaly", "day": "2026-10-10", "usd": 31.4, "median": 9.1, "threshold": 24.6, "days": 14 },
    "anomalies": [{ "day": "2026-10-10", "usd": 31.4, "threshold": 24.6, "severe": false }],
    "top": [{ "label": "claude-sonnet-5-5", "group": "wrkspc_01…", "usd": 27.9 }, { "label": "claude-haiku-4-5", "usd": 3.5 }]
  }],
  "runs": [{ "id": "run_…", "monitorId": "mon_…", "monitor": "Support triage fleet", "agent": "triage-bot-3", "task": "Triage ticket #4812", "usd": 6.2, "running": true }],
  "monitors": [{
    "id": "mon_…", "name": "AI spend", "state": "degraded", "sources": [], "resolvedSources": ["provider:sps_0n3q8jz4k7m2p9x1c5v"],
    "dailyCapUsd": 50, "monthlyBudgetUsd": 800,
    "last": { "at": "2026-10-10T14:05:00.000Z", "todayUsd": 31.4, "mtdUsd": 212.75, "projectedUsd": 640.1,
              "reasons": [{ "kind": "anomaly", "severity": "degraded", "message": "Anthropic (production org): $31.40 so far today, 3.5× the usual $9.10 (unusual above $24.60, from 14 days)" }] }
  }],
  "plan": { "spendMonitors": 5, "spendProviders": 3, "spendAnomaly": true }
}

series is aligned with days; null means unknown, 0 means nothing was spent. baseline.status is learning, normal, anomaly or severe. locked marks a provider account beyond what the plan reads (kept, with its key, and simply not polled). days can be 7 to 90.

Field-level detail for every operation is in the REST API reference.

#In upbutler.yaml

A spend monitor can be declared in upbutler.yaml like any other monitor: an entry with a spend object (the kind is inferred from it), with the same fields as the API.

monitors:
  - id: ai-spend
    name: AI spend
    spend:
      sources: [runs]            # omit for automatic
      dailyCapUsd: 50
      monthlyBudgetUsd: 800
      anomaly:
        enabled: true
        sensitivity: 3.5
        minUsd: 5
      autoStopRuns: false

Source keys are the ones the API uses. runs is portable: it means Agent Runs in whatever workspace the file is applied to. balance:<monitorId> and provider:<sourceId> are ids of one workspace (read them from GET /spend), so a file that lists them only applies to that workspace; leave sources out to let the monitor choose automatically. upbutler export writes spend monitors with the keys they have. Provider accounts themselves are not declared in the file: their keys are secrets and are connected in the dashboard or with POST /spend/sources.

#Plans

A spend monitor also counts as one of the plan's monitors. When a plan ends or a trial runs out, nothing is deleted: provider accounts beyond the plan stay connected but are no longer read (locked), and spend monitors beyond it are paused. See Plans & billing.