# Incidents in PagerDuty

> Connect PagerDuty and the kill switch, over-settlements, held high-value payments and spent budgets open one incident each, resolved when they end.

Source: https://immiscible.fly.dev/docs/guides/pagerduty

## What opens an incident

| Trigger | Opens | Resolves | Severity |
|---|---|---|---|
| Kill switch | an agent frozen (`agent_freeze`) or a fleet (`agent_freeze_batch`) | its unfreeze, or the batch lifted | warning for a pause, error for a security hold, critical for a fleet |
| Incident | an over-settlement, an over-volume, unmatched Ramp charges (`agent_incident`) | (by a person in PagerDuty) | critical for an over-settlement, otherwise error |
| Held request | a payment held for approval at or above the threshold (default 1,000.00) | its approval, denial, expiry or cancellation | warning |
| Budget | the gateway held or refused calls because a budget is spent | (by a person) | error when refused, warning when held |

Drills never page. A freeze Immiscible makes because of an incident is left to that incident. Each incident has a dedup key built from the ledger (`imm:<workspace>:freeze:<agent>`, `imm:<workspace>:budget:<scope>:<id>:<month>`, and so on), and Immiscible remembers which are open, so a budget refusing a thousand calls is one incident, and a resolve is only sent for one Immiscible opened. **Manage** chooses the triggers and the threshold.

Delivery is off the request path, in order, with up to six attempts and backoff on a 429 or a 5xx. PagerDuty deduplicates on the key, so a retried delivery that did arrive changes nothing.

## Connect

- **Connect PagerDuty** (when the operator has registered an app): sign in to PagerDuty, choose a service, and Immiscible adds an **Events API v2** integration named Immiscible to it. It keeps only that integration's key and lets the sign-in go.
- **Or paste a key**: on a service, Integrations, add **Events API v2**, and paste the 32-character Integration Key. Choose US or EU.

Either way a test incident is opened and resolved before anything is saved.

## For the operator: register the PagerDuty app once (optional)

1. In PagerDuty, **Integrations**, **Developer Tools**, **App Registration**, **New App** (Manager, Global Admin or Account Owner).
2. Add **OAuth 2.0**, choose **Classic User OAuth** with the `write` scope (classic apps work on any account without PagerDuty's review), and the redirect URL `https://<your PUBLIC_URL host>/connect/pagerduty/callback`.
3. Set `PAGERDUTY_CLIENT_ID` and `PAGERDUTY_CLIENT_SECRET`. Without them the card offers the pasted key only.

Sources: [Events API v2](https://docs.pagerduty.com/developer/send-alert-event), [rate limits](https://docs.pagerduty.com/developer/events-api-rate-limits), [OAuth](https://docs.pagerduty.com/developer/oauth-functionality), [register an app](https://docs.pagerduty.com/developer/register-an-app), [REST schema: createServiceIntegration](https://raw.githubusercontent.com/PagerDuty/api-schema/main/reference/REST/openapiv3.json).