Incident response

An incident is any of:

Production user-facing surface is degraded for > 5 minutes (nowyourlink.com, api.nowyourlink.com, admin.nowyourlink.com).
Data integrity is suspected — wrong winner persisted, incorrect invoice issued, payment double-charged, KYC data leak.
Auction window missed its close at 00:00 UTC.
Sentry classifier critical fires (see Sentry alerts).

If you are not sure, treat it as an incident. Downgrading is cheaper than discovering later.

1. Declare

Post in #incidents (Slack): “Declaring incident — — — IC: @”.

You are now Incident Commander. Hand off only when someone else explicitly takes IC.

2. Page the right on-call

Symptom area	Who	How
Auction state (DO, ranking, anti-snipe)	Auction on-call	`@auction-oncall` in `#incidents`
Payment saga, Stripe, invoicing	Payment on-call	`@payment-oncall`
Identity, KYC, OAuth, Stripe Identity	Identity on-call	`@identity-oncall`
Infra (DNS, Workers routing, Terraform)	Infra on-call	`@infra-oncall`
Anything else	Generalist on-call	`@oncall`

Two on-calls is normal for cross-cutting incidents (e.g. saga + auction).

3. Capture

Before mitigating, snapshot what you see — Sentry tabs, Cloudflare dashboard panels, the failing request’s request-id (echoed in every response per the request-id middleware in packages/middleware), the timestamps. The IC keeps a running timeline in the incident thread; don’t trust memory.

T+0   declared
T+1   IC: @arne, paging @payment-oncall
T+3   payment-oncall acked
T+4   saga workflow stuck on PaymentIntent.create — Stripe 503
T+6   confirmed via Stripe dashboard — global Stripe degradation
T+8   mitigation: pause auction close cron (apps/cron disabled)
T+12  Stripe recovered; resuming

4. Mitigate

Prefer mitigation that stops the bleed without destroying evidence:

Pause apps/cron to stop auctions from closing into a broken saga: deploy apps/cron with the paused flag in its config (wrangler.toml) and re-deploy.
Drop the saga workflow for an in-flight bid: wrangler workflows terminate <instance-id> — only after confirming with payment-oncall, since this strands the bid.
Rollback the offending Worker — Rollback.
Cloudflare WAF rule to block a malicious source — dashboard → Security → WAF.

Do not:

DELETE or UPDATE rows in D1 unless an owner has confirmed.
wrangler tail against production for an extended period — it streams logs that may contain PII.
Push a hotfix directly to main — branch protection blocks it; use a PR with the incident label so reviewers prioritize.

5. Communicate

Update status in #incidents every 15 minutes minimum (set a timer).
If user-visible: post #status-public with the same content the public would see — keep it factual, no internal jargon.
Resolve in #incidents: “Incident resolved — — postmortem assigned to @“.

6. Postmortem

A blameless postmortem is required for every declared incident, even short ones. Use the template in docs/superpowers/runbooks/postmortem-template.md (TBD — placeholder). The IC owns getting it filed within 5 business days.