Sentry alerts

Sentry classifies every captured event so alert rules can route by surface (auction.*, payment.*, identity.*, moderation.*, infra.*). The classifier itself ships from packages/observability; per-spec classifiers are owned by their sub-system spec.

Spec 03 §11 #9 — moderation PR-E ships the concrete error-rate alert rules below. The exact thresholds are tuned for the moderation queue’s expected traffic (~50 ads/day MVP); revisit at Wave 2 when the post-launch baseline is established.

Severities

Severity	Means	Routing
critical	User-visible outage or data integrity at risk	Pages the rotation immediately; declare an incident
high	Degraded but not outage; on-call investigates this shift	Sentry digest within 15 min
medium	Anomalous; review within 24h	Daily digest
low	Noise budget; ignore unless trending	Weekly digest

Classifiers

Classifier	Owner spec	Examples	Default severity
auction.bid-rejected	04	Validation order failure, rank desync	medium
auction.close-missed	04	00:00 UTC close did not run	critical
auction.do-stuck	04	AuctionDO alarm fired but RPC did not advance	critical
payment.saga-failed	05	Saga ended in failed state	high
payment.cascade-fired	05	Cascade to runner-up triggered	medium (per-event), critical (rate spike)
payment.webhook-replay-stuck	05	Replay queue depth > threshold	high
identity.kyc-stalled	02	Stripe Identity verification stuck > 24h	medium
identity.setupintent-failed	02	First-bid SetupIntent failure	high
moderation.queue-overflow	03	Pending moderation queue > threshold	high
moderation.classifier-disagrees	03	AI pre-classifier and human reviewer disagree above noise floor	medium
infra.deploy-failed	01	wrangler deploy failed after 3 retries	high
infra.terraform-drift	01	terraform plan non-empty on schedule	medium
ws.fanout-lag	06	WS broadcast lag > 2s	medium

Spec 03 moderation alert rules (PR-E)

The moderation pipeline emits structured logs with stable error_ids. PR-E pages on-call when any single error_id exceeds 10 events / 15 min (spec §11 #9). The rule shape is identical for every bucket; the table below lists each bucket and its severity so an operator triaging the page knows whether to escalate.

Upload pipeline (apps/api — owned by PR-A #88)

error_id	Severity	What it means	Likely cause
ads.upload.dedupe_lookup_failed	high	D1 outage on the per-user sha256 lookup	D1 transient 5xx, deploy-time index drift
ads.upload.dedupe_cross_user	critical	Defence-in-depth assert fired — findCreativeAssetBySha256 returned a foreign asset	SQL regression dropping the user_id predicate; data-integrity bug
ads.upload.r2_put	high	R2 PUT at upload time failed	R2 transient 5xx, quota, signature mismatch
ads.upload.d1_after_r2_failed	medium	D1 insert failed after R2 PUT, R2 object successfully deleted (compensated)	Concurrent UNIQUE collision on the dedupe race
ads.upload.r2_orphan_uncompensated	critical	D1 insert AND the compensating R2 delete both failed — orphan R2 key recorded	R2 sustained outage; needs sweeper backfill
ads.upload.moderation_post_commit	critical	q-moderation send failed after D1 commit — the ad is pending with no classifier signal	Queues outage; classifier orphan
ads.upload.transforms_post_commit	high	q-transforms send failed after moderation send succeeded — renditions delayed	Queues outage; sweeper-recoverable
ads.upload.deduped_moderation_post_commit	critical	Same as moderation_post_commit but on the dedupe path	Queues outage on dedupe twin
ads.upload.malformed_image	low	Client uploaded a structurally-corrupt PNG/JPEG/WebP (400 to caller)	User-side; only paged on a rate spike (possible scan attempt)
ads.upload.bad_user_id	critical	asUserId re-brand failed on session.user_id — data-corruption signal	User-table corruption; needs immediate triage

Transforms consumer (apps/queue — owned by PR-B #106)

error_id	Severity	What it means	Likely cause
transforms.r2_get	high	R2 GET on the EXIF-stripped original threw	R2 transient outage
transforms.r2_object_missing	medium	R2 GET returned null — object was deleted between upload and consumer	GDPR purge race; uploader DELETE before classifier runs
transforms.run	medium	Photon decode / WASM error	Malformed image bytes slipped past the upload sniff (rare)
transforms.r2_put	high	One of the three rendition PUTs failed; orphan_keys payload names the leaked R2 keys	R2 transient outage
transforms.d1_update	high	D1 UPDATE failed after R2 PUTs succeeded; full orphan_keys triple in payload	D1 transient outage
transforms.bad_user_id	critical	asUserId re-brand on creative_assets.user_id failed	Asset-table corruption
transforms.unexpected	critical	Outer-catch escape — real handler bug	Code defect; treat as P1

Admin queue API + UI (apps/admin — owned by PR-C #108 + PR-D #110)

error_id	Severity	What it means	Likely cause
mod.already-claimed	low	Two admins clicked claim on the same ad within 10 min	Expected UX collision; only paged on a sustained spike
mod.state-conflict	medium	§10 E9 race — SLA cron or other admin flipped the row first	Admin lost a decision race; check the audit log for the other decision
mod.bad-json	low	Admin client posted malformed JSON to /api/moderation/:ad_id/decide	Admin UI bug; only paged on a rate spike
mod.bad-payload	low	Schema fail (verdict, reason_code, notes type, verdict↔reason_code mismatch)	Admin UI bug; only paged on a rate spike

Rule scaffolding

All critical classifiers above page the on-call rotation immediately (no debounce). All high classifiers digest every 15 minutes to #oncall-moderation. All medium and low classifiers digest to the daily #sentry-digest.

When the official Sentry provider for our setup is available, infra/terraform/sentry.tf should declare each row in the tables above as a sentry_metric_alert resource keyed off the error_id tag with aggregate = "count()", threshold = 10, and window = 15 (minutes). Until then the rules are manually configured in the Sentry UI; the rule names should match the error_id 1:1 so an audit (or a future Terraform import) is mechanical. The on-call runbook below documents the manual-setup steps.

Manual Sentry rule setup (Wave 1, until Terraform lands)

1. Sentry → Alerts → Create Alert Rule → Metric Alert.
2. Conditions: event.tags["error_id"] equals "<bucket>".
3. Trigger: count() above 10 over 15m.
4. Action: PagerDuty integration nyl-oncall-moderation (critical) or #sentry-digest Slack webhook (medium / low / high).
5. Save with name = the error_id verbatim.

Repeat for every row in the spec-03 tables above. The infra/runbook/sentry-rules.csv (TBD — placeholder) tracks which rules are set and by whom; review on the first Monday of each month.

See also

• Incident response — when an alert turns into an incident
• On-call rotation — who receives what
• Spec 03 — Ad Moderation for the classifier scaffold + the error_id definitions
• Spec 13 §5 — observability conventions