06 — real-time-hub (Design)

Status: Draft Date: 2026-05-23 Owner: backend (WS DO specialist) Depends on: 02 identity-onboarding (session token), 04 auction-engine (AuctionDO state) Binding refs: master §4.4, §10.2; research/astro-cf-monorepo-patterns.md §3–§5, §10.

1. Purpose & scope

Purpose. Deliver low-latency, cinematic live signals (bidder count, bid pulses, “Top 3” indicators, reveal cue) to the public site, the bidder dashboard, and the lobby viewer — under the cost envelope set by master §10.2 — without leaking bid amounts or bidder identity.

In scope (this spec owns):

HubDO (singleton “lobby”) + per-auction WebSocket lifecycle attached to AuctionDO.
Hibernatable WS lifecycle (acceptWebSocket, serializeAttachment, webSocketMessage, webSocketClose, webSocketError, auto-response pair).
Subscribe and reconnect resume protocol (last_seq replay/resync).
MVP wire schemas for bid_placed_pulse, top3_position_changed, auction_closed, and spotlight_revealed; secondary events listed in §5 are deferred to #231.
Per-auction broadcast batching (≤1/s) and ring-buffer-based reconnect.
Per-user fan-out for top3_position_changed.

Out of scope (not owned):

Authoritative auction state and bid validation (04 owns).
Bidder identity, session minting (02 owns); 06 only validates the session token in the subscribe message.
Saga events that drive auction_extended / auction_closed (04/05 produce them; 06 only fans out).
The cinematic UI itself (07 / 09 own animation).
Browser-side realtime application (nyl:seq:<auction_id>, last_seq, batch/replay/resync handling, and UI state updates) is tracked in #230, owned by 07/09.

2. Topology

                              ┌──────────────────────────┐
   public site / lobby ──ws──▶│  HubDO ("lobby")         │  default room
                              │  - bidder_count_changed  │  cross-cutting events
                              │  - spotlight_revealed    │  (reveal, archive ping)
                              └──────────────────────────┘

   bidder dashboard ──ws──▶  AuctionDO("2026-06-04")  per-auction room
                              - bid_placed_pulse        bidders + watchers
                              - top3_position_changed   (per-user msg)
                              - auction_closed
                              - auction_extended        (#231)
                              - bidder_count_changed    (#231)
                              - heat_index / velocity_tick (#231)

Two DOs, not one. Co-locating per-auction WS inside AuctionDO eliminates cross-DO RPC on the hot path (for (const ws of this.ctx.getWebSockets()) ws.send(msg) — see forest-show.ts:112–128). HubDO carries only cross-cutting events; its state is tiny and one DO holds the public-landing fan-out within CF’s ~32 k-connection / ~1 k-msg/s soft cap (research §4).

Why not single-hub? It would re-marshal every per-auction message through a routing layer (doubled CPU), blow the 1 k msg/s ceiling at ~20 concurrent auctions × 50 bidders, and prevent any DO from hibernating. Sharded per-auction DOs naturally cool down between bursts.

Why not pure per-auction? The landing page needs aggregate “X auctions live” without subscribing to N DOs — that’s HubDO.

3. Hibernatable WS pattern (mandatory per §10.2)

Both HubDO and AuctionDO use the Hibernation API. Pattern (mirrors forest-show.ts and research §3):

// constructor — wire keep-alive once
this.ctx.setWebSocketAutoResponse(
  new WebSocketRequestResponsePair("ping", "pong"),
);

// fetch — upgrade
const [client, server] = Object.values(new WebSocketPair());
this.ctx.acceptWebSocket(server, [`auction:${auctionId}`, `tier:foreground`]);
server.serializeAttachment({
  hashed_bidder_id,   // SHA-256(bidder_id + daily_salt); never raw
  role,               // "bidder" | "watcher"
  joined_at,          // ms epoch
  last_seq,           // last delivered broadcast seq, updated on each send
}); // attachment cap 16 KB — IDs only

// message handler — runs only on real client frames; "ping" never wakes us
async webSocketMessage(ws: WebSocket, raw: string | ArrayBuffer) { ... }

// close/error — clean local maps; with compat_date ≥ 2026-04-07
// `web_socket_auto_reply_to_close` handles the protocol close frame.
async webSocketClose(ws: WebSocket)  { ... }
async webSocketError(ws: WebSocket)  { ... }

Without hibernation we’d bill wall-clock for hours per idle viewer (research §10). Sockets are accepted with a tier:foreground marker tag. Tags are not mutable after acceptWebSocket, so the live runtime tier is carried on the rewritable attachment instead (#231): a {type:"pause"} frame sets tier:"background" there; non-critical broadcasts skip background sockets and {type:"resume", last_seq} restores foreground delivery + replays the ring.

Client → DO (first frame after upgrade):

type ClientSubscribe = {
  type: "subscribe";
  auction_id: string;             // "YYYY-MM-DD" or "lobby"
  session_token?: string;         // KV-validated; required for bidder role
  last_seq?: number;              // for resume; omit on fresh connect
};

DO → client (response):

type ServerSnapshot = {
  type: "snapshot";
  auction_id: string;
  seq: number;                    // next broadcast will be seq+1
  state: {
    bidder_count: number;
    closes_at: string | null;
    // #231: present only while OPEN (omitted for IDLE/CLOSED, like a non-live
    // closes_at). Recomputed from the rolling velocity window at snapshot time,
    // so it decays. `velocity` == velocity_tick.bids_per_minute.
    heat_index?: number;
    velocity?: number;
    // #315: OPEN-only — the `closing_window` tier inputs the client folds into
    // its UI on subscribe, then keeps current via `anti_snipe_config_changed`.
    // Tier inputs ONLY (never the strategic extension_ms/max_total_extension_ms).
    anti_snipe?: { enabled: boolean; window_ms: number };
  };
};

// then, if last_seq present and gap is recoverable:
type ServerReplay = {
  type: "replay";
  events: BroadcastEnvelope[];    // ordered, exclusive of last_seq
};
// or, if gap exceeds ring buffer:
type ServerResync = { type: "resync"; reason: "buffer_overflow" };
// client must then treat the snapshot as authoritative and discard local deltas.

session_token is opaque; the DO validates it against KV (NYL_SESSION). On invalid token the socket downgrades to role: "watcher" — connection stays open, but top3_position_changed will never be routed to it. No echo of the token back.

5. Message schemas (exact TS)

Single envelope wrapping all server-pushed deltas; seq is a monotonic per-room counter:

type BroadcastEnvelope =
  | { type: "bidder_count_changed"; seq: number; auction_id: string; count: number }
  | { type: "bid_placed_pulse";     seq: number; auction_id: string; count: number; recent_hashes: string[] }
  | { type: "top3_position_changed";seq: number; auction_id: string; position: 1 | 2 | 3 | null }
  | { type: "auction_extended";     seq: number; auction_id: string; new_closes_at: string }
  | { type: "auction_closed";       seq: number; auction_id: string }
  | { type: "spotlight_revealed";   seq: number; date: string; winning_ad_id: string }
  | { type: "heat_index";           seq: number; auction_id: string; value: number }
  | { type: "velocity_tick";        seq: number; auction_id: string; bids_per_minute: number }
  | { type: "anti_snipe_config_changed"; seq: number; auction_id: string; enabled: boolean; window_ms: number };

MVP support matrix.

Implemented for #6 close: bid_placed_pulse, top3_position_changed, auction_closed, spotlight_revealed.
Implemented in #231: bidder_count_changed, heat_index, velocity_tick, auction_extended — all per-room, state-like, coalesced to the latest value on the 1-Hz flush (auction_extended carries new_closes_at, inferred from a successful in-window anti-snipe bid that pushed closesAt later) — plus the per-socket auth_downgraded notice and server-side pause/resume tier throttling (attachment-based; non-critical broadcasts skip background sockets, terminal auction_closed reaches all, resume replays the ring).
Implemented in #315: anti_snipe_config_changed — per-room, state-like, coalesced on the 1-Hz flush; inferred from a successful runtime setAntiSnipeConfig (spec 04 §6/§9). Carries ONLY the two closing_window tier inputs { enabled, window_ms } so the client re-computes the closing-window UI tier without a reconnect; the strategic extension_ms / max_total_extension_ms never cross the wire (D7/D8, admin-only via #317). The anti_snipe tier inputs also ride the §4 snapshot.

Privacy invariants:

No bid amount ever crosses the wire (cinematic-only signal).
No raw user_id; only hashed_bidder_id = SHA-256(user_id || daily_salt) (same salt scheme as pixel ingest, research §9).
top3_position_changed is per-user: routed by matching ws.deserializeAttachment().hashed_bidder_id to the event’s target hash; iterate with this.ctx.getWebSockets() and skip non-matching sockets. position: null means “fell out of top 3”.

6. Broadcast batching (≤1/s per auction)

The AuctionDO runs a 1-Hz scheduler via this.ctx.storage.setAlarm(now+1000) whenever the pending-event queue is non-empty. On alarm():

Drain pending events.
Coalesce by type — keep the latest for state-like signals as those producers land in #231. top3_position_changed is per-user and immediate, not queued.
Pulse-like signals (bid_placed_pulse) are coalesced to count + last-3 hashes — UI shows “5 bids in last second” with three sparkle pulses.
Assign seqs, append envelopes to the ring buffer (§7), iterate WSes once, send batch as { type: "batch"; envelopes: BroadcastEnvelope[] }.
Reset alarm if queue is empty; else schedule another tick.

Why 1s, not immediate? Every send bills CPU (research §10). 30 bids/min × 50 sockets = 1500 un-batched sends/min vs ~60 batched. 1s also matches human “burst” perception — coalesced pulses look livelier than one-by-one. auction_closed bypasses the 1-Hz wait but still uses the same {type:"batch", envelopes} frame; if a pulse queue is pending, the final batch contains the coalesced pulse followed by auction_closed. spotlight_revealed is bypass-batch in HubDO for the reveal ceremony.

7. Reconnect / resume

Client persists last seen seq per auction_id in localStorage (nyl:seq:<auction_id>).
On reconnect, client sends { type: "subscribe", auction_id, last_seq }.
AuctionDO keeps a ring buffer of the last 200 per-room envelopes in this.ctx.storage as persisted arrays (ws_ring, ws_ring_floor). The original SQLite-table sketch (events(seq INTEGER PRIMARY KEY, payload BLOB)) is an accepted implementation deviation: the current access pattern is append, cap, and replay-by-seq, so a storage array is simpler and covered by unit tests. Migrate to ctx.storage.sql only if #231 adds query-heavy event types.
If last_seq is within buffer range → replay. Otherwise → resync + fresh snapshot.
Buffer survives DO hibernation (DO storage is persisted), so reconnects after long pauses work.
200 entries is chosen so that a tab backgrounded for ~3 min during peak bid velocity (~60 batched envelopes/min) still resumes without resync.

8. Cost-control measures (master §10.2 checklist)

Hibernation API — acceptWebSocket, no server.accept().
Auto-response keep-alive — setWebSocketAutoResponse(new WebSocketRequestResponsePair("ping","pong")) so app-level pings never wake the DO.
Broadcast batching — 1Hz tick, coalesced by type.
Sampling — heat_index and velocity_tick fire per accepted bid but coalesce to one envelope each per 1-Hz flush (latest value wins, §6), so the send rate is already bounded by the batch tick; no separate sampler is needed (#231).
Attachment-based throttling — a {type:"pause"} frame marks the socket tier:"background" on its (rewritable) attachment; non-critical broadcasts skip it, {type:"resume", last_seq} restores delivery + replays the ring (#231). The original tag-based sketch was infeasible (tags immutable post-accept). Terminal auction_closed is still sent to all connected sockets.
Per-user routing via attachments — top3_position_changed scans getWebSockets() and matches deserializeAttachment().hashed_bidder_id. This supersedes the earlier getWebSockets("bidder:<hash>") sketch because Workers WebSocket tags are fixed at acceptWebSocket, while bidder identity is learned only after subscribe auth.

9. Per-user vs per-room

Per-room (the default): for (const ws of this.ctx.getWebSockets()) ws.send(...).
Per-user (top3_position_changed): on subscribe, after validating session_token against KV, the DO calls ws.serializeAttachment({ hashed_bidder_id, role }). Per-user broadcast iterates this.ctx.getWebSockets() and filters each socket’s attachment. A bidder may have multiple tabs; all matching tabs receive the message.
Session-token-less (“watcher”) sockets never get a hashed_bidder_id attachment and therefore never receive per-user events.

10. Edge cases

Auction closes with subs connected. The close path emits a final batch including auction_closed, appends it to the replay ring, then iterates WSes and calls ws.close(1000, "auction_closed"). Client treats it as terminal.
DO hibernates while sockets idle. Correct path: real frames re-instantiate the DO; attachments rehydrate via ws.deserializeAttachment() (research §3). Pending setAlarm survives.
Client tab backgrounded. Server-side {type:"pause"} / {type:"resume", last_seq} throttling is implemented (#231): pause marks the socket background (attachment tier), non-critical broadcasts skip it, resume restores delivery + replays the ring. The browser-side visibility handling that emits these frames is tracked in #230.
Thundering herd on reveal ceremony. spotlight_revealed is broadcast by HubDO only (not multiplied across AuctionDOs); we chunk the fan-out into 256-socket batches with await scheduler.wait(0) between to spread CPU across the second. Day-page HTML is already in CF cache (research §5).
Token rotated mid-session. Subscribe-time token failures downgrade to watcher. A re-subscribe whose token no longer validates (was a bidder, now a watcher) also emits a per-socket auth_downgraded frame (#231) so the client drops its per-user UI; the frame carries no identity and is not ringed.

11. Acceptance criteria

wrangler tail apps/api for per-auction sockets and wrangler tail apps/ws for lobby sockets show zero CPU log lines for 60s on an idle WS while auto-response ping/pong flows.
Two tabs on same auction_id: a bid in tab A produces bid_placed_pulse in tab B within 1.2s (1s batch + RTT).
Disconnect 90s with 5 bid events; reconnect with last_seq → exact replay (seq continuity verified).
Disconnect 30 min (≥200 events); reconnect → resync + fresh snapshot, no stale events.
top3_position_changed for bidder X arrives at tab X but not at a different-hashed_bidder_id tab Y.
1000-socket simulated reveal: spotlight_revealed arrives within 2s of activation (p95).
Broadcast rate ≤1 send/socket/second/auction across a 10-min sustained-bid load test.
Deliberate throw inside webSocketMessage is captured by Sentry via instrumentDurableObjectWithSentry.

12. Agent-team hints

Actual worker layout: AuctionDO lives in apps/api/src/auction-do.ts and is reached directly by apps/api HTTP routes plus forwarded per-auction upgrades from apps/ws/src/index.ts through the AUCTIONS cross-script binding. HubDO/WebSocketHub lives in apps/ws/src/websocket-hub.ts for lobby and reveal traffic. The earlier packages/shared/src/do/* target path was not used.
Mirror forest-show.ts: copy setWebSocketAutoResponse, getWebSockets() broadcast, webSocketClose count-update patterns. Add serializeAttachment/deserializeAttachment for hashed_bidder_id.
Cache-API snapshot: mirror forest-show.ts:149 (sendForestState) — cache initial snapshot 5s via Cache API to absorb reconnect storms.
DO migration entry: AuctionDO’s new_sqlite_classes entry lives in apps/api/wrangler.jsonc; WebSocketHub’s migration lives in apps/ws/wrangler.jsonc. The apps/ws AUCTIONS binding references AuctionDO by script_name and does not own its migration.
Per-user fan-out test: subscribe multiple sockets with sessions for the same user and assert matching attachments receive top3_position_changed while watchers and other hashes do not.
Don’t declare DO bindings on apps/web (Astro) — call apps/ws via service binding only (research §10).
Sentry for DOs: wire instrumentDurableObjectWithSentry — forest-show.ts:1–5 skipped this in world-forest; we do it here.

06 — real-time-hub (Design)

06 — real-time-hub (Design)

1. Purpose & scope

2. Topology

3. Hibernatable WS pattern (mandatory per §10.2)

4. Subscribe protocol

5. Message schemas (exact TS)

6. Broadcast batching (≤1/s per auction)

7. Reconnect / resume

8. Cost-control measures (master §10.2 checklist)

9. Per-user vs per-room

10. Edge cases

11. Acceptance criteria

12. Agent-team hints