ADR 0130 — Refuse to deliver a sealed payload to a flow that cannot open it

• Status: Accepted (2026-10-12) — user approved the kernel guard 2026-10-12; implemented.
• Date: 2026-10-12
• Origin: review of loom #575 (the r2-erasure flake).
• Builds on: ADR 0110 (the durable inbox, D9 erasure marker), ADR 0085 (bounded untrusted-content walks).
• Placement: kernel (@loom/workflow-runtime, the inbox delivery path). Not @loom/plugin-crypto.
• Version impact: none. No event kind is added or changed, so EVENT_SCHEMA stays 1.24.0; no SPI signature or semantics change (EVENT_STORE_SPI, SNAPSHOT_STORE_SPI), KERNEL_ABI and the wire vectors are unchanged. The only public additions are the error class SealedPayloadUnreadableError and its classification. (The 2026-10-14 residuals below add one coordinated bump, EVENT_SCHEMA 1.25.0 and EVENT_STORE_SPI 2.5.0.)

Context

@loom/plugin-crypto seals fields of an inbox message (INBOX_MESSAGE.payload, and the copy in MESSAGE_CONSUMED.payload) inside EncryptingEventStore, at the store boundary: { __loomEnc: "loom/aes-256-gcm", ... } replaces the plaintext on append and is opened again on read. A node whose store is NOT configured to open that path (a different sensitive map, or no crypto plugin at all) reads the same stream through its raw store. Before this ADR it then:

1. handed the flow the envelope as if it were the user’s payload, and
2. journaled the envelope as the MESSAGE_CONSUMED payload,

so the corruption was durable and replayed deterministically. Nothing refused it. (__loomEnc appeared in the kernel only in blankStream.ts, a diagnostic.)

Decision

D1 — Placement: the kernel, not a 1P crypto check

The failing case is a node that does not have the crypto plugin, or has it configured differently. A check shipped in the plugin cannot run on a node that does not load the plugin, so it cannot protect exactly the node that needs protecting. The inbox delivery path is kernel code that every node runs. The guard is fail-closed, adds no dependency (it recognises the envelope by SHAPE, an own __loomEnc key, and knows nothing about keys or ciphers), and the kernel already walks untrusted payloads elsewhere. Kernel placement is the one that holds when the plugin is absent.

D2 — The invariant

A message whose payload contains a sealed envelope after the store has had its chance to open it is never delivered to a flow and never journaled as MESSAGE_CONSUMED.

At delivery time every envelope the node’s store recognised has already been opened on read, so an envelope that survives is by construction “sealed and not openable here”.

What “contains” means. The sealed paths are configuration (pathsFor / mapPaths in EncryptingEventStore accept nested paths such as payload.secret, and the default inbox registration is the whole payload). The kernel cannot know a given deployment’s paths, so the guard looks for a plain object with an OWN __loomEnc key whose value is a STRING (the algorithm id) at the top level or at any depth, in objects and arrays. Inherited keys do not count, and neither does a non-string value ({ __loomEnc: 1 } is an ordinary payload): the predicate stays algorithm-independent (it never reads WHICH algorithm) and has fewer false positives. A caller that must relay FOREIGN ciphertext (a __loomEnc string key it did not produce) wraps it as a string, for example JSON.stringify(envelope).

Bounded walk. Iterative and breadth-first (a shallow envelope is found before any budget matters, and depth cannot blow the stack), capped at SEALED_WALK_BUDGET = 100 000 visited nodes, the same budget as blankStream’s carriesKey. Exhausting the budget (a huge, very deep, or cyclic in-process payload) is NOT “clean”: the guard fails closed and treats it as unverifiable (SealedPayloadUnreadableError.unverifiable).

D3 — One shared check, every delivery site

assertDeliverable(executionId, message) in inbox.ts runs BEFORE any MESSAGE_CONSUMED is built, at both sites that consume an inbox message into a flow:

• WorkflowContext.satisfyFromInbox (the live run; closure 1 of the lost-wakeup protocol), and
• WorkflowRuntime.notifyAttempt’s scan (the notify path: the post-park re-read afterInboxPark, TimerService, recoverAbandoned, the send wake).

Every MESSAGE_CONSUMED append was audited. The others write no user payload: the operator discard (discardInbox, D8) and the TTL expiry (D10a) journal payload: null, and a duplicate delivers nothing (its payload is not copied), so they need no check. An operator discard is deliberately still allowed on a sealed message: it is the escape hatch for a message nobody can open.

D4 — Park, do not fail; refuse the attempt, not the execution

Every site leaves the message (or the execution) as it was and appends nothing. What the flow sees differs by site, and in none of them does the execution fail:

• Live run: the flow stays parked on its receive. Failing the execution would make the refusal permanent (EXECUTION_FAILED is terminal), destroying exactly the “stay available for a correctly configured node” property. The run parks normally; the post-park notify pass raises the typed error.
• Notify path: SealedPayloadUnreadableError is thrown to the caller, which already routes errors to onStorageError / TimerService.onError / recoverAbandoned’s onError. The park and the inbox are untouched. The notify pass also writes the durable INBOX_WAKE_AT row (whichever caller got there: the post-park read, a send’s wake, a timer poll), so a node configured to open the payload picks the message up on its next poll without waiting for a new send.
• Replay: ctx.select / ctx.receive serves a RECORDED inbox answer from the folded _waitAnswers. If that recorded payload is still a sealed envelope on this node (a capable node journaled it sealed in MESSAGE_CONSUMED; this node reads the raw log), the flow is NOT handed the ciphertext. The attempt is refused: the context latches the refusal and seals its journal, so even a flow that swallows the rejection cannot append, complete or park on it, and the context is also frozen like a park: no NEW tool call, effect, decision or child start is dispatched after the refusal (a swallowing flow’s later ctx.run simply never starts); the runtime rejects the handle with the typed error (recorded: true) and writes nothing, in particular no EXECUTION_FAILED. The execution stays where it was (a resume of an unrelated gate on that node already appended its own EXECUTION_RESUMED, so it reads running) for a capable node, which sees plaintext after decryption, passes the check and completes it via recoverAbandoned or the next resume. Old logs are unaffected: a plaintext answer is clean, so replay on any node is identical to before.

Head-of-line blocking is per EXECUTION, not per topic. The live return delivered and the notify path’s throw both abort the whole scan for that execution, not just the sealed message’s topic. A sealed head message therefore holds every topic of that execution for this node until a capable node consumes it or an operator discards it. This is deliberate: skipping the sealed topic and delivering another would let two nodes with different crypto configuration pick different select branches (and so different journaled histories) for the same execution. Delivery is FIFO per topic and a cursor cannot skip, so skipping would also silently reorder or lose the sealed message.

Back-off on an incapable node. TimerService.fireInbox catches SealedPayloadUnreadableError and applies an exponential back-off (the re-drive schedule, no attempt cap) that NEVER retires the row, because capable nodes still need it. onError is reported on the first refusal and then every 16th, not on every poll. The per-execution back-off map is bounded (1024 entries, oldest evicted); beyond that cap it degrades to periodic polling rather than growing. recoverAbandoned treats a refused re-drive like a diverged one: it counts it as failed (never recovered, no onRecovered), reports it through onError on the first refusal and then every 16th, and skips that execution on this node for an exponential back-off window (same bounded map, same degradation).

D5 — The error

SealedPayloadUnreadableError(executionId, messageId, topic, unverifiable = false). Ids only; the message text and fields never carry the ciphertext. It is classified TERMINAL in packages/agent/src/toolFailure.ts (and its fitness test): retrying on the same node cannot help, and a model must not be told to “try again” by an isError result.

D6 — The ingest rule (send)

send refuses a payload that contains an __loomEnc envelope, with InboxMessageRefusedError, before anything is appended, like the forged $shredded marker. The walk runs over JSON.parse(json), the same JSON the store persists and the $shredded check reads (a toJSON can mint an envelope the live object does not show), and AFTER the byte-size check so the parse is bounded. Why this layer and why it is unconditional: sealing happens inside the event-store wrapper at append, AFTER send has run, so at send time a raw user payload carrying __loomEnc is always forged (or relayed ciphertext read through a raw store). A genuine seal can only be produced by the configured store, below send. Without the rule a sender could plant an envelope that is indistinguishable from a real seal at delivery, and the guard would be a denial-of-service lever (park any execution’s topic) instead of a safety net. A payload too large or deep to check inside the budget is refused as well (fail closed); the default inboxMaxPayloadBytes (64 KiB) is far below the budget, so this only bites a raised limit.

Residual: a writer that bypasses send and appends to the reserved inbox stream directly is below the kernel’s authority (the stream is kernel-owned; only send writes it).

Consequences

What is guaranteed, and what is not:

• Guaranteed. A node that cannot open a sealed inbox payload never delivers the ciphertext to a flow and never journals it, on any of three paths: a live consume, a notify-path consume, and a replay of an answer a capable node already journaled. In each case it appends nothing, never fails the execution, and leaves it consumable by a capable node (a durable INBOX_WAKE_AT row exists for the notify-path and live refusals, written by notifyPass/markContended; a replay refusal in run() ALSO leaves the row since the residuals D10 landed (slice S6): run() leaves a REFUSAL row (TimerRecord.refused: { head }, an ordinary wakeAt) for it and for a stream-level refusal, the row is a timer-store row, never a log append, and the TimerService poll re-drives the running execution on a capable node only on evidence of abandonment (the host’s isAbandoned, or an EXISTING expired lease; never supersede a live holder), through the routine recoverAbandoned shares; without a timer store the row is a no-op and recoverAbandoned stays the only recovery). A forged envelope cannot be planted through send.
• Mixed fleet. “A mixed fleet can no longer corrupt a log” holds for the INBOX_MESSAGE and MESSAGE_CONSUMED payloads this ADR guards. It is NOT a claim about every sealed field.
• Cost: one bounded walk per delivered (non-duplicate) message, per replayed inbox answer, and per send.

Residuals, named:

1. Other sealed event kinds on an incapable node. Closed by the capability marker (D7, implemented in slice S2): the crypto store stamps metadata.seal on every envelope it seals, and the kernel refuses, denial-shaped, any stream carrying a marker the node cannot open (or whose key it lacks), before fold, replay or authorization. What remains: a stream sealed by a writer that does not stamp (a pre-1.25.0 node, a third-party sealer) is unmarked and keeps the old exposure for its non-inbox kinds.
2. The snapshot store. A snapshot holds the folded state, including _waitAnswers. A capable node checkpoints PLAINTEXT there, so a snapshot store shared with an incapable node hands that node the plaintext (it passes the replay check, correctly, but the plaintext is at rest unencrypted: already documented in docs/compliance.md). Conversely a snapshot that holds a SEALED answer is caught by the replay check like any other. Give snapshots the same encryption as the log, or do not share a snapshot store across differently configured nodes.
3. A pre-existing plaintext payload carrying a string __loomEnc key. send now refuses it, but one journaled before this ADR would be refused on replay forever. No such payload is expected (the key is the crypto plugin’s envelope marker).
4. Head-of-line blocking is per execution (D4), until a capable node consumes the message or an operator discards it. Narrowed by slice S5 (D9, implemented 2026-10-14): an operator may quarantineInbox the sealed head, after which only that TOPIC is held (the execution’s other topics deliver again), and a capable node lifts the hold by itself (kernel.inbox.autoRelease, default on). The hold is a topic-level head-of-line for every node that cannot open the message, ending at an auto-release, releaseInbox or the "discard" disposition.
5. A writer that bypasses send (D6).

Residuals (Accepted 2026-10-14 (user))

• Status: Accepted 2026-10-14 (user). The user chose the stronger option for each residual (marked Chosen below); the other options stay as considered-and-rejected with the reason “user chose the stronger option”. The analysis is kept as written when the residuals were Proposed. Nothing here is implemented yet: the design is under “Accepted design” and the slices are in .superpowers/plans/2026-10-14-adr-0130-residuals.md (local, not committed).
• Version impact (ONE coordinated bump, reserved first): EVENT_SCHEMA 1.24.0 -> 1.25.0, EVENT_STORE_SPI 2.4.0 -> 2.5.0. SNAPSHOT_STORE_SPI (1.1.0) and WIRE_ABI (1.0.0) are unchanged (the wire vectors v15-v18 are added); KERNEL_ABI: unchanged (no syscall; to be confirmed by the reservation slice). The EVENT_SCHEMA 1.25.0 shapes, all additive: envelope metadata.seal (D7, fold ignores it); new kinds INBOX_QUARANTINED and INBOX_RELEASED (D9); folded variable _quarantined; MESSAGE_CONSUMED additionally drops a released quarantine entry (gated, no effect on logs without _quarantined); EXECUTION_STARTED.continuedFrom.carried.quarantined (a validated seed).
• Scope: the four residuals logged after #576 merged (this ADR’s own residuals 1, 2 and 4, plus a gap in the replay refusal’s wake). Residuals 3 and 5 above stand as written.
• Placement bias: outward (ADR 0083, the triage rule). Each residual gets a verdict.
• Version batching: loom batches kernel bumps. Only the bumps an option NEEDS are listed; “none” means no EVENT_SCHEMA, EVENT_STORE_SPI, SNAPSHOT_STORE_SPI, KERNEL_ABI or wire-vector change.

R-a — Other sealed event kinds on an incapable node

Confirmed. assertDeliverable is called at exactly two sites, packages/workflow-runtime/src/WorkflowContext.ts:2114 (live consume) and packages/workflow-runtime/src/WorkflowRuntime.ts:4933 (notify scan), plus the replay check at WorkflowContext.ts:1936 (recorded INBOX answers only, arm.kind === "inbox"). Nothing else looks for an envelope. EncryptingEventStore seals any event type named in its sensitive map (plugins/crypto/src/EncryptingEventStore.ts:86 pathsFor, :162 append, :385-415 read and subscribe), and the kernel itself registers kernel-authored kinds: defineConfig seals EXECUTION_STARTED.principal.identity.* and OUTCOME_RECORDED.recordedBy (packages/app/src/principalPolicy.ts:99-101). The other places a recorded value reaches a flow are therefore unguarded: STEP_COMPLETED.output served on replay (WorkflowContext.ts:859/887/1077 write it, replay serves it), the EXECUTION_STARTED.input handed to the flow, effect and decision results, human-step answers.

Minimal repro (describe only). Node A: EncryptingEventStore with sensitive: { STEP_COMPLETED: ["output"] }; run a flow const s = await ctx.run(tool); await ctx.receive("go"); return s to the park. Node B: same stream through a RAW store, resume/recoverAbandoned after send("go"). B replays the recorded step, s is { __loomEnc: "loom/aes-256-gcm", ... }, the flow completes, and EXECUTION_COMPLETED.output journals the envelope (or a downstream step is invoked with ciphertext as input). Variant with the principal: B reads principal.identity.tenant as an envelope object; whether authz fails closed depends on the policy comparing a string to an object (not verified here, it needs a test before anyone relies on it).

Severity. Correctness plus integrity of the log, not confidentiality (ciphertext is shown, never plaintext). It is the same durable-corruption class D2 closed for the inbox, so the “mixed fleet can no longer corrupt a log” claim is bounded to two event kinds today. Kernel-authored sealed kinds (principal, outcome) are the part that cannot be handled by “run such kinds only on capable nodes”, because defineConfig opted the app in without the operator choosing the kind.

Options.

1. Document only. Rejected: user chose the stronger option. Keep Consequences residual 1; add the principal and outcome kinds to docs/compliance.md as “mixed fleets must be homogeneous in security.crypto”. Free; leaves a footgun the kernel itself arms.
2. Generalise the replay and fold guard to every recorded value. Rejected: user chose the stronger option (its shape sniffing is superseded by the marker; the D2 shape guard stays as the fallback for unmarked streams). One check on the fold input: when the kernel hydrates an event for a VM, any envelope in a kind that is not the inbox raises SealedPayloadUnreadableError (recorded: true) from the fold or the VM seed, refusing the whole attempt like the replay refusal. Kernel, because only the kernel sees every read. Cost: a bounded walk per event folded, which is O(total payload bytes) on every fold and replay (not per delivered message), so it needs a cheap pre-filter (substring test for __loomEnc on the already-serialised JSON where the store keeps it) or a per-execution “checked up to seq” watermark. False positives: an app payload with a string __loomEnc key (residual 3) would be refused on EVERY kind. No version bump if it only refuses; a new error context field is additive.
3. Capability handshake instead of shape sniffing. Chosen (user, 2026-10-14). The store declares what it seals; the runtime refuses to start an execution (or resume one) whose stream header says “sealed under config X” unless this node has X. Needs a stream-level sealed-config marker (a new field on EXECUTION_STARTED or a metadata record) and a decorator capability bit. Fixes the class properly and also covers (b) partially, but is a durable-format change: EVENT_SCHEMA minor bump (and EVENT_STORE_SPI minor if the marker lives in store metadata, as the ADR 0129 incarnation anchor did). It is also the largest option and has no consumer asking.

Placement. Option 2 is kernel (read-path choke point; same argument as D1). Option 3 is kernel plus 1P (@loom/plugin-crypto writes the marker). Option 1 is docs.

Recommendation (superseded by the user’s decision: Option 3). Option 2, scoped to the kernel-authored sealed kinds first (EXECUTION_STARTED principal and OUTCOME_RECORDED), because those are sealed without the operator choosing them. Keep the arbitrary-kind case as documented (Option 1) until a deployment names a second kind; revisit Option 3 only if the user wants the marker for (b) too. Bumps: none for the scoped Option 2 (it adds a refusal, no event shape); EVENT_SCHEMA 1.25.0 (+ EVENT_STORE_SPI minor) only if Option 3 is chosen.

R-b — An unencrypted shared snapshot store holds plaintext of sealed content

Confirmed, but narrower than first logged. An encrypting snapshot decorator already exists (plugins/crypto/src/EncryptingSnapshotStore.ts, sensitive dot-paths into the folded ExecutionState, same subject key as the events, so a shred erases both). The gap is that it is opt-in and the kernel does not check it on the ordinary path:

• WorkflowRuntime.ts:6519 and :7009 call snapshotStore.save(...) with the decrypted fold (checkpoint on settle, periodic) whatever the snapshot store is.
• The only cross-check is LogLifecycle.ts:155, which refuses COMPACTION when the event store encrypts but the snapshot store does not share its KeyStore. A plain checkpoint is not checked.
• docs/compliance.md:203-206 documents that variables._waitAnswers is keyed by a dynamic waitId, “which a sensitive path cannot name”, so even an EncryptingSnapshotStore configured with static paths does not seal a recorded inbox answer. The fix named there is “seal the whole state”, which EncryptingSnapshotStore does not offer (it maps declared paths only).

Minimal repro (describe only). Node A: EncryptingEventStore with INBOX_MESSAGE/MESSAGE_CONSUMED payload sealed, plain InMemorySnapshotStore. Run a flow that receives a secret and settles. snapshotStore.loadLatest(id) returns state with variables._waitAnswers.<waitId>.payload in clear. Shred the subject: the event log is unreadable, the snapshot still serves the secret (and node B, incapable, replays from the snapshot, passes the D4 check because it is plaintext, and completes).

Severity. Confidentiality, and erasure-theater (K10.2/A1 is the same hole for static paths, closed by EncryptingSnapshotStore). Not a liveness or corruption issue. For the guard itself it is benign (an incapable node served plaintext is correct), so it is a compliance gap rather than a hole in D2.

Options.

1. Document and doctor. Chosen, together with 2 and 3 (user, 2026-10-14: BOTH 1P and kernel). Keep the compliance note; add a loom doctor finding (structural keys property check already exists, packages/event-runtime/src/installOptional.ts:28) “event store seals X but snapshot store does not share the KeyStore”. Outward, no kernel change, no bump.
2. Whole-state sealing mode in EncryptingSnapshotStore. Chosen (user, 2026-10-14). A sensitive: "all" (or sealState: true) option that encrypts variables._waitAnswers and the other dynamic-key containers wholesale under the subject key. 1P plugin change; the snapshot payload stays an opaque ExecutionState to the store contract, so no SPI change if the decorator re-presents the same shape on load. Cost: snapshots lose partial readability (tooling that inspects variables), and a load needs the key (already true for static paths).
3. Kernel refuses the plaintext checkpoint. Chosen (user, 2026-10-14); it is a behaviour change. At snapshotStore.save the runtime refuses (or skips the save) when security.crypto.events is composed and the snapshot store lacks the shared KeyStore, generalising LogLifecycle.ts:155 to every save. Kernel behaviour change: a deployment that today checkpoints plaintext starts failing or losing checkpoints (a skip is safe for correctness but regresses read cost). No bump; a behaviour change needing a CHANGELOG “Changed” entry.

Placement. Option 1 and 2: 1P (@loom/plugin-crypto, loom doctor). Option 3: kernel, but the kernel only enforces “same KeyStore”, never the cipher.

Recommendation (superseded: the user chose 1, 2 AND 3). Options 1 and 2 together, no kernel change; defer Option 3 unless the user wants fail-closed by default (it is the only option that makes the unsafe configuration impossible rather than detectable, and it changes behaviour for existing deployments). Bumps: none for 1/2/3 (SNAPSHOT_STORE_SPI unchanged because the shape is preserved).

R-c — Head-of-line blocking is per execution

Confirmed, and designed. WorkflowContext.ts:2116 (return delivered) abandons the whole scan, and WorkflowRuntime.ts:4933 throws out of the whole notify scan; the scan loop covers every topic of the execution (for (const w of waits) at :4925-4932), so a sealed head on topic A holds topic B’s deliverable messages. D4 gives the reason: skipping would let nodes with different configs pick different select branches.

Repro (describe only). Execution parks on select([receive("a"), receive("b")]); send("a", sealedUnderNodeA) then send("b", "plain"). Capable node A consumes a (lowest inboxSeq) and completes the select with arm a. Incapable node B refuses the scan and never delivers b: correct, since delivering b on B would diverge from A’s branch.

Severity. Liveness only, on the incapable node only; capable nodes are unaffected and the log is never corrupted. The harm is a fleet whose only runner of an execution is incapable (a misconfigured deploy): the execution stalls behind a message nobody can open, with an onError every 16th poll and an operator escape hatch (discardInbox, D3; TTL expiry D10a if configured).

Why most options are unsound. Skipping a sealed topic is unsafe for the reason in D4. A weaker rule, “block only messages with a higher inboxSeq than the sealed one”, is already what FIFO plus a deterministic first-answered-arm gives for a single select; it does not unblock any case that is not already blocked by that ordering, so it buys nothing.

Options.

1. Document and accept (WONTFIX in the kernel). Head-of-line semantics are KEPT (user, 2026-10-14); the document-only stance is rejected as the whole answer: user chose the stronger option (keep + quarantine). Name it as designed (already Consequences residual 4); point operators at discardInbox and the journaled TTL. Zero change.
2. Observability, no semantics change. Kept as a by-product (the quarantine slice emits the blocked-state data); a standalone analytics advisory is not part of this decision. Surface a stuck-behind-sealed state: an onError payload carrying the blocked topics and the sealed message’s age, and an analytics advisory (@loom/analytics, advisory per ADR 0034) “execution X blocked behind sealed message M for > T”. 1P, no bump.
3. Operator quarantine verb. Chosen (user, 2026-10-14); the shape below is superseded by D9 (a dedicated event pair that holds the topic and preserves original order on release, not a MESSAGE_CONSUMED dead letter). A first-class quarantineInbox that journals the sealed message as a dead letter (MESSAGE_CONSUMED with payload: null plus a reason) so the head clears while preserving the audit trail, an explicit, human-triggered version of the existing discard. Possibly already expressible by discardInbox with a reason; if a new reason value is needed it is an additive field. Kernel only if a new journaled reason is required (EVENT_SCHEMA minor), otherwise no bump.

Placement. 1: docs. 2: 1P analytics. 3: kernel if it adds a journaled reason, else a recipe over discardInbox.

Recommendation (superseded: keep the semantics AND add the quarantine verb). WONTFIX the semantics (Option 1) and do Option 2 so the state is visible; do Option 3 only if discardInbox’s journal does not already record who discarded and why (check before proposing a schema bump). Bumps: none for 1/2; EVENT_SCHEMA minor only if 3 needs a new reason value.

R-d — A replay refusal writes no wake row

Confirmed. markContended (WorkflowRuntime.ts:4861) is reached only from notifyPass’s catch (:4847), i.e. the notify path. The replay refusal is raised inside run() (WorkflowContext.ts:1936-1945, handled WorkflowRuntime.ts:6874-6885), which warns, rejects the handles and returns: no timerStore.schedule, no event. The execution is left running (an unrelated resume already appended EXECUTION_RESUMED) or whatever it was. The ADR already admits this (Consequences, “a PURE replay refusal in run() writes no row”).

What recovers it today: recoverAbandoned on a capable node (recoverAbandoned.ts:280, status running or pending), but that needs an app to run the sweep with an isAbandoned predicate or a lease store (recoverAbandoned.ts:60-75, the predicate is REQUIRED with no default), and the staleness window must elapse. The incapable node’s own sweep skips it for a back-off window (:392, :491).

Minimal repro (describe only). Capable node A journals MESSAGE_CONSUMED with a sealed payload, then crashes before the next step; the execution is running. Incapable node B resumes (resume of any gate, or a sweep): replay hits WorkflowContext.ts:1936, B refuses; no timer row exists. Node A is later redeployed capable but no send, no timer, no resume arrives, and the app runs no sweep: the execution sits running indefinitely.

Severity. Liveness, narrow: needs a mixed fleet AND no sweep on a capable node. Not a confidentiality or correctness issue.

Options.

1. Document that a sweep is the contract. Rejected: user chose the stronger option. State that a deployment with an incapable node must run recoverAbandoned on a capable node; no code.
2. Write the durable wake on a replay refusal too. Chosen NOW (user, 2026-10-14). In the run() catch at WorkflowRuntime.ts:6874, call markContended(executionId) (same best-effort .catch(onStorageError)). Then TimerService’s inbox poll (INBOX_WAKE_AT) re-drives it on any node. But the wake row’s job today is to trigger the NOTIFY scan for a PARKED execution; for a running execution the notify path may find no unanswered park and retire the row, so the row alone does not re-run the flow. Either the poll must learn to resume-redrive a running execution carrying the row, or the row must be paired with a distinct reason. Kernel, small, no event change; needs a decision on what the poll does with a running execution (this is why it is not a one-liner).
3. Park the execution on the refusal. Rejected (breaks “appends nothing”; also user chose Option 2). Instead of leaving it running, journal a reserved park (like FLOW_DIVERGED_GATE, ADR 0113) so it is suspended and every existing wake path (notify, timers, recoverAbandoned’s parked-execution notify pass) applies. Strongest liveness, but it appends an event on a refusal, which contradicts D4 “appends nothing” and the property that an incapable node never writes about an execution it cannot read; it also adds a reserved gate name (EVENT_SCHEMA minor if the gate is a new journaled condition kind, none if it reuses SuspendError with a reserved on). Highest risk.

Placement. Kernel (1: docs).

Recommendation (superseded: Option 2 now). Option 2 only after the poll semantics for running executions are decided; until then Option 1 plus a recoverAbandoned recipe in docs/ naming this case. Reject Option 3 (breaks the appends-nothing rule). Bumps: none for 1 and 2; EVENT_SCHEMA minor only for 3.

Summary of decisions (user, 2026-10-14)

Residual	Real?	Decision	Placement	Bumps
R-a other sealed kinds	yes, incl. kernel-authored principal/outcome	stream-level capability marker written by the crypto plugin, checked before fold/replay	kernel + @loom/plugin-crypto	EVENT_SCHEMA 1.25.0 + EVENT_STORE_SPI 2.5.0 (shared)
R-b plaintext checkpoints	yes	BOTH: loom doctor + whole-state EncryptingSnapshotStore, AND the kernel refuses a plaintext checkpoint when the event store seals	1P + kernel	none beyond the shared bump
R-c head-of-line	yes, by design	keep the semantics by default; add an operator quarantine / release pair that HOLDS THE TOPIC (per-topic head-of-line) so a release preserves original order; a CAPABLE node AUTO-RELEASES a held message (user ruling 2026-10-14)	kernel	EVENT_SCHEMA 1.25.0 (shared)
R-d no wake on replay refusal	yes	write the wake row now; the timer poll re-drives a running execution that carries one	kernel	none

Accepted design (2026-10-14)

Everything below is design; no code exists yet. One coordinated version bump carries all schema and SPI changes, reserved FIRST (slice S1, the W2-0 pattern: freeze the shapes, ship the constants, the reserved fold handlers and the optional SPI method with no behaviour, then the behaviour slices fill them in). docs/ registries (compatibility matrix, spec section, cheatsheet, usage strings) are updated in slice S1, not in this ADR PR.

D7 (R-a) — The capability marker: per-envelope metadata.seal, no new event

Where it lives. The kernel assigns sequence numbers, so the crypto plugin cannot insert a marker EVENT into a stream, and a store-side side table would not commit atomically with the sealed row (a crash between them leaves sealed data unmarked, the worst failure). The marker therefore rides ON the sealed envelope, in EventEnvelope.metadata (an existing open Record<string, unknown>), written by EncryptingEventStore in the SAME append that seals the fields:

// EventEnvelope.metadata.seal  (EVENT_SCHEMA 1.25.0, reserved key)
interface SealMarker {
  v: 1;
  cipher: string;            // e.g. "loom/aes-256-gcm" (the algorithm id the envelopes carry)
  covers: readonly string[]; // sorted, deduped "EVENT_TYPE.dot.path" entries actually sealed in THIS envelope
}

It is stamped on exactly the envelopes in which at least one field was sealed (not on every envelope of an encrypting node), so covers is exact, never stale after a config change, and carries no key material, key id or subject. “Stream-level” is the aggregate: a stream is sealed iff any of its envelopes carries metadata.seal. This covers every kind the node seals through sensitive, including the ones defineConfig registers itself (EXECUTION_STARTED.principal.identity.*, OUTCOME_RECORDED.recordedBy) and STEP_COMPLETED.output, with no per-kind kernel code: the marker is derived from what was actually sealed.

The SPI (EVENT_STORE_SPI 2.5.0, optional, additive).

interface SealingCapability {
  readonly cipher: string;
  /** Every "EVENT_TYPE.path" this node's store seals AND therefore opens. */
  readonly covers: readonly string[];
  /** True iff this node can open an envelope stamped with `marker` (same cipher, every entry of marker.covers is opened here). */
  opens(marker: SealMarker): boolean;
}
interface EventStore {
  /** 2.5.0. Absent, or returning undefined: this store seals nothing and opens nothing. */
  sealingCapability?(): SealingCapability | undefined;
}

Listed in OPTIONAL_EVENT_STORE_METHODS, so every decorator must forward, override or refuse it (compile error otherwise): plain decorators forward; EncryptingEventStore overrides it from its sensitive map (live, so registerSensitive is reflected); ShardedEventStore FAILS CLOSED on disagreement (it defines the method when ANY shard has it, and a shard that lacks the method entirely counts as sealing nothing, i.e. undefined, so an encrypting shard beside a plain SQLite/InMemory shard throws; when NO shard has the method it is omitted, as for any all-plain fleet): if every shard reports an identical capability (same cipher, same covers) it returns that capability; if no shard seals it returns undefined; any other combination (some shards sealing and some not, different covers, different cipher) throws the exported, typed misconfiguration error ShardSealingMismatchError (carries the shard indexes and each shard’s cipher/covers, never key material). It is evaluated LIVE on every sealingCapability() call, because registerSensitive can change one shard’s capability at runtime. The reason is that two questions share this one method and must never diverge: D7 asks “what can this node open” and D8 asks “does this store seal at all” (non-empty covers); an intersection answers only the first, and would report a store that seals some paths on some shards as sealing nothing, letting D8 allow a plaintext checkpoint. A store that does not claim 2.5.0 simply lacks the method: that IS the fallback, and it means “incapable”, which is the correct reading for a store that cannot open sealed data. Conformance: runEventStoreConformance(..., { eventStoreSpi: "2.4.0" }) skips the new cases (the ADR 0129 option).

The check (kernel, before fold and replay). assertStreamReadable(executionId, events, store) is called at every funnel through which stored envelopes reach the fold or the VM seed (the implementer enumerates them; a fitness test pins the list, in the style of testTimeoutGuard): for each envelope with metadata.seal, if store.sealingCapability?.()?.opens(marker) is not true it throws SealedStreamUnreadableError(executionId, sequence, covers) before anything is folded, handed to a flow, journaled or authorised against. Cost is O(1) per envelope (a property read), not a payload walk, so it also replaces the pre-filter worry of the rejected shape-sniffing option. The principal is refused BEFORE the policy is consulted (user ruling 2026-10-14, from the S0 probe, PR #579). S0 found that on a raw node ownerForAuthz hands the policy owner === undefined for a sealed principal, so a denial rests ENTIRELY on the policy’s unknownOwner: "deny" default: with unknownOwner: "allow" a raw node ALLOWS a stranger to loadState (fail-open), and the legitimate owner is locked out too (closed, but not correct, and untyped). That is not acceptable for a kernel invariant. The kernel therefore refuses with SealedStreamUnreadableError (the denial-shaped refusal described under “The error”, independent of unknownOwner and of any policy) at ownerForAuthz’s read of first, BEFORE authzDenial runs: (a) when first.metadata.seal.covers names any EXECUTION_STARTED.principal.* entry (or OUTCOME_RECORDED.recordedBy for the outcome path) and this node cannot open it; and (b) as a structural backstop for an UNMARKED legacy stream, when the principal read for authorization is itself a ciphertext envelope (a plain object with an own string __loomEnc, an O(1) check on the small principal object, never a payload walk). (b) is the single place where an unmarked stream is refused on shape, because the alternative is a fail-open authorization decision; a plaintext principal is never refused. A system-granted internal read does not consult the principal and is unaffected. The refusal LOOKS LIKE A DENIAL to the caller (user ruling 2026-10-14; see “The error” below): it is an ExecutionDeniedError whose caller-visible shape is the ordinary unknown-owner denial, so a caller who names an id learns nothing about whether it exists or is sealed. The owner is not told the fix by the error; the sealed / key-unavailable detail goes to operators only.

The error: a denial to the caller, detail to the operator (user ruling 2026-10-14; removes the former “existence leak” accepted residual). A distinct typed refusal would be an existence oracle: a caller naming an id would learn that it exists and is sealed. So the stream-level refusal is not a new caller-visible outcome:

• SealedStreamUnreadableError extends ExecutionDeniedError (including the reason: "key-unavailable" variant, which stays a distinct internal cause, not a distinct class). Constructed as new SealedStreamUnreadableError(op, executionId, detail) it calls super(op, executionId, UNKNOWN_OWNER_REASON), where UNKNOWN_OWNER_REASON is the exact string the default executionAccessPolicy gives an execution whose owner it cannot establish ("the execution's owner is unknown to this policy", exported from one module so the two cannot drift). Everything a caller can observe equals the ordinary denial of the same call: name stays "ExecutionDeniedError" (the subclass does NOT override it), message, op, executionId, reason. (The call stack is NOT the same: a sealed refusal is built in the guarded read and re-thrown by sealedOp at the public method, not at denyOp; stack frames are not a trust boundary, see below.) The sealed detail (reason: "marker-unopenable" | "key-unavailable", sequence, covers) is NOT a property of the error: it lives in a module-private WeakMap<Error, SealedDetail> and is read only through sealedDetailOf(err), exported from the operator entry (@loom/workflow-runtime/operator, not the main index), so JSON-serialising or enumerating the error at an RPC/HTTP edge cannot leak it. Stack frames are not a trust boundary (an in-process caller already shares the process); edges serialise name, message, op, executionId, reason only.
• Where it is thrown. Only where the caller has NOT yet proven anything: the stream-level reads that precede or feed authorization (ownerForAuthz’s read of first, loadState, send, resume, cancel, discardInbox, quarantineInbox, releaseInbox, recordOutcome). It goes through sealedOp (one wrapper on EVERY public method that reads a stream: loadState, parkedAtSequence, followToHead, start on a caller-chosen id, resume, resumeAs, restoreFromState, cancel, acquireFence, fork, send, discardInbox, recordOutcome, and, for the kernel-internal callers, expireInbox and notifyInbox), which relabels the refusal with the op the call gates and fires onExecutionDenied(request, reason) where the ordinary denial of that op does (a plain query never audits its denial, so a sealed query does not either). The wording is what the same op gets for an ordinary existing execution with an UNKNOWN owner, including op-level rules (user ruling 2026-10-14, round 2): sealedOp consults the policy ONCE, with the owner UNSET (exactly the input an ordinary unknown-owner execution produces; never the sealed principal, so no sealed data reaches it), purely to PHRASE the error. A denial gives its reason (discard: "deny" reads "discard" is not permitted, as for any ordinary id); if that call would ALLOW (e.g. unknownOwner: "allow") the generic UNKNOWN_OWNER_REASON is used; a throwing policy falls back to the generic wording. The refusal itself stays UNCONDITIONAL (it happens even when the phrasing call would allow, and the S0 refusal-before-policy rule is unchanged: the verdict never decides anything). The same wording goes to onExecutionDenied. The one input it cannot reproduce is the workflow id ("", unreadable), so a policy with allowWorkflows words every sealed refusal as a workflow denial. A start on an existing sealed id gets the ExecutionAlreadyExistsError an existing plaintext id gets, not a storage error. The existing delivery-guard refusal SealedPayloadUnreadableError (D4/D5) is unchanged and stays a distinct error: it is raised on the node’s own notify/replay paths and on releaseInbox("deliver") AFTER the caller passed authorization on a plaintext, openable principal, so it tells the caller nothing they have not already proved they may know.
• What operators see. Every refusal is also reported through the runtime’s operator channel (onStorageError, the channel the delivery-guard refusals already use, plus the structured log line) with the real cause: the error object itself (use sealedDetailOf), carrying executionId, sequence, covers and reason. The first-then-every-16th reporting and the sealedBackoff apply to it unchanged. A caller sees “denied”; an operator sees “sealed, cannot open here, fix: run a capable node / supply the key”.
• The EXECUTION_DENIED audit fact. An ordinary denial of resume/cancel best-effort records EXECUTION_DENIED { op, reason?, actor? }. For a sealed refusal the fact, if written, would be byte-identical in shape to the ordinary denial’s (op, the GENERIC reason, the caller’s own actor; never covers, sequence, reason: "key-unavailable" or any cipher or key detail), so it could not leak more than today’s. Decision (confirmed by the user 2026-10-14): it is NOT written. (1) The ADR 0130 invariant is “a refusal appends nothing”; an incapable node appending a plaintext fact into a log it cannot read is exactly what the guard exists to prevent, and the unfenced afterSequence + 1 append would also let any stranger who can reach an incapable node burn sequence numbers on the owner’s stream (a DoS on the capable writer). (2) The audit need is met without it: onExecutionDenied fires (generic reason) and the operator channel carries the true cause. S2 asserts the log is untouched. If a deployment later wants the fact, the shape above is the only permitted one.
• How internal code still tells it apart. instanceof SealedStreamUnreadableError (an ExecutionDeniedError subclass check, so it must be tested BEFORE any generic ExecutionDeniedError handling), and a brand guard isSealedUnreadable(e) (a Symbol.for("loom.sealedUnreadable") property, true for BOTH SealedStreamUnreadableError and the existing SealedPayloadUnreadableError, which cannot share a base class with ExecutionDeniedError). TimerService.fireInbox, recoverAbandoned, the replay refusal, the notify scan and the back-off maps test the brand guard, so the ADR 0130 back-off and “first then every 16th” reporting apply unchanged. packages/agent/src/toolFailure.ts (and its fitness test) classifies both TERMINAL, and checks the brand BEFORE the ExecutionDeniedError branch; a model sees only the generic denial text. Like the existing refusals it never fails the execution.
• Indistinguishability, stated exactly. Under the default deny-unknown-owner policy a missing id and a sealed id both return the same ExecutionDeniedError for every gated op. Two configurations keep an oracle, both of which already expose unsealed executions to a stranger: no authorizer at all, and a policy with unknownOwner: "allow" (documented UNSAFE), where a missing id is “not found” rather than denied. A custom ExecutionAuthorizer whose unknown-owner reason differs from UNKNOWN_OWNER_REASON shows a different string: such a deployment should return the exported constant for an unknown owner. These are configuration caveats, not residuals of this design.

What a sealed refusal reveals, exactly (the R3 residual, unchanged). A sealed refusal reveals what the denial of an ownerless EXISTING execution already reveals, and nothing more. The R3 existence rule applies to an identified caller exactly as before: a missing id returns null from loadState / No execution found from resume, so an identified caller can tell “missing” from “denied” for an ordinary existing id as well; a sealed id adds nothing to that. One stated difference: the onExecutionDenied request carries workflowId: "" (the workflow of an unreadable stream is unknown). The operator channel (onStorageError, first then every 16th) is the only place the real cause appears.

A node whose KeyStore lacks the key (S0 fix-round finding, PR #579; user-directed 2026-10-14). A defineConfig node with crypto but a DIFFERENT or EMPTY KeyStore is the same fact as “cannot open”, reached through the other door: the marker (covers, cipher) matches, but the key for the envelope’s subject is absent here. Today the encrypting store reads keys.get(subject) === null and throws PayloadShreddedError from inside the read, before the authorizer, on loadState/send/resume/discardInbox/recordOutcome (only cancel reaches the authorizer, typed-denied): closed, but UNTYPED, and it reports “erased” for what is really “key not here”, conflating a misconfigured node with a crypto-shred. The fix has two parts.

1. Distinguishing the two (the KeyStore cannot). KeyStore.get returns null for “never existed OR destroyed” by contract (keystore.ts), so the key store alone cannot tell them apart and the SPI is not changed. What CAN tell them apart is the log: a real shred (shredSubject) writes a durable PAYLOAD_SHREDDED tombstone to every affected execution at shred time (and ADR 0106 compaction folds it into the snapshot’s state). So, when the key is null, EncryptingEventStore (and EncryptingSnapshotStore, EncryptingSnapshotStore.ts:92) decides: a tombstone present for the subject (the raw tip of the stream, or the latest snapshot’s folded shred record, reusing the shred-status helper in shred.ts) => a real shred, PayloadShreddedError exactly as today (erasure semantics, tolerant onShredded modes, tombstone-aware cancel all unchanged); NO tombstone => KeyUnavailableError. The snapshot store throws PayloadShreddedError only when it can PROVE a shred (the snapshot’s folded record, or, with the optional EncryptingSnapshotStoreOptions.events, a tombstone in the raw log); otherwise KeyUnavailableError. (Round 2: the earlier “without events keep PayloadShreddedError” made a wrong-key node report a LIVE execution as erased, an existence oracle that was not denial-shaped and not audited.) A snapshot taken before a shred cannot record it, so the kernel’s own probe (a key-unavailable snapshot load) reads the WHOLE stream and looks for PAYLOAD_SHREDDED itself (a tombstone sits above the snapshot, not at sequence 0) and lets the real erasure error win; cancel’s tombstone-aware path and the back-offs therefore still key off it. A caller outside the kernel (the S4 content GC) may see KeyUnavailableError for a shredded subject’s pre-shred snapshot; it accepts that name only beside the execution’s own PAYLOAD_SHREDDED tombstone. Cost: with events, every key-unavailable read does a full inner.read of the stream to look for the tombstone (a positive answer is cached, a negative one cannot be: a tombstone may appear any moment); it is bounded because a refused stream is backed off by every poller and reported first-then-every-16th, and there is deliberately no cache. A marked envelope that a PRESENT but different key cannot open reads as KeyUnavailableError too (same fact: this node cannot open it), and a crypto node’s live subscribe drops a marked envelope it cannot open instead of handing on ciphertext (new, plugin-level, carries a stable code, the subject id and the stream id, never ciphertext). The residual ambiguity is stated, not hidden: between keys.destroy(subject) and the tombstone write of a shredSubject that failed part-way (the documented retry window, ShredIncompleteError), a read has no tombstone and reports KeyUnavailableError, a refusal (denial-shaped to the caller, D7) that clears when the retry completes; it never reports plaintext and never mislabels a shred as available. Round 4: the retry window needs durable intent. A wrong-key node (key null, no tombstone) is indistinguishable by reads alone from that window, and treating it as the window let eraseSubject report “erased”, shredSubject append a FALSE tombstone (destroying an absent key is a no-op, while the real key lives on capable nodes) and exportSubject return a redacted export. So shredSubject now writes, BEFORE keys.destroy, an ERASURE_PENDING (existing kind: ignorable for park classification, folded as pure audit into _erasure, no schema change) with { subjectId } into each affected execution’s log that does not already hold one; the order is intent, destroy, tombstone (a crash after the intent leaves the key present and the retry destroys it; a crash after the destroy leaves intent without a tombstone, which is the window). If the key is already absent and sealed data is present with no intent, it throws KeyUnavailableError having written and destroyed nothing. EraseCoordinator and exportSubject accept a KeyUnavailableError carrying a redacted view only when that view holds the same subject’s intent (erasureWindowView); otherwise they rethrow. The tombstone now follows the intent in sequence (head + 2 on a fresh erasure). Round 5: intent is not destruction. A DANGLING intent (a crash before keys.destroy, or intent N of M failed) leaves the key alive on the node that wrote it, so a retry on a wrong-key node saw intent, destroyed nothing and tombstoned (or exported a redacted view). Only the key store can prove destruction: the OPTIONAL KeyStore.wasDestroyed(subjectId) is true only if THIS store destroyed (tombstoned) the key (additive; InMemoryKeyStore implements it; conformance covers it, skipping stores without it). With the key absent and an intent present, shredSubject proceeds ONLY on wasDestroyed === true, else throws KeyUnavailableError having written nothing, unless the operator passes force: "key-destroyed-elsewhere" (an assertion that the key died in the authoritative store). erasureWindowView(err, destroyed) (async) accepts the window only with the same proof, supplied to EraseCoordinator/exportSubject/compliance.erase as destroyed; absent, it fails closed. Round 6: wiring and the operator assertion. defineConfig defaults destroyed to (s) => keys.wasDestroyed(s) when the key store behind security.crypto implements it (an explicit compliance.erase.destroyed overrides; a store without the method gives no proof and still refuses). destroyed stays a PROOF function only: () => true is no longer suggested, since it produces a false erasure record for a key not destroyed in the authoritative store. The operator assertion is the per-call force: "key-destroyed-elsewhere" on eraseSubject / exportSubject (mirroring ShredOptions.force), passed through to the shredder and to the window check.
2. Typed, and before the policy. KeyUnavailableError.code lives in @loom/event-runtime beside the store error contract (the crypto plugin already depends on it, the kernel does not depend on the plugin). The kernel’s read funnels (the same list as assertStreamReadable, including ownerForAuthz’s read of first) translate it into SealedStreamUnreadableError with reason: "key-unavailable" (versus "marker-unopenable" for the capability check), so the caller sees ONE outcome for “this node cannot open this stream” whichever way it cannot (the generic denial above; the cause is operator-only), the isSealedUnreadable handlers (back-off, first-then-every-16th) apply, and the authorizer is never invoked. PayloadShreddedError stays exclusively the erasure outcome. Inbox streams keep their non-loud read (D9 of ADR 0110) and are unaffected.

What an unmarked stream means (exactly). An envelope without metadata.seal makes NO claim. It is either truly plaintext, or it was sealed by a writer that did not stamp (a pre-1.25.0 node, a third-party sealing decorator, a store whose decorator stack dropped metadata). The kernel therefore does not read absence as “safe to open”: it behaves EXACTLY as it does today on that envelope, with the D2/D4 shape guard still guarding inbox payloads. Nothing is refused because a marker is missing, and nothing existing is bricked: a pre-1.25.0 stream with no markers folds and replays byte-identically on every node. The marker only ADDS refusals, and only where a writer said “sealed” and this node cannot open it. Documented limit: a stream sealed by an unstamping writer keeps the pre-0130 mixed-fleet exposure for its non-inbox kinds; upgrading the writer fixes new envelopes, and old ones are not back-filled (no rewrite of a log).

Rolling upgrade. A pre-1.25.0 node ignores unknown metadata keys (the fold reads only the keys it knows; to be verified by a test in S1), so it keeps today’s behaviour on marked streams; it is a node that does not yet have the protection, not one that corrupts differently. Deploy order: upgrade every node, then rely on the marker. The envelope grows by ~60-200 bytes per sealed envelope (the covers list); hash chains (plugins/audit-chain) hash the envelope the wrapper appended, so the marker is inside the chain, deterministically.

Replay byte-identity. Logs written with no crypto plugin are byte-identical to before. For an encrypting node the envelope already differs run to run (a random IV in every ciphertext), so DST seeds that compare logs run with the deterministic cipher seam (user ruling 2026-10-14, slice S2): EncryptingEventStore (and EncryptingSnapshotStore) take an injectable IV/nonce source, ivSource?: (ctx: { streamId: string; sequence: number; path: string }) => Uint8Array, whose production default is crypto.randomBytes(12). The seam exists so a seeded simulation can derive the IV from (seed, stream, sequence, path) and compare logs byte for byte. A reused IV under AES-GCM breaks confidentiality, so the seam is exported ONLY from a @loom/plugin-crypto/testing subpath (never the main entry), the seam is a BRANDED value (seededIvSource mints it; the brand is a module-private WeakSet, so a plain function, or an object that merely carries the right property, is refused), it runs ONLY under a test (NODE_ENV === "test" or VITEST set; fail closed: an UNSET NODE_ENV is refused, and "production" always is), and a fitness test pins that the default path calls randomBytes and that no non-test source passes the seam. The marker (a pure function of the config and the event type) adds no new nondeterminism.

False positives. None from content: the marker is written by the sealer, never inferred from a payload, so an app payload with a string __loomEnc key (residual 3) is not affected by D7. (An app can still forge a metadata.seal? No: metadata is set by the runtime and the store; a flow cannot set envelope metadata. The send rule D6 is unchanged.)

D8 (R-b) — Whole-state snapshot sealing (1P) and the kernel refusal of a plaintext checkpoint

1P.

• EncryptingSnapshotStore gains sealState: true (mutually exclusive with sensitive, which stays for compatibility): the folded ExecutionState is serialised and sealed as ONE envelope under the same subject key as the events (so a shred erases both), and stored as { executionId, workflowId, <the snapshot's index fields the store contract needs in the clear>, state: { __loomEnc: ... } }. The exact plaintext index fields are whatever SnapshotStore.loadLatest/list must filter or order by (the implementer reads the contract; none of them are payload). This is what covers the dynamic variables._waitAnswers.<waitId> keys that no static path can name. A load with a destroyed key throws PayloadShreddedError, as today. Slot binding (AAD). The whole-state envelope is a new format, so it is bound to its slot: AES-GCM additional authenticated data is loom-snap-v1|<executionId>|<sequence>, and the envelope carries aad: "loom-snap-v1" so the format is self-describing (the bound context itself is not stored; it is re-derived from the row on load). A sealed row copied into another execution’s slot, an older snapshot rolled into a newer sequence, and an envelope without the marker all fail to open with the typed SnapshotUndecryptableError (never raw GCM text, and distinct from PayloadShreddedError so a defect cannot read as an erasure). encryptValue/decryptValue take AAD optionally, so field-level (static-path) envelopes are unchanged. Downgrade. legacyPlaintext: "load" | "refuse" (default "load": old snapshots stay readable, per the migration paragraph below): "refuse" throws PlaintextSnapshotRefusedError on any state that is not a whole-state envelope, so a plaintext row cannot be substituted for a sealed one once none legitimately remain. Legacy opening. A snapshot written by a static-path store has nested envelopes; in sealState mode the old paths are unknown, so each envelope-shaped value is tried and left exactly as stored if it does not open (an application’s own ciphertext under another key no longer fails the load); the optional legacySensitive paths restrict this to the named paths (without it, an application value sealed under the SAME subject key by the same helper is opened). Census. loom doctor reads the wiring, not the data at rest: SnapshotStore has no enumerating method, so counting plaintext snapshots still under a sealState store would be a per-backend full scan. That residual is documented, not detected. Content GC. A sealState snapshot read RAW is one opaque envelope, so sweepUnpinnedContent cannot see the refs it cites; an erasing pass throws SealedSnapshotScanError (“pass the decrypting snapshot store”) and a dryRun lists sealedSnapshots. The store exposes the structural marker sealsWholeState: true (a plain property, like keys, so neither loom doctor nor the kernel depends on @loom/plugin-crypto).
• loom doctor finding (snapshot-plaintext-under-encrypting-events, severity error): event store seals (sealingCapability has covers, or the structural keys probe) while the snapshot store neither shares the KeyStore nor reports sealsWholeState; also warns when sensitive (static paths) is used, naming _waitAnswers. Names the fix and the allowPlaintextCheckpoints escape.
• docs/compliance.md is updated in the same slice (the _waitAnswers caveat becomes “use sealState”).

Kernel (behaviour change). The kernel knows the event store seals from the D7 capability: eventStore.sealingCapability?.() with non-empty covers. No new SPI is needed: the snapshot store is judged structurally, as LogLifecycle already does for compaction (keysOf), extended with sealsWholeState.

• Rule. When the event store seals, a checkpoint (snapshotStore.save, the on-settle and periodic sites in WorkflowRuntime, and LogLifecycle.compact, which refuses a plaintext checkpoint with PlaintextCheckpointError in addition to its shared-KeyStore check - the earlier claim that it “already had a stricter check” was wrong: a static-path store on the same keys passed it) is permitted iff the snapshot store sealsWholeState === true and shares the event store’s KeyStore. A static-path EncryptingSnapshotStore is NOT sufficient (it leaves _waitAnswers plaintext).
• When it fires. At runtime CONSTRUCTION (fail fast, the primary form): new WorkflowRuntime/defineConfig throws PlaintextCheckpointError (a config error, listing the two ways out) when both stores are known and the rule fails. Defensively again at each save (stores can be swapped or composed late): there it throws the same class, which the existing best-effort checkpoint try/catch surfaces through onStorageError WITHOUT failing the run (a checkpoint is an optimisation; the log is the source of truth). A runtime with no snapshot store is unaffected.
• Error type. PlaintextCheckpointError extends Error in @loom/workflow-runtime (fields executionId?, reason: "not-sealed" | "keys-not-shared"), not retryable.
• Escape hatch (kept, confirmed by the user 2026-10-14). security.crypto.allowPlaintextCheckpoints: true (explicit, off by default; the opt-out for a snapshot store on an encrypted volume and the migration path) skips the rule and makes loom doctor downgrade to info. It is the migration path for a deployment that cannot move yet.
• Default snapshot store (user ruling 2026-10-14). With storage: { kind: "custom", eventStore } and no snapshotStore, defineConfig builds the in-memory default with sealState on the event store’s KeyStore when that store encrypts, instead of refusing to boot; an EXPLICIT snapshot store that fails the rule is still refused. Surface (S3). LoomApp.security.crypto.allowPlaintextCheckpoints is exposed for loom doctor; the archive snapshot store of LogLifecycle follows the same rule (unsafe-archive) and has its own doctor finding archive-snapshot-plaintext-under-encrypting-events. The rule is evaluated on the runtime’s seal-guarded event store (capability and keys carried up), and the fork seed checkpoint is skipped (reported through onStorageError) rather than failing a fork.
• Residuals (review of #589, accepted). (a) Forged marker. The rule is structural: a plaintext tee/cache decorator that inherits both keys and sealsWholeState from a sealing store passes. The threat is misconfiguration, not a hostile host object. (b) Subject mapping. The sealed default keys snapshots by executionId; the event store’s subjectOf is private, so an app with a different mapping must build the snapshot store itself (compliance.md). (c) Volatile default. The in-memory default is node-local; defineConfig warns through onWarn when it is chosen for an event store it cannot see to be in-memory, and falls back to the plain store (so the kernel refuses, as before) when the keys marker is not a KeyStore. (d) Key-less nodes. A node with no keys that shares a snapshot table with sealed checkpoints cannot fold an execution (foldState reads the snapshot first), so operator ops on it (for example quarantineInbox from an incapable node) are refused as an unknown owner; give such a node its own snapshot store (it folds from the log) or run the op on a capable node. (e) Save-time refusals are reported to onStorageError first-then-every-16th per execution.
• Migration. An existing deployment with an encrypting event store and a plaintext snapshot store stops booting with a message naming the fix, which is the intended behaviour change; CHANGELOG “Changed” (breaking-by-design) entry, and the loom doctor finding ships in the PREVIOUS slice so it can be run first. Snapshots already written in plaintext are not rewritten or deleted (no pruning, user ruling); they stay readable. A sealed snapshot loaded by a node that cannot open it (top-level __loomEnc in the loaded state) raises SealedStreamUnreadableError from the load path, never a half-state.

D9 (R-c) — quarantineInbox / releaseInbox: a journaled per-TOPIC hold that preserves original order

User rulings (2026-10-14). (1) The held bytes live in the reserved stream __loom_quarantine__:<chainId>, default cap 256 per chain. (2) A release MUST preserve the message’s ORIGINAL order. The first cut of this section (release re-sends the message at the topic’s tail) is rejected, and the “re-delivered at the tail” residual is gone.

What “original order” means, and the three designs

The inbox promises FIFO per topic and nothing across topics (D5 of ADR 0110; select takes the first arm, in park order, that has a deliverable message). “Original position” therefore means: relative to the other messages of the SAME topic. The folded state is one monotone number per topic, _inbox.cursors[topic]: every stream entry at or below it is disposed of (consumed, skipped as a duplicate, discarded), and the next message delivered is the first one above it. Once a later message m2 of a topic has been delivered, its cursor is above the held m1, and any way of delivering m1 afterwards shows the flow m2 before m1.

Design	Original order?	Verdict
(i) The quarantine HOLDS THE TOPIC: no later message of that topic is delivered until the held one is released or discarded; other topics continue	Yes, by construction. The flow can never see m2 before m1, because m2 is never delivered while m1 is held.	Chosen
(ii) A cursor with holes (a consumed-set / skip-list): later messages flow, the held one is delivered later under its original inboxSeq	No. The flow sees m2 then m1: out of order, and select can already have chosen a branch from m2. Only the label (inboxSeq) is original. Rejected.	rejected
(iii) A rewindable cursor	Impossible: it would re-deliver consumed messages (their answers are already folded and replay serves them), breaks cursor monotonicity and the dedupe window, and makes replay depend on when the rewind happened.	rejected

Plainly: only (i) honours original order, and it changes residual (c) from per-EXECUTION head-of-line (D4) to per-TOPIC head-of-line for a quarantined topic. The trade-off, stated: a quarantine is a decision that the held topic stops delivering to every node that cannot open the held message, and to capable ones only until the first capable node lifts the hold by itself (“Auto-release” below); it stays held for as long as NO capable node runs, or until an operator releases or discards it. The topic’s later messages accumulate behind the hold (they are bounded by the per-topic backlog and cap, InboxFullError for THAT topic only on per-topic streams). What the user gains over today: the execution’s OTHER topics deliver again (today a sealed head stalls them all), the stall is a journaled fact every node agrees on (deterministic select, no per-poll refusal and onError storm for the held topic), and the sealed message no longer pins the original stream (see “Cursor and trim”). If a deployment would rather have the topic keep flowing, its answer is discardInbox (the message is gone), not a quarantine that reorders.

The facts (owner’s log, kernel-stamped, folded only from a kernel-stamped envelope like the discard facts; both are ignorable for the fence, like MESSAGE_CONSUMED)

// INBOX_QUARANTINED: the topic's head message is held.
{ topic: string; inboxSeq: number; key: string /* dedupeKeyOf(msg), 64 hex */; messageId: string /* raw, <=256 */;
  heldAt: number /* seq in __loom_quarantine__:<chainId> */; operator: Owner; reason?: string /* <=256, event only */ }
// folds: _inbox.cursors[topic] = max(cursor, inboxSeq)          // the original entry is disposed of
//        _quarantined[topic].push({ key, inboxSeq, heldAt, state: "held" })   // FIFO queue per topic
//        NOT added to the dedupe window (the release delivery must not read as a duplicate)
//        ignored when total entries would exceed QUARANTINE_FOLD_CAP (1024, a fold constant)

// INBOX_RELEASED: the topic's OLDEST held entry is released, or dropped.
{ topic: string; key: string; messageId: string; disposition: "deliver" | "discard";
  operator?: Owner;   // present on a MANUAL release (releaseInbox); absent on an automatic one
  auto?: true;        // present, and only with disposition "deliver", on an AUTOMATIC release (below)
  reason?: string }
// exactly one of `operator` / `auto` is present; a fact with both or neither is ignored by the fold
// folds: only for the topic's first entry, and only if its state is "held" (else ignored):
//   "deliver": state = "released" (deliverable, still ahead of everything above the cursor)
//   "discard": entry removed; key added to the dedupe window (a sender's retry of that id is a duplicate)

_quarantined is { [topic]: Array<{ key, inboxSeq, heldAt, state: "held" | "released" }> } (a topic whose queue becomes empty is deleted from the record, and the _quarantined variable itself is deleted when the record becomes empty, in both MESSAGE_CONSUMED and the INBOX_RELEASED discard branch, so a full hold, release and consume cycle returns to exactly the never-quarantined shape): no reason, no payload, ~100 bytes per entry, hard-capped by the fold at 1024 entries (a fold cannot read a runtime setting, the lesson of ADR 0110 D6; the runtime default quarantineMax is 256 and may be raised to the ceiling, never above). The queue shape is always released? held* (AT MOST ONE released entry: the fold can only release the oldest held one, and the delivery consumes it before the next release; a carried seed with two released entries for a topic is invalid and ignored) because release only ever takes the OLDEST held entry (a release of a younger one is invalid: releasing it first would deliver out of order).

One existing handler changes, gated on _quarantined. MESSAGE_CONSUMED additionally removes the topic’s first entry when it is released and its key equals the fact’s messageId (the dedupe key). A log with no _quarantined folds byte-identically to before (golden vectors, S1). The delivery fact is otherwise unchanged: inboxSeq is the original sequence, which is at or below the cursor, so the cursor does not move and the dedupe window records the key like any delivery.

Delivery rule (the one rule every node, the live run and the notify scan apply, from FOLDED state only)

For a wait on topic T, in park order as today:

1. If _quarantined[T] starts with released entries, the next message is the FIRST released entry, read from the quarantine stream at heldAt (it precedes everything above the cursor, which is its original position). It passes assertDeliverable like any message (an incapable node refuses, D4, and the refusal is again the execution-wide head-of-line for that node).
2. Else if _quarantined[T] starts with a held entry: a node that CAN open the copy AUTO-RELEASES it (below) and delivers it as in step 1, in the same atomic batch; a node that CANNOT open it has NO deliverable message on T and skips T entirely (does not read it, does not refuse, does not report except the one operator signal “held, awaiting a capable node”). Other topics are scanned. Skipping is deterministic because it depends on a journaled fact plus this node’s own ability to open the copy, which the batch’s fence re-validates; the choice is recorded in the log before it reaches a flow, and replay serves recorded answers.
3. Else T is read past its cursor as today.

select across topics therefore stays deterministic across nodes: the set of satisfiable arms is a function of the folded state plus the inbox contents, as before; the hold removes T’s arm from the set of every node that cannot open the held copy, and a node that can opens it by journaling an auto-release first, so the branch a flow takes is always the one recorded. A select whose only inbox arms are held topics parks until a release (or a timer/gate arm fires).

Auto-release (user ruling 2026-10-14): a capable node lifts the hold by itself

Semantics, plainly. A quarantine is a CAPABILITY hold: it exists because some node could not open the head message, and the topic is held, in its original order, until a node that can open it exists. As soon as one does, the system releases the message with no operator: the capable node delivers it FIRST, from the quarantine copy, with its original inboxSeq, ahead of everything above the cursor, and the topic resumes. The operator verbs remain: quarantineInbox (the initial hold), releaseInbox("deliver") (force the same thing now, from a capable node), releaseInbox("discard") (drop it; the only way out when no node can ever open it).

Reads of the quarantine stream are non-loud. inboxChainOf gives the quarantine stream the inbox’s non-loud read in EncryptingEventStore (a missing key redacts the payload to the SHREDDED marker instead of throwing, ADR 0110 D9). So a release or auto-release must NEVER treat a redacted copy as deliverable (a copy whose payload reads as the marker is a key-unavailable refusal in the D7 split, not a message), and quarantineInbox’s not-sealed check must not be fooled by it: a redacted read is not evidence that the message is openable or that it is plain.

Who initiates the hold (decision, confirmed by the user 2026-10-14). The initial quarantine STAYS an operator act; an incapable node does not auto-quarantine. Reasons, in order of weight: (1) ADR 0130’s core guarantee is that an incapable node APPENDS NOTHING; auto-quarantine would make every raw poll write a copy and two facts, and many incapable nodes would contend to write them; (2) without a hold the default (D4) already recovers by itself: a capable node that reaches the message delivers it in order, so auto-release only matters once someone chose to unblock the OTHER topics, which trades away that topic’s flow and is the operator’s policy call, not the kernel’s; (3) it keeps the hold bounded by quarantineMax under operator control, so a burst of sealed traffic cannot fill the quarantine stream and the folded _quarantined unattended. A deployment that wants automatic initiation can drive quarantineInbox from its own onError handler (the refusal reports the blocked topic and inboxSeq); the kernel does not decide it. A hold is NOT a moderation hold: it is only permitted for a message unopenable on the quarantining node (not-sealed otherwise), so a capable node lifting it is always the intended outcome. To stop a message, use discardInbox/releaseInbox("discard").

When a capable node releases: lazily, at the one moment a delivery would happen, never on a timer of its own. The triggers are exactly the existing delivery points, so there is no new poller, no new lease, and no new failure surface:

1. the notify scan (the post-park read afterInboxPark, a send’s wake, TimerService.fireInbox, recoverAbandoned), when the parked execution waits on a topic whose first entry is held;
2. the live run, when the flow reaches a receive/select arm on a held topic (satisfyFromInbox applies the same delivery rule);
3. a quarantine writes the INBOX_WAKE_AT row (as any sealed refusal does), so a capable node’s poll reaches a parked execution that nobody else would otherwise wake. The wake row is NOT retired by an incapable node while a waited topic is held and unopenable to it (the poll backs off with sealedBackoff and reports first-then-every-16th “held, awaiting a capable node”); it is retired by whichever node delivers.

A topic nobody waits on is not released (nothing would consume it); it releases the first time something does. That is the same laziness as an unconsumed ordinary message.

The act (one atomic batch). The capable node (a) reads the copy at heldAt from the quarantine stream, checks its key matches the entry, and passes assertDeliverable (a node that cannot open it is not capable: it skips as above); (b) appends, under the park’s conditional fence (advanceInbox/the notify batch, ADR 0118), ONE batch: INBOX_RELEASED { topic, key, messageId, disposition: "deliver", auto: true } immediately followed by MESSAGE_CONSUMED { inboxSeq: <original>, ... } for the copy and then the park’s answer, exactly the composition the notify path already uses for any delivery. The fold sees released then the consumed drop in order; the cursor does not move; the dedupe window records the key. Because both facts are in one batch there is no window in which the entry is released but undelivered, so the manual-release crash window (“3” below) does not arise on this path; a manually released entry (an operator on a capable node, or a manual release followed by a crash) is delivered by the same step 1 on the next notify, unchanged.

The auto marker; how authz treats it. auto: true replaces operator (an INBOX_RELEASED never has both, the fold ignores a fact that has both or neither), chosen over a reserved system principal because a reserved actor name could be impersonated by a caller who sets principal.actor, whereas auto is a field only the kernel stamps (the fold honours it only from a kernel-stamped envelope, like the other facts). An automatic release is NOT a caller operation: it takes no principal, is not an ExecutionOp, and consults no authorizer, for the same reason the delivery of any message does not: the node is already driving this execution, and the release grants nothing, since the bytes were authorized when they were sent and it delivers the very message the cursor was going to deliver, only now able to open it. It cannot discard, cannot reorder, and cannot release anything but the topic’s OLDEST held entry. The audit trail reads INBOX_RELEASED auto versus INBOX_RELEASED operator=...; the stamping envelope carries the node that did it. A runtime option kernel.inbox.autoRelease (default true) turns it off for a deployment that wants every hold lifted by an operator; it is a node-local policy, never read by the fold (the hold remains a shared journaled fact).

Exactly one node releases; races. Both the delivery and the release are one append under the park’s conditional fence, and staleInboxBatch is extended with one more clause: a batch carrying an auto-release is refused when the topic’s first entry is no longer held in the refolded state. Therefore: two capable nodes race, the fence admits one, the other refolds, sees the entry consumed (cursor/dedupe), and carries on with the next message, never delivering it twice (at most once, by the dedupe window and the stale-batch check); an auto-release races a manual deliver: the first to land wins, the loser finds the entry released/gone and is a no-op (manual deliver is idempotent); an auto-release races a manual discard: if the discard lands first the auto batch is stale and refolds (the message is gone, the topic is unblocked), if the auto-release lands first the discard is invalid (not held) and nothing is dropped; it races a notifier that built its batch before the hold or before the release: refused by the same clause (no delivery from a held topic except the first released entry). Exactly one outcome on every node.

Liveness. If no node that can open the message ever runs, the topic stays held, indefinitely, by design: the message stays safe and the OTHER topics keep flowing. That is documented, not hidden; the operator signal is the first-then-every-16th “held, awaiting a capable node” report and _quarantined on the analytics surface, and the exit is releaseInbox("discard"). A capable node starting later releases it on its first poll of the wake row.

Cursor and trim

Advancing the cursor at quarantine is what keeps the held message from pinning the stream: the inbox trim deletes the consumed prefix up to the cursor, so the original entry is reclaimable (the copy is the retained record). On the legacy shared stream the messages that arrive BEHIND the hold on T remain unconsumed and pin trimming exactly as an unconsumed hot topic does today (ADR 0110 D8’s pin); per-topic streams (D10b) confine that backlog to T. Operators who quarantine should run per-topic streams.

Retention, copy-first

quarantineInbox first copies the head message verbatim into __loom_quarantine__:<chainId> as an INBOX_MESSAGE with a deterministic envelope id (quarantine:<chainId>:<key>:<inboxSeq>, so a retry is a duplicate append, treated as already copied), preserving the envelope bytes and its metadata.seal: it is appended through the node’s own store, which on an incapable node is raw (the sealed value is never opened, and an encrypting decorator must not double-seal a value that is already an envelope; S5 test), then journals INBOX_QUARANTINED with heldAt. Crash between them: an orphan copy (harmless, a retry reuses it), never a cleared head without a retained payload. The prefix __loom_quarantine__: joins the reserved-stream registry (S1) and inboxStreamsOfChain, so deletion with the chain, the compliance purge, reclaimInbox and the crypto subject mapping (inboxChainOf, same owner subject as the chain’s inbox) all cover it. Copies are trimmed by prefix once they are positively disposed of (see the implementation notes, “Retention”: never a copy that is live or in flight).

Bound. quarantineMax (default 256, ceiling 1024) bounds the RETAINED span of the quarantine stream per chain (head minus the oldest still-pinned copy, orphans and in-flight copies included), refusal invalid at the cap. It also bounds the folded _quarantined size and therefore snapshots.

Verbs

• quarantineInbox(executionId, { topic, reason? }, { principal }): holds the topic’s HEAD message (the first non-duplicate above the cursor, after journaling any duplicates ahead of it in the same atomic batch, as the notify scan does), refused not-sealed unless that message is sealed and unopenable on THIS node (the verb is not a general message-skipper; discardInbox already is that). Result: { messageId, inboxSeq, heldAt }.
• releaseInbox(executionId, { topic, messageId, disposition? }, { principal }): disposition defaults to "deliver"; the message must be the topic’s OLDEST held entry. "deliver" requires this node to be able to open the copy (read it, assertDeliverable; an incapable node gets the typed SealedPayloadUnreadableError and nothing is appended), then journals INBOX_RELEASED { operator } and runs the notify path, which delivers it by the delivery rule above, at its original position. It is what auto-release does, on demand and attributed to a principal; with auto-release on it matters only to force the release now. "discard" needs no ability to open (the escape hatch for a message no node can open), journals the drop, and unblocks the topic. Releasing anything not held is invalid; repeating a completed release is idempotent (nothing appended).
• Authorization. As discardInbox: a principal is REQUIRED (no system-grant stand-in), refused with ExecutionDeniedError before anything is read (no existence oracle), against the chain head’s owner. New ExecutionOps "quarantine" and "release" (deliver); a "discard" disposition is judged as "discard", since it destroys the message. executionAccessPolicy maps the new ops to "deny" by default like "discard". Refusals: a sibling InboxQuarantineRefusedError with the InboxDiscardRefusedError reasons (not-parked, contended, invalid) plus not-sealed. The owner must be parked (the facts land in its own log under the park’s conditional fence via advanceInbox), bounded to existing sequences, reason <= 256, idempotent.

Interactions

• Races and fencing across nodes. The facts are one atomic batch under the park’s conditional fence (ADR 0118), ignorable, so the park stays unanswered. A notifier that built its batch before the quarantine landed is refused by an extended staleInboxBatch and refolds: it refuses an entry at or below its topic’s cursor (as today) AND any entry for a topic that is held, except the first released entry of its queue. Either the delivery wins (the quarantine then sees the cursor past the head: invalid, nothing held) or the quarantine wins (the delivery refolds and skips T). Exactly one outcome, on every node. The fold itself is not made to refuse (old logs replay as they always did); this is the runtime guard of ADR 0110 D8, extended.
• select across topics: above. Replay: replay serves recorded answers and reads neither the inbox nor the quarantine stream; a released message’s answer is its MESSAGE_CONSUMED like any other (sealed in the log as any delivered payload: D4’s replay refusal applies on an incapable node).
• discardInbox: unchanged. On a held topic it may skip messages ABOVE the cursor (they sit behind the hold, order among survivors is kept); it never touches held entries (through at or below the cursor appends nothing). Dropping a held entry is releaseInbox with "discard".
• TTL expiry (D10a): applies to the messages behind the hold exactly as before (it scans above the cursor); it never expires a held entry, which is an operator-owned record bounded by quarantineMax, not a queue. A hold is therefore the one durable state that outlives messageTtlMs by design.
• Per-topic streams (D10/D10a/D10b): the cursor value is in either range; the hold is keyed by topic name, independent of the layout. The topic’s registry entry stays live while it holds backlog; a held topic is not retired as “drained” because its cursor reaching the stream head is no longer the criterion once entries are held (retire requires an empty _quarantined[topic]).
• Erasure and shred: the copy is sealed (or byte-preserved sealed) under the chain’s owner subject, the same key as the original, so a key shred destroys the original, the copy and the log together; the compliance purge deletes the quarantine stream with the chain’s other streams. A plaintext held message cannot exist (not-sealed).
• Continue-as-new: _quarantined is carried like _inbox (continuedFrom.carried.quarantined, validated seed, invalid parts ignored, at most one released entry per topic; the chain id, and so the quarantine stream, survives rotation). Fork / re-open: a fork starts a fresh chain; it copies the quarantine stream prefix its copied log refers to (same sequence numbers) into its own chain’s stream, so the folded heldAt stays valid (the implementer verifies this with the fork tests; the fallback if copyPrefix cannot, is a typed refusal to fork a chain that holds entries, never a fork with dangling holds).
• Observability (R-c Option 2 by-product): the sealed-refusal operator report (isSealedUnreadable) carries the blocked topic and the sealed message’s inboxSeq/age so the operator knows what to quarantine; a held topic is listed on the analytics surface by _quarantined (no new events). A topic that stays held with no capable node reports “held, awaiting a capable node” on the same first-then-every-16th schedule.

Failure modes

Quarantine on a running owner: not-parked, retry. A sealed head that became openable meanwhile: not-sealed, nothing appended. Crash after the copy, before the fact: orphan copy, retry reuses it. Crash after INBOX_QUARANTINED: the hold is durable, nothing else pending. Crash after a MANUAL INBOX_RELEASED, before the delivery: the released entry is part of folded state, so ANY node’s notify path (the wake row, recoverAbandoned, the post-park read) delivers it exactly once (the fact is the intent; delivery is the ordinary fenced MESSAGE_CONSUMED). An AUTOMATIC release is one atomic batch with its delivery, so a crash either appends both or neither: before the batch nothing changed and the next notify retries; after it the work is done; the copy being read but not yet delivered leaves no trace. Two capable nodes racing, or an auto-release racing a manual release/discard, are settled by the fence and staleInboxBatch (above).

Implementation notes (slice S5, 2026-10-14)

What shipped, where the text above left a choice open:

• Names. WorkflowRuntimeDeps.inboxQuarantineMax / inboxAutoRelease (defineConfig kernel.inbox.quarantineMax / kernel.inbox.autoRelease; the ceiling 1024 is validated at composition). The sealed-by-a-held-topic refusal is QuarantineHeldError extends SealedPayloadUnreadableError (so the existing operator machinery - wake row kept, sealedBackoff, first-then-every-16th - applies unchanged); it says “awaiting a capable node” or, with auto-release off on a capable node, “awaiting an operator release”.
• One delivery rule, three callers. inboxDeliverable (inbox.ts, pure, from folded state) is applied by the notify scan (heldDelivery), the live run (WorkflowContext.deliverHeld, which commits the auto-release as commitAtomic([INBOX_RELEASED, MESSAGE_CONSUMED])) and the guard (staleInboxBatch). A held topic is never read past its cursor; a held topic this node will not lift is skipped, the other topics are scanned, and only when nothing else is deliverable does the pass throw QuarantineHeldError (after journaling the duplicates it passed).
• staleInboxBatch extension, exactly (defence in depth: the cursor clause alone already catches every batch built before a hold, and a redundant auto fact is ignored by the fold; the extended clauses shape the verbs’ and the notifier’s ANSWER - stale / changed: false / invalid - instead of appending facts the fold would ignore; no fleet scenario isolates the held-topic MESSAGE_CONSUMED clause, every pre-hold batch being stale by the cursor clause too, so it is pinned by the unit tests). A MESSAGE_CONSUMED for a topic with a quarantine queue is stale unless it is the first released entry (matched by key); a reserved discard fact (__loom_discard__, a message ABOVE the cursor: TTL expiry, discardInbox) stays allowed on a held topic; an auto/manual INBOX_RELEASED is stale unless it names the topic’s first held key; an INBOX_QUARANTINED is stale once its message is at or below the cursor, or when the fold would ignore it for the 1024 cap. The verbs journal through advanceInbox(..., guard = true), which applies the same check to the state the batch will land on (answer stale), so two quarantiners of one head yield exactly one fact.
• The copy. Read from the stored envelope at inboxSeq (legacy or the topic’s own stream), appended verbatim with id quarantine:<chainId>:<key>:<inboxSeq> (appendQuarantineCopy reuses an existing copy by id; the DESIGN text above says quarantine:<key>, but a SQL store keeps envelope ids unique across ALL streams and the same sender/message id recurs in other chains, so the chain is part of the id, and the ORIGINAL inboxSeq is too, so a stale copy of an earlier message that shared the key - a retry, a key the dedupe window evicted - is never reused). The copy’s metadata.quarantine = { topic, inboxSeq } is its stamp: readQuarantineCopy accepts a copy only when key AND stamped inboxSeq match the entry (a fork re-ids copies, so the match is on the stamp, not the id). Sequence 0 of the quarantine stream is a payload-free ANCHOR ({ anchor: true }), never a message: a store keeps sequence 0 through every prefix truncation, so a message copied there could never be freed; the first copy is at heldAt 1. A head whose payload reads as the { $shredded: true } marker (any depth) is SealedPayloadUnreadableError, never not-sealed, and nothing is copied. EncryptingEventStore stores a copy whose marked values are already cipher envelopes exactly as given (no second seal, metadata.seal kept); the stream maps to the chain’s owner subject (inboxChainOf).
• Release resolution. releaseInbox matches messageId against the OLDEST entry’s copy (the raw id) or its 64-hex dedupe key (the escape hatch when the copy is gone); the optional sender (additive on ReleaseInboxInput) pins the dedupe key when several senders reused one id; a younger entry is invalid. A repeat is judged from the owner’s log by a BOUNDED backward search from the tail (findReleaseFactBack: 64-event chunks, 4096-event budget, never a read from sequence 0, stopping at the chunk that holds the newest match) that continues through the continued predecessors (a release journaled before a continue-as-new): INBOX_RELEASED for that message, same disposition => changed: false, the other disposition => invalid; facts for more than one key within the matching chunk, with no sender given, are invalid (ambiguous), never an answer for the wrong sender. A race lost to an auto-release/other release is re-judged from the facts that landed since the caller’s fold. The idempotent repeat is guaranteed only WITHIN the budget: a repeat whose release fact is more than 4096 events behind the tail is answered invalid (“not held”), not changed: false; the release itself happened, so a caller retrying a release must treat that invalid as already done.
• At most once, also across keys held and delivered. A key can enter the dedupe window while an entry for it is held (a same-key message on another topic delivered normally; a release that raced one). (a) At every held delivery point (the notify scan’s heldDelivery, WorkflowContext.deliverHeld) an entry whose key is in the window is DROPPED by a MESSAGE_CONSUMED with payload: null (preceded, for a still-held entry, by the auto INBOX_RELEASED in the same atomic batch; the fold already drops the released entry on the key match and delivers nothing for a duplicate), needing no ability to open the copy. (b) quarantineInbox refuses (invalid) a head whose key is already HELD (any topic): it can neither be held twice (a second copy whose release would deliver the message again) nor skipped as a duplicate (a duplicate fact would put the key in the window and turn the held delivery itself into one); the retry is an ordinary duplicate once the held one is delivered. (c) A copy is matched by original position as well as key (above).
• Retention. Only a POSITIVELY disposed copy is freed (trimQuarantine): no live _quarantined entry references it AND its stamped original inboxSeq is at or below its topic’s cursor (it was released and consumed, discarded, or lost the race to a plain delivery, so no fact can ever name it). Every other copy is PINNED: live, or IN FLIGHT (appended by a quarantiner whose INBOX_QUARANTINED has not landed; its original is still above the cursor). The prefix below the oldest pinned copy is truncated; the caller’s fold may be stale, which only pins more (a fact landing after the fold moves the cursor onto an original this fold still reads as above it), so no extra fold variable is needed (the earlier rule, “everything below the oldest live heldAt”, freed an in-flight copy and stranded its message). When nothing is pinned every observed copy is freed, the anchor aside, so no message bytes remain at rest after the last disposal (a copy that lands after the read has a higher sequence and survives). The trim runs wherever the inbox trim runs (every park and every hold/release verb) for a chain whose held copies this node wrote or read (a node that restarted after the last disposal keeps the freed-by-nobody copies until the next hold or release on the chain, or the chain’s terminal cleanup). quarantineMax bounds head - oldest pinned heldAt + 1, orphans included. The fold’s cap (1024 entries) is modelled by staleInboxBatch (an INBOX_QUARANTINED the fold would drop is stale), so a verb never reports a hold the fold ignored; the loser of a race for the last slot gets the typed invalid refusal naming the cap. A held topic is not retired as drained (per-topic layout). deleteInbox, the compliance purge (EraseCoordinator), reclaimInbox and delete-with-chain remove the stream.
• Continue-as-new and fork. carriedOf carries _quarantined; a fork first VERIFIES every referenced heldAt against the parent’s quarantine stream (present, same key, same stamped inboxSeq) and then copies the stream through the highest one into the fork’s own chain with copyPrefix (verbatim ciphertext: as for the rest of a fork’s prefix, the fork must share the parent’s erasure subject to open it); it is refused with InboxQuarantineRefusedError (verb fork) BEFORE anything is written when the store has no copyPrefix or a hold does not resolve, and the child’s quarantine stream is removed again if a later step of the fork fails.

Implementation notes (slice S5b, 2026-10-14): the verbs fold the owner from the LOG when the checkpoint is sealed

User ruling 2026-10-14. With sealed checkpoints the norm (D8), a node that cannot open the owner’s checkpoint (the very node quarantineInbox is for) shares the snapshot table and used to be refused by the verbs’ first fold as an unknown owner. Now:

• Where. ONLY quarantineInbox / releaseInbox (their owner fold, the chain-head fold in sendTarget they call, the idempotence search and the post-hold fold). foldState(id, verb) takes the verb only from those callers. Replay, resume, loadState, send, notifyInbox, fork keep refusing per D7: the node still cannot serve a sealed owner anywhere else.
• What triggers the fallback. SealedStreamUnreadableError from the snapshot load, i.e. both a snapshot whose state is itself a ciphertext envelope (marker-unopenable) and a KeyUnavailableError translated by SealGuardedSnapshotStore (key-unavailable). Anything else rethrows unchanged, in particular the erasure errors (PayloadShreddedError). The erasure rule, precisely: a decorated node raises its own PayloadShreddedError from the strict read; on a node WITHOUT the crypto decorator, and on the key-less raw read below, no store can see the shred, so the one place a state is folded past sealed payloads (transientQuarantineFold) checks the folded variables._shredded (a PAYLOAD_SHREDDED fact in the log, or compacted into a plaintext checkpoint) and raises the same-named erasure error. A sealed checkpoint’s key-unavailable load already runs the erasure probe first.
• Completeness signal. There is no compaction marker; truncatePrefix[Keeping] always deletes sequence 1, so a compacted log has a gap after sequence 0, and a log compacted to the tip is [0] with the checkpoint ABOVE it. The fold therefore requires (1) every event i has sequence === i and (2) the log’s tip reaches the checkpoint’s sequence, read from SnapshotStore.loadLatestSealed (the at-rest form: its sequence is plaintext; no key, no plaintext), else from the refusal detail ONLY when the refusal came from the checkpoint itself (SealedDetail.checkpoint; a key-unavailable load’s erasure probe reads the LOG, and its refusal names a LOG event’s sequence), else (sequence unknowable) a tip of at least 1. A store without loadLatestSealed whose refusal carries no sequence cannot distinguish a fresh [0] log, so it is refused too. Otherwise InboxQuarantineRefusedError reason checkpoint-unreadable-log-compacted (not retryable) names the fix: run the verb on a capable node. Nothing is appended.
• Authz is unchanged and runs first. operatorInboxHead authorizes against the plaintext recorded owner (sequence 0) before any fold; a sealed principal is still the D7 denial, a stranger still gets ExecutionDeniedError, and the typed refusal above is only ever shown to a caller already authorized as quarantine / release / discard on that execution (no oracle). One ordering note: the chain walk (sendTarget following EXECUTION_CONTINUED, findReleaseFactBack stepping to predecessors) folds a successor or predecessor before that hop is authorized on its own; this is not an oracle because a hop is followed only after its continuedFrom / successor link is verified, and a verified link shares the owner of the head the caller was already authorized against.
• Which fields the verbs need. The cursors, dedupe window, _quarantined and park state come from kernel facts (MESSAGE_CONSUMED, INBOX_QUARANTINED, INBOX_RELEASED, the suspend) whose routing fields are plaintext; only payload is sealed. The fold never opens a payload.
• Cost. One full-log read per fold (a verb folds up to three times), O(log length), bounded by nothing but the log; findReleaseFactBack adds one more fold PER predecessor hop it takes (a continued chain: each hop is a full-log fold of that link, up to maxChainHops); operator verbs are rare, and a log too long to fold here belongs on a capable node (a checkpoint is unreadable there anyway).
• Residual closed (user ruling 2026-10-14: read past sealed payloads). The verbs’ reads of the owner’s log (the fold in foldFromCompleteLog, the tip classification, the park walk, the release-fact search) go through a read-only view, SealGuardedEventStore.quarantineReadView(), reached only via quarantineReadsFor(verb). The exact covered-field rule: a marked envelope is stepped over, as an OPAQUE ciphertext value (never decrypted, never tried, no key asked), iff its event type is MESSAGE_CONSUMED, its covers is non-empty and every entry is MESSAGE_CONSUMED.payload or MESSAGE_CONSUMED.payload.<path>, and none of the fields the fold interprets (waitId, topic, messageId, inboxSeq) is itself a ciphertext envelope. Why: the MESSAGE_CONSUMED handler copies payload into _waitAnswers and reads only those four routing fields; INBOX_QUARANTINED / INBOX_RELEASED and the park state come from plaintext kernel facts. Every other covered path (another event type, any other path such as STEP_COMPLETED.output, empty or malformed covers) is still the ordinary D7 refusal. A marker the node CAN open passes as before. Transience. The state so folded is branded (markTransientFold, a module-private WeakSet) and frozen; SealGuardedSnapshotStore.save and restoreFromState throw on it, the verbs return only ids and counts (never the state), and it is never cached. Unreachable elsewhere: loadState, replay, resume, send, notifyInbox, fork and the general foldState keep the strict guard; sealGuardFitness.test.ts pins that quarantineReadView is named only in sealGuard.ts and quarantineReadsFor, that quarantineReadsFor is called only from the five verb read methods, and that the snapshot and restore guards exist. The sealed-principal case is unchanged (authz reads sequence 0 through the strict guard first). A KeyUnavailable on the snapshot is still the trigger; erasure errors rethrow.
  • Every owner read path (round 3). With logFoldFor set, the no-snapshot read and the tail read after a PLAINTEXT snapshot go through the same view (not only the whole-log fallback), so no snapshot + sealed tail and plaintext snapshot + sealed tail no longer refuse. A state is branded transient only when the view actually left a sealed value in place.
  • No key is asked for a stepped-over event. The EventStore SPI 2.5.0 (unreleased, no new bump) gains the OPTIONAL readAtRest(): AtRestEventReader (read, readFrom, readRange?, head?; envelopes exactly as stored, never decrypted; no write, fence or subscribe), listed in OPTIONAL_EVENT_STORE_METHODS. EncryptingEventStore implements it with its inner store’s raw reads (or the inner’s own readAtRest); every decorator forwards it per its DecoratorPlan; ShardedEventStore routes each call to the stream’s shard like read (offered only when every shard has it). The view reads through it first and keeps the result only when EVERY marked envelope is one the rule steps over (and no unmarked envelope carries a top-level ciphertext field, the pre-marker legacy case; nested sealed paths of such an unmarked row pass through as ciphertext and the verb then decides on the ciphertext, which is a refusal or a step-over, never a leak); otherwise it falls back to the ordinary decrypting read under the opaque check. So a crypto-decorated node WITHOUT the key (the typical incapable node) never consults its KeyStore for an admitted event. A store without the method (or a decorator that drops it, which the decorator kit fails) falls back to the decrypting read: correct, but the key is asked and a key-less node refuses. Conformance: a store that implements it returns envelopes byte-identical to what was appended when it seals nothing.
  • At-rest means as stored, including unresolved content refs (round 5). The at-rest view sits beneath EVERY decorator, so a ContentResolvingEventStore above it never resolves a ref: an admitted transient fold sees ContentRefs, and erased content does not raise ContentMissingError on that path. Harmless for the routing fields the verbs read (kernel facts, plaintext) unless an app externalizes a routing field. Also, SealGuardedEventStore REFUSES readAtRest (decorator plan refuse), so runtime.events.readAtRest is undefined and the view cannot be used to read past D7; only the quarantine view reads it, from its own inner store (sealGuardFitness.test.ts pins that kernel readAtRest calls appear only in sealGuard.ts).
  • Narrow read view. quarantineReadView() returns QuarantineEventReader (read, readFrom, readRange?, head?): no append, truncate, subscribe or fence.
  • Known ciphers only. opaqueToQuarantineFold admits only a marker whose cipher is in an allowlist (loom/aes-256-gcm, a plain object with a string __loomEnc): the marker is metadata, and an unknown cipher’s value shape is unknown, so it keeps the ordinary refusal.
  • Transient hardening. markTransientFold returns a NEW object whose variables are a private structured clone frozen deep (never freezing a structure shared with a snapshot store; a value structuredClone cannot copy is a typed refusal, not a DataCloneError); a spread copy loses the brand, so SealGuardedSnapshotStore.save also refuses any state whose _waitAnswers (only an envelope whose __loomEnc is the KNOWN kernel cipher id loom/aes-256-gcm; user data merely shaped like one saves. Residual: an app value that itself carries the kernel cipher id in a wait answer is refused too) holds a ciphertext envelope (bounded walk), independent of the brand and of whether the snapshot store seals.
  • Pinned callers. sealGuardFitness.test.ts pins, per verb-accepting method (operatorInboxHead, openOwnerState, afterHoldChange, findReleaseFact[Back], sendTarget, classifyTip, locatePark, foldState, foldFromCompleteLog), the exact methods allowed to pass a verb to it, and proves a planted new caller is reported.
• Test fixtures. A harness no longer needs to give the incapable node its own empty snapshot store (the S3 workaround); sharing the sealed store is the realistic setup, covered by quarantine.logFold.test.ts.

D10 (R-d) — A replay refusal leaves a wake row; the poll re-drives a running execution that carries one

Implemented (S6). redriveAbandoned(runtime, executionId, { source }) lives in redriveAbandoned.ts and is shared by the sweep and the poll, with ONE per-runtime sealed back-off judged on the runtime clock (so a refusal seen by either suppresses the other). A replay refusal writes a REFUSAL row stamped with the log head in the new optional TimerRecord.refused: { head, holds? } field (an ordinary wakeAt of now; an additive, nullable column in the SQL stores, TimerStore is unversioned); every other inbox row on a running execution (a contended notify pass, an ambiguous-tail send) keeps the ordinary rules and is retired. The poll (TimerService.fireRefused) re-drives such a row only when this node is not driving or starting the execution (drivesLocally), the log head still equals the stamp, there is the SWEEP’S evidence (the isAbandoned option, or with leases an existing, expired lease: an absent lease is none, because no normal driver holds one), the fence is taken conditionally on that head (ifHeadAt: observation + acquireFenceIf, one winner), and the attempt cap and maxRecoveries have room. With neither isAbandoned nor a lease store configured, the node cannot prove abandonment: it asks the store for no refusal rows at all (due({ refused: false })), and recovery comes from a configured recoverAbandoned sweep. A row a node may not act on yet is HELD by rescheduling it in the store to wakeAt + back-off(holds + 1) (same head, holds + 1): a DETERMINISTIC instant with no node-local jitter, so N pollers that read the same row all upsert the SAME successor and rows never multiply per generation. A held row is not due again until then, costs no read in the meantime, and cannot starve a real timer. A stamp that lands on a real timer’s own (executionId, wakeAt) key is routed to the ordinary timer path BEFORE any hold branch (a hold completes its key), and holdRefusal never completes a key a real timer owns. A same-key upsert keeps the MAX head and MAX holds in every store (never last-writer-wins); a partial index covers refused IS NOT NULL. WorkflowRuntime.lastRefusalAt (the strictly-increasing refusal instant) is per process and resets on restart: harmless unless the wall clock goes backwards across the restart. isAbandoned on the poll carries the SWEEP’s contract: it must be a sweep-grade predicate. A bare time heuristic can double-enter an execution that is alive in a long silent step; the residual is closed by the S7 heartbeat leases. Mixed fleet: a node that predates refused reads the row as a plain wake it can never match to a park and retires it as obsolete (it never re-drives it); the recovery then comes from a sweep or a node that knows the field. On an UNMARKED legacy stream the incapable node cannot know in advance: each backed-off re-drive appends only the re-drive’s own EXECUTION_RECOVERED marker, bounded by the cap; a MARKED stream is refused by the read before any fence acquire and writes nothing. Residual: a store without appendIfUnchanged/acquireFenceIf falls back to a head re-check plus the unconditional acquire. The wake row itself is a timer-store row, never an event-log append.

Write. In WorkflowRuntime.run()’s catch for the replay refusal (the WorkflowContext latch, handled today by warning and rejecting the handles), call markContended(executionId) (same best-effort .catch(onStorageError)), now for BOTH SealedPayloadUnreadableError (replay) and the new SealedStreamUnreadableError. It appends no event (D4 holds). No timer store: no-op, documented (the recoverAbandoned contract is then the only recovery).

Poll. Today TimerService.fireInbox runs the notify path, which finds no unanswered park for a running execution and retires the row. New rule in fireInbox, before the notify path: if state.status === "running" and the row is the inbox wake (INBOX_WAKE_AT), do NOT retire; classify:

1. Another node is live on it. A lease/liveness probe says a live holder exists (the same LeaseStore the poll already claims under, ADR 0118 D4): leave the row, short back-off (the existing wakeBackoff, with its attempt cap so a row on a genuinely long-running execution does not live forever without bound reporting). The poll never supersedes a live holder.
2. Abandoned or unowned. Call the SAME routine recoverAbandoned uses for one execution (extracted as WorkflowRuntime.redriveAbandoned(executionId, { source: "wake" })), so fencing, the diverged-run policy and the failed/recovered accounting are shared, not re-implemented. On success the execution moves on (suspends, completes, or keeps running under the new driver) and the row is retired when the state is no longer running-and-stuck (the next poll sees a parked or terminal state and the ordinary rules apply; a terminal execution retires the row).

Fencing; capable vs incapable do not fight. The re-drive acquires the execution fence like any resume. To keep an incapable node from taking the fence just to refuse: (i) D7’s assertStreamReadable runs BEFORE the fence acquire, from a read of the log, so a marked stream is refused with no write at all; (ii) for an UNMARKED sealed stream (the legacy case) the incapable node cannot know, so it may acquire and then refuse at replay; the acquire only bumps the fence token (a store row, not a log event), the refusal appends nothing, and the incapable node’s per-execution sealedBackoff (the ADR 0130 map, shared, exponential, no attempt cap, bounded 1024) keeps it from retrying soon, while the row stays for a capable node. A capable node that acquires the fence after it wins, replays, passes, and proceeds; the incapable node’s stale token is fenced out on its next write attempt (it has none to make). Two incapable nodes polling the same row serialize on the same lease claim. The fence bump is the only cross-node effect of a refusal and is already tolerated by every resume path (ADR 0118).

Back-off and reporting. Refusals reuse sealedBackoff and the first-then-every-16th onError. A re-drive that fails for another reason uses the existing wakeBackoff with its attempt cap and RedriveExhaustedError. The row is never retired by an incapable node.

Interaction with recoverAbandoned. The sweep and the poll converge on the same routine and the same backoff maps, so a refusal in one suppresses the other for the window (the sweep already skips a refused execution on this node, ADR 0130 D4). The sweep stays the safety net where no timer store exists; the wake row makes recovery timely without requiring an app to run a sweep with an isAbandoned predicate.

Residual risks after this design

• A stream sealed by a writer that does not stamp metadata.seal keeps the pre-0130 non-inbox exposure (D7, unmarked).
• Pre-1.25.0 nodes in a mixed rollout have no marker check.
• A quarantine holds its TOPIC for every node that cannot open the held message until a capable node auto-releases it or an operator releases/discards it (D9): later messages of that topic wait behind it, by design, because that is the only way a release can keep original order. If no capable node ever runs, the topic stays held indefinitely (the other topics flow; the exit is releaseInbox("discard")).
• A plaintext snapshot already at rest is not rewritten.
• LeaseKeeper (S7b) store-side clock residual: the holder self-fences on its own clocks (monotonic AND wall, valid only while both are inside ttl - ttl/6; the margin is fixed at ttl/6, Q3(a)), but the lease EXPIRY is decided by the store’s clock (Postgres clock_timestamp(), SQLite Date.now()), a wall clock that can step. An NTP step FORWARD of more than the margin on the store host expires live leases early and the holder cannot observe it; the fence remains the only write guard.

D11 — Heartbeat leases (Accepted 2026-10-14, user)

• Status: Accepted 2026-10-14 (user): option (a), the recommendation, for every decision question below. Slices S7a-S7e are implemented (see 9 and 10). Requested by the user after the S6 (D10) review. Slices: .superpowers/plans/2026-10-14-adr-0130-residuals.md ”## S7” (local, not committed).
• Placement: kernel (the runtime is the only thing that knows when a driver starts and stops driving) plus 1P (LeaseStore backends in plugins/postgres and plugins/sqlite). No 3P part. ADR 0083 (second-consumer rule) is satisfied: the consumers are TimerService.fire, recoverAbandoned, the D10 poll, orphanCensus and “only one node should run this execution”. ADR 0004’s 2026-09-14 amendment deferred exactly this (“a LeaseStore/heartbeat SPI … waits for a second, unrelated consumer”); the LeaseStore SPI and renewWhileHeld shipped later (#380, L5a), and this ADR is the “lease every run” step that amendment and orphanCensus (#408) left open.

Problem

The kernel has no liveness signal for a normally-driven execution. A running execution driven by a healthy node and one orphaned by a dead node are byte-identical in the log (ADR 0004 amendment). Today only two paths ever hold a lease: TimerService.fire (a wake) and recoverAbandoned’s leases mode (a re-drive). Everything else, start, resume, the send/notify winner, runAgentOnce, a worker-runtime dispatch, drives with no lease. So:

• “A lease exists and has expired” is evidence of an abandoned driver, but only for the two paths above. “No lease” is not evidence of anything, and “I acquired a lease” is not evidence either (it only proves nobody else held one).
• The D10 poll (S6) may therefore re-drive a running execution only with a host isAbandoned predicate or an existing expired lease.
• The residual: a live but silent driver (mid a long step, nothing appended since a sealed refusal) plus a time-heuristic isAbandoned can be double-entered. With fencing the log stays correct, but the effect is paid twice.

Goal. Every driver holds a heartbeat lease for the whole time it drives. Then “a lease existed and expired” is real evidence everywhere, an unexpired lease is a veto on any heuristic, and the poll can re-drive by default with no host predicate.

Invariant to state precisely. Lease = liveness evidence only (it can be wrong in one direction, see 3). Fence = the write guard (it is never wrong about who may append). A re-driver needs both. Neither is a substitute for the other.

1. Who holds a lease, and when

Rule (recommended). Every attempt to drive an execution holds a lease from before its first append until its park or terminal commit. “Driver” means each of: start (including caller-chosen and idempotent starts), resume, the send/notify winner (the node that won acquireFenceIf), runAgentOnce, the recoverAbandoned and D10-poll re-drives (which already claim one), TimerService.fire, and a worker-runtime/dispatcher attempt (the worker process is the driver, the lease owner is the worker node, not the dispatcher).

• Lease before fence. The lease is the cheap liveness claim; the fence is the write guard. Order: acquire the lease, then (for a resume) win the fence, then append. A null lease (someone else holds an unexpired one) means a live driver exists: stand down, drive nothing, acquire no fence. A winner of acquireFenceIf holds a proof the execution is parked, so a lease left behind by a slow release must not block it: the acquire for that caller may steal (see S7a, an optional steal flag valid only when the caller holds a conditional-fence proof).
• Lease before the first append also fixes crash-before-bootstrap (ADR 0103): a stream that exists was created by a driver that held a lease, so “stream exists, status running, a lease row once existed and is expired” is real evidence. A lease with no stream (crash between acquire and the first append) is a ghost; the poll GCs it (deletes the expired row) without re-driving.
• Owner identity. owner = <nodeId>:<incarnation uuid> (a fresh uuid per process start), NOT a bare node name. acquire treats the same owner as a renewal; a restarted pod reusing its name would otherwise silently inherit its dead predecessor’s leases and mask the very crash this detects.
• Release. At the suspend barrier commit (status becomes suspended) and at every terminal commit the driver awaits a (group-committed, ms-window) release before resolving the handle. A failed release is not an error: the lease expires, and an expired lease on a non-running execution is ignored by the poll. Release is awaited, not fire-and-forget, so a prompt send winner is not blocked by our own stale lease (the steal flag is the backstop, not the plan).
• Continue-as-new. The successor is a new execution id. Rule: acquire the successor’s lease before appending its EXECUTION_STARTED, while the predecessor still holds its own; the predecessor releases after its CONTINUED_AS_NEW commit. No instant exists at which a stream is running and unleased. awaitLiveHandle(successor) keeps its current registration order (liveHandles before successor seq 0).
• Sweep and poll re-drives hold the lease across the resumed run exactly as recoverAbandoned does today (renewWhileHeld), now through the shared keeper (5).

Options.

1. Every driver, eager acquire (recommended). Simple evidence: any stream whose lease row is expired and whose driver is not us is abandoned. Cost in 5.
2. Lazy: acquire only once a run exceeds a grace period (e.g. TTL/3). Cuts lease writes for short executions to zero. The cost is the evidence: a driver that dies inside the grace window has no lease, and “no lease” is again ambiguous, so such a crash needs isAbandoned. Could be recovered with “no lease AND head older than grace + margin” but that compares an event timestamp (node clock) with a store clock, and needs a fleet-wide “everyone leases” declaration to be sound. Rejected as default, offered as an optimisation if the bench (S7f) shows eager is too costly. (Amended 2026-10-15: the S7f bench showed exactly that, so this option is now the default; see “Amendment 2026-10-15 (S7g)” at the end of D11.)
3. Only long-running flows (a per-flow opt-in). Leaves most drivers unleased, so the poll still cannot be safe by default. Rejected as the goal, acceptable as a stop-gap.

2. Lease store default

Today timers.leases.store is explicit config; defineConfig wires nothing by default, and stores.leaseStore is present only when configured. Three options:

1. Co-located with the event store, derived by default (recommended). The durable backends already create a leases table in the same database (PostgresLeaseStore runs migrate on the shared schema; SqliteLeaseStore is single-host). defineConfig derives the lease store from the same factory/pool as the event store when it offers one, so a multi-node Postgres deployment gets cluster-visible leases with no extra config, and the lease and the log share one failure domain (if you can reach the log you can reach the leases; no third dependency). The accessor lives on the plugin factory (postgresStores({...}) -> { events, snapshots, leases, timers }), NOT on the EventStore interface, so there is no EVENT_STORE_SPI bump. An explicit durability.heartbeat.store overrides.
2. In-memory default. Leases are visible to the one process only. Across nodes the evidence is meaningless and dangerous: node B sees no lease for A’s execution and (wrongly) reads “no lease” as nothing, or worse a future rule reads it as abandoned. An in-memory lease store is correct only when the event store is also in-memory/single-process (then there is no other node to disagree). Recommended only as the default paired with an in-memory event store.
3. Required config, fail-closed. A durable event store with no derivable lease store: leases are disabled, the poll re-drive reverts to today’s rule (host isAbandoned or an existing expired lease), and loom doctor emits a finding “durable event store without cluster-visible leases: heartbeat disabled”. Never silently fall back to in-memory against a durable shared store.

Recommended default. (1) when the event store’s factory exposes leases; (2) only with an in-memory event store; otherwise (3). defineConfig records which it chose in stores/the doctor surface so the diagnostic can say “leases: co-located postgres | in-memory (single process) | disabled (no store)”. The existing stores.leaseStore absence-vs-empty rule (a diagnostic must not render “disabled” as “all expired”) stays.

3. TTL, renewal, stalls, clock skew

• Renewal runs on a timer independent of step progress (today’s renewWhileHeld already does: setInterval, unref, not tied to appends). A 20-minute step never lapses. Heavy synchronous CPU can starve the event loop; the existing advice stands (sandbox worker pool, or a larger TTL).
• Cadence vs TTL. Default TTL 30 s, renew every TTL/3 = 10 s (defaultRenewEvery), so two consecutive renewals can fail before expiry. Detection latency of a dead driver is at most TTL plus the poll interval (about 30 s + poll). Both configurable under durability.heartbeat. (Today’s explicit-config default is 60 s; changing the default to 30 s is a decision, see Q3.)
• Clock. Expiry is stamped and judged on the store’s clock (L29: Postgres uses clock_timestamp() DB time; SQLite is single-host), never the node’s. Node skew cannot make a live lease look expired. The driver’s own stop deadline uses a local monotonic clock measured from just BEFORE the successful renew request was sent (conservative by the round trip), so only rate drift, not offset, matters, covered by a margin of TTL/6.
• A lease that lapses under a live driver (GC pause, event-loop stall, SIGSTOP, VM freeze longer than TTL; or a node-to-store partition) is the central hazard. The evidence is then false in one direction (a live driver looks dead). Three layers, in order of strength:
  1. Self-fencing deadline (the driver stops itself). The keeper tracks deadline = lastRenewSentAt + TTL - margin. A driver that has not had a confirmed renewal by its deadline treats the lease as lost.
  2. Lost-lease signal. A lost lease (renew returns false, or the deadline passes) aborts the run’s AbortController with reason LeaseLostError. This is the same controller that is already threaded into every effect context, so effects that honour AbortSignal are cancelled. It is ephemeral: it appends nothing (unlike cancel(), which commits EXECUTION_CANCELLED; another node now owns the log) and rejects the handles with the existing “superseded, not failed” error class (ADR 0004 amendment L20), so a dispatcher does not count it as a failed attempt.
  3. Gate at dispatch and append. Before dispatching each effect and before each append the pipeline checks the keeper’s local monotonic validity (a clock compare, no I/O). This closes the gap a stall opens: after a stall longer than TTL, the first thing the resumed event loop does is run queued continuations, and the gate refuses them before any new effect starts. The fence still rejects the append of anything that slips through.
• State the residual. (a) An effect already in flight when the stall began, which ignores AbortSignal, completes: the step’s side effect is paid twice (once by the stalled driver, once by the re-driver). That is the same at-least-once ADR 0004 accepts for a genuine crash; the lease turns “any slow driver” into “a driver stalled for more than TTL”. Under clock skew between the store and the node (the store’s clock ahead), the row can read expired before the node’s own deadline: the superseded driver is still entitled, so it can also START effects after the takeover, through every step it reaches before its oldest unacknowledged write is refused. The journal writes behind (docs/architecture.md D3), so that window is about one store round trip, and ADR 0021 A3 rejects a per-effect durable wait on the hot path. The fence still refuses every one of those writes (pinned by the entitlement-window scenarios in tools/dst/src/lease.dst.test.ts). (b) A timer cannot fire during a stall; the gate (3) is what fires first on resume. (c) The log stays correct throughout (the fence), and STALE_FENCE from the stalled driver’s next write is expected and handled as superseded (L20).

4. Lease vs fence: who proves what

• Fence stays the only write guard (ADR 0118): a re-driver that took the fence makes every older driver’s append fail. Unchanged.
• Lease is evidence for the decision to re-drive and a veto against a heuristic. It never authorises a write by itself.
• Re-driver algorithm (poll and sweep share redriveAbandoned):
  1. Evidence: the execution’s lease row is expired on the store clock (expired()), or isAbandoned says so AND no unexpired lease exists (the new veto: an unexpired lease overrides a time-heuristic isAbandoned; this directly closes the residual).
  2. acquire the lease (an atomic takeover of the expired row; null means someone else just took it, stand down).
  3. Read the head and fence (H, F). If the execution is no longer running, release and stop.
  4. Conditional takeover: append EXECUTION_RECOVERED with appendIfUnchanged(id, ev, { head: H, fence: F }) (EVENT_STORE_SPI 2.2.0, R11; it never presents or adopts a token). If the “expired” driver was actually alive and appended since step 3, this fails and the re-driver backs off without ever taking the fence. Only on success does it acquireFence and drive.
  5. A store without appendIfUnchanged falls back to the plain acquireFence path (today’s behaviour).
• Why both: the lease says “nobody has said they are alive for TTL”; the conditional append says “and the log did not move while I decided”. Together they shrink the double-entry window to a driver that is stalled for more than TTL and writes nothing in the observation gap, which the self-fencing layers in 3 then stop at the driver. Clock skew widens this to a driver the store sees as expired while its own deadline has not passed (residual (a) above).

5. Cost

Reference numbers (docs/benchmarks.md): ~5.7K durable executions/s per runtime on one Postgres primary for a 1-step no-op-effect flow, the log being the bottleneck; with real I/O-bound effects the write path is nowhere near binding (sustaining 5.7K/s of 2 s steps needs ~11K concurrent executions).

• Per execution: one lease row insert (acquire) and one delete (release), plus T/renewEvery renewals over its life. Naively 2 statements per execution at 5.7K/s = 11.4K lease statements/s, doubling the write pressure on the very store that is the bottleneck. This is why batching is not optional.
• Batching (recommended): one LeaseKeeper per node. One timer, one set of held ids. Acquire/release are group-committed (collected in a 1-5 ms window or one microtask-drain, then one acquireMany/releaseMany statement); renewal is one renewMany(owner, ids, ttl) per tick for everything the node holds, chunked at ~1000 ids. At 11K concurrent long executions this is about 11K row updates per 10 s tick (about 1.1K row-writes/s in a handful of statements), not 11K statements.
• Postgres: UPDATE leases SET expires_at = <db now> + $ttl WHERE owner = $o AND execution_id = ANY($ids) RETURNING execution_id (the returned set is the held set, the diff is the lost set), INSERT ... ON CONFLICT ... WHERE owner = $o OR expires_at <= now as a multi-row statement for acquireMany, DELETE ... WHERE owner = $o AND execution_id = ANY($ids). The table is small, hot and update-churned: keep fillfactor below 100 for HOT updates, an aggressive per-table autovacuum, and the expires_at index only on the (cheap to maintain) (expires_at) column the poll scans. These are S7a deliverables, not assumptions: S7f measures and gates (proposed: no more than a 10% regression on the existing no-op ceiling bench tools/bench-throughput, and zero regression at I/O-bound effect durations). If eager misses the gate, fall back to the lazy option in 1.
• Not proposed: a per-execution timer, per-execution renew statements, or renewal piggybacked on log appends (that would make a silent long step lapse, the exact case we need).

6. Failure modes

• Lease store down or slow, node still reaches the event store. The node cannot confirm renewals. The same node, however, cannot tell this apart from a partition where other nodes still reach the store and may take over after TTL. The only rule that keeps the evidence true is the self-fencing deadline: a driver that cannot confirm a renewal stops at its deadline (aborts with LeaseLostError, appends nothing; the execution stays running with an expiring lease and is re-driven once the store is back). Blast radius: a store outage longer than TTL aborts the node’s in-flight long steps, at-least-once cost on re-run. Opt-out durability.heartbeat.onStoreError: "continue": drivers keep running through an outage and the log is still protected by the fence, but the lease no longer proves liveness, so the poll’s default re-drive must treat that fleet as “heuristic” (see 7). Recommended default: stop (evidence must be true), because the product claim of D11 is “expired lease means abandoned”.
• Re-drive is fail-closed. If expired()/acquire errors, the poll and sweep do nothing and report via onError (rate-limited like the sealed refusal report). They never fall back to re-driving “because we cannot check”.
• Recovery from an outage: quiet period. After the first successful store call following an error, pollers wait one full TTL before acting on expired rows (drivers renew first; renew never resurrects a lease another node took but does succeed on an expired-but-untaken row, so live drivers re-establish before anyone can usurp).
• Partition node A from the store only. A’s leases lapse; B takes over A’s executions after TTL; A’s deadline fired first (it stops itself at TTL minus margin), so A has stopped before B starts, up to clock rate drift and the unabortable-effect residual in 3.
• Partition node A from the event store only (store reachable). A keeps renewing (so nobody takes over) while its appends fail: the existing StorageWriteError path fails the run. A lease held by a driver that can no longer write is released on that failure. Documented, not new.
• Crash mid-renew / mid-release. Renew is idempotent. A crash mid-release leaves an unreleased lease that expires; for a non-running execution it is ignored, for a crashed running one it is exactly the evidence.
• Keeper itself dies / is starved (the renew timer cannot run): identical to a stall; layers 1-3 of section 3 apply on resume.

7. Migration and compatibility

• Executions with no lease (legacy streams, a store with no lease SPI, or a node that has heartbeat disabled) keep today’s behaviour byte-for-byte: the poll and sweep re-drive them only with a host isAbandoned, never on “no lease”. Unwired is unchanged (G2).
• Mixed fleet. A node without S7 neither acquires nor renews. Its drivers have no lease, so an S7 node’s poll cannot see them (safe: no re-drive without isAbandoned). The reverse risk: an old node’s recoverAbandoned ignores the lease veto, so a pre-S7 node can still double-enter an S7 driver with a heuristic. That is today’s risk, not a new one, and it ends when the fleet is upgraded. A deployment can declare durability.heartbeat.required: true once every node runs S7; only then does a missing lease for a running execution older than TTL become evidence (a future follow-up, not in S7a-f, because it needs the monotonic-vs-event-time comparison that 1 option 2 rejected).
• Versions. LEASE_STORE_SPI_VERSION 2.1.0 -> 2.2.0, additive and optional: acquireMany, renewMany (returns the held set), releaseMany, and the steal option on acquire; a store without them is driven through the single-id methods by the keeper (correct, just slower). EVENT_SCHEMA: none (no event, no field; ghost leases and the veto are store-side). EVENT_STORE_SPI: none (the lease accessor is on the plugin factory, not on EventStore). SNAPSHOT_STORE_SPI, WIRE_ABI: none. KERNEL_ABI: expected none (no new syscall; a lost lease is an ephemeral abort, not a durable cancel), to be confirmed by S7c. New typed error LeaseLostError (additive export). The ADR 0130 coordinated bump (EVENT_SCHEMA 1.25.0, EVENT_STORE_SPI 2.5.0) is not touched. CHANGELOG: Added (heartbeat, config), Changed (the poll’s default re-drive, the lease veto on isAbandoned).
• Config. defineConfig({ durability: { heartbeat: { ttlMs?, renewEveryMs?, store?, onStoreError?, enabled? } } }), in the existing durability group (flat spellings throw by project rule). Default: on, co-located store (2). enabled: false is the explicit off-switch and reproduces today.

8. Testing

Deterministic tests only (wait on conditions, injected clocks; *.perf.test.ts for budgets; per the AGENTS.md test rules, no wall-clock sleeps).

• Store conformance (runLeaseStoreConformance, new cases): acquireMany/renewMany/releaseMany agree with the single-id methods; renewMany returns exactly the still-held set; steal only with the proof flag; a renew never resurrects a lease another owner took; expiry judged on the store clock under an injected skewed node clock.
• Unit (fake clock, LeaseKeeper): cadence; deadline fires LeaseLostError exactly at lastSent + TTL - margin; the dispatch/append gate refuses after a simulated stall; quiet period after an outage.
• DST (the saga.dst harness, virtual clock, N nodes over one in-memory log + lease store):
  1. Stall longer than TTL: node A’s loop is frozen across TTL; B’s poll takes over; A resumes, its gate refuses new effects, its in-flight abortable effect is aborted, its next append is STALE_FENCE and handled as superseded; exactly one EXECUTION_COMPLETED.
  2. Partition A from the lease store: A stops at its deadline BEFORE B’s takeover (assert ordering on the virtual clock).
  3. Crash mid-renew and mid-release: orphan expires and is re-driven once; a released-then-crashed park is ignored.
  4. Live silent driver must NOT be double-entered: N nodes polling, isAbandoned = an aggressive time heuristic that says “yes”, the driver mid a long step (nothing appended since the refusal). Assert: zero re-drives, effect count 1, across all interleavings the harness explores.
  5. Ghost lease (acquire then crash before first append): GC’d, no re-drive.
  6. Continue-as-new handoff: no instant with a running unleased stream.
  7. Store outage then recovery: quiet period holds, no usurp of a live driver.
  8. Mixed fleet: an unleased legacy driver is never re-driven by an S7 node without isAbandoned.
• Integration (sqlite, pg not needed locally): every driver path (start, resume, send winner, runAgentOnce, worker-runtime attempt) holds and releases a lease; park and terminal release before the handle resolves; D10 poll re-drives only on expired evidence; isAbandoned is vetoed by an unexpired lease.
• Perf (*.perf.test.ts, S7f): lease overhead on tools/bench-throughput against the gate in 5; batching statement count (a count, not a time).
• Enforcement pin (packages/spec): a test that every WorkflowRuntime drive entry point calls the keeper (the vacuous-check pattern: a new driver path that forgets to lease must fail a test).

9. As built in S7c (review round 2026-10-14)

• Bounded store calls. Every keeper acquire/release call is raced against storeTimeoutMs (default min(TTL/6, 1000) ms, within the self-fencing margin). A release that times out settles anyway (the lease expires on its own), is reported through onError (LeaseStoreTimeoutError), and an acquire parked behind it stops waiting; an acquire that times out REJECTS like any store error and arms the quiet period, so a hung store can never wedge a handle or block a start forever. A late answer is ignored (a late grant simply expires; a late release is owner-conditional).
• Ghost leases lose no start. A caller-chosen start (and a continue-as-new successor, and LocalWorkerRuntime.allocate) that finds another owner’s unexpired lease reads sequence 0: present means a real stream (ExecutionAlreadyExistsError, “another node won”); absent means the holder may be a ghost (it leased, then died before its bootstrap), so the start stands down with the RETRYABLE StartLeaseHeldError, which isSupersededRefusal does NOT settle (same rule as StartFencedOutError). TimerService keeps the schedule cursor and retries the occurrence on the next poll.
• Write-gate window. The gate checks the lease immediately before an append reaches the store; the window it cannot close is the append’s own in-flight latency, covered by the TTL/6 margin (the fence is the guard past it). The bootstrap appends that bypass the journal (a caller-chosen start, a successor, a reopen) check lease.valid() themselves immediately before writing.
• Deliberately unleased writes. Cancel-by-id, cancelForward and denial/control records are kill/control writes: they must work against a stuck or lease-less driver and are never gated on a lease.
• One lease store per runtime. When a runtime has a heartbeat keeper, TimerService’s own leases store and owner are IGNORED for claiming (its presence only switches claiming on); the keeper’s store and owner (<nodeId>:<incarnation>) are what other nodes see. A doctor check should read the keeper’s store, not timers.leases.
• Timer wakes. An in-process wake whose run loses its lease before the wake took durable effect keeps its timer row (LeaseLostError), so the next poll judges it; a wake that did land finds its park answered and retires as obsolete.
• Poller fences. A poller/sweep that dispatches takes its (unconditional) fence while it STILL holds the claim and releases the claim only before the dispatch, so a node that leases and begins driving in the gap is never fenced out under its valid lease.
• onStoreError: "continue" was not supported in S7c (rejected at construction); S7d implements it (see 10).
• Holds. DriverLeases counts a pending acquire synchronously (a concurrent last-release defers the keeper release until it settles), remembers a loss for a listener that attaches after it (delivered immediately on attach), and LeaseKeeper.close() reports LeaseLostError("closed") for every id still held, so a closed keeper aborts its runs with the keeper’s real owner.
• Late release (round 3). A release that times out on the wire may still be applied by the store later. The keeper remembers the underlying call per id (lateReleases) and a same-owner acquire for that id (parked or fresh) WAITS for it to settle, at most one more storeTimeoutMs, before the acquire is sent; a store that still has not answered rejects the acquire with LeaseStoreTimeoutError (the quiet-period rules apply). So the late release is always applied BEFORE the re-acquired row exists and validNow can never be true without a row; nothing waits unboundedly.
• Ghost-lease retry is caller-driven and unbounded. The LeaseHeldError/StartLeaseHeldError stand-down for a ghost lease is retried by the caller (the dispatcher or the next poll); nothing bounds the retries. A LIVE holder that never writes sequence 0 keeps renewing, so it blocks that schedule head (a TimerService claim keeps standing down) until the holder dies. Accepted: a renewing holder is by definition alive and responsible for its stream.
• Claim-release to worker-acquire gap. A poller/sweep releases its claim before dispatching to a local worker, which then acquires; another driver can take the lease in that gap, and the local worker then stands down with LeaseHeldError. That is a supersede refusal (nothing appended, the timer row is retired by a later poll), so it is safe.
• runtime.closeHeartbeat() exists for the host; WorkflowRuntime has no close() of its own to wire it into.

10. As built in S7d (consumers: poll/sweep evidence, veto, conditional takeover)

• Evidence rule (one function per consumer, leaseEvidence.ts judgeLease). The poll and the sweep read the runtime keeper’s store (and an explicit leases store when it is a different one) and rank the answer: live (unexpired) > expired > quiet > expired-heuristic > absent. live VETOES everything, isAbandoned included, on BOTH the poll and the sweep (the sweep asks the predicate first, then the veto, so a host’s cheap predicate narrows the store reads). expired is lease evidence only when the host opted in with leases (the S6 sweep contract) or the default holds: this node runs heartbeat in "stop" mode AND store.scope === "cluster" (the S7e clusterVisible). An isAbandoned, when configured, still has to agree. With no lease evidence (absent lease, a "continue" owner’s expired lease, an unknown or non-cluster store) only the host’s isAbandoned re-drives, exactly as before S7; an ABSENT lease with an explicit leases option stays “no evidence at all” (S6).
• Default re-drive. canJudgeRefusals is also true when the default evidence holds, so a node with heartbeat on over a cluster-visible store fetches refusal rows with no isAbandoned and no leases. enabled: false (no keeper) and a non-cluster store reproduce S6.
• Conditional takeover. redriveAbandoned(.., { conditional: true }) (set whenever the runtime has a keeper) calls WorkflowRuntime.claimRecovery: observe (head, fence, anchor) with appendIfUnchanged(id, null), require head to equal the head the re-drive just folded, then append the EXECUTION_RECOVERED envelope with appendIfUnchanged(id, envelope, observation). A moved/gone answer is skipped with NOTHING written and NO token taken. Only on success does it acquireFence (plain: the claim already serialised the contenders) and drive; the resume is told recoveryClaimed and does not write a second marker, so unproductiveRecoveries still counts one marker per re-drive. A store without appendIfUnchanged (or a decorator that drops the anchor) falls back to S7c. A DISPATCHED re-drive is not claimed here (the worker’s own resume writes the marker under its own lease) and keeps lease, then fence, then dispatch. The lease is always taken BEFORE the claim and the fence (the sweep now claims through the keeper in source mode too, not only in lease mode), and no steal is ever used by the consumers: the takeover acquires an EXPIRED row, which needs none.
• Ghost GC. collectGhostLeases(runtime, { limit }) examines at most limit expired leases per pass, skips any with a sequence 0 (an orphan, left to the poll and the sweep), skips any owned by a "continue" node (that starter may still be alive), takes the expired row without the keeper (nothing renews it), re-checks there is still no stream, and releases it only then; if a stream appeared the row is left to EXPIRE as an orphan’s lease (never deleted), so a running stream keeps an expiring lease the default re-drive can judge. The store SPI has no cursor, so orphans would pin the soonest-first window: the scan ROTATES (a pass that collected nothing and filled its window reads one page deeper on the next call, up to 64 pages, then wraps), and a lease whose stream read throws is treated as an orphan instead of aborting the pass. The TimerService runs it at the end of each tick, at most once per TTL on the keeper’s monotonic clock, 100 leases per pass; an error is reported (first, every 16th) and never fails the poll.
• onStoreError: "continue". The keeper does not enforce the self-fencing deadline (validNow and the watchdog stay true); a lease the store reports taken over (a successful renew that omits the id) still ends the hold; a NEW driver whose acquire rejects with a store error runs UNLEASED (absent lease, no evidence) instead of failing. The mode travels in the owner (<nodeId>:<incarnation>~continue), the one field every node reads, so a "stop" poller treats a continue peer’s expired leases as a heuristic without any config about its peers, and a continue-mode poller’s default re-drive is off for the whole fleet. Residual: set it fleet-wide; a "stop" node still re-drives by default on a "stop" peer’s expired leases, which is correct for that peer. The doctor advisory now says the default re-drive is off.
• Fail closed + quiet period. A lease-store read error in the re-drive path throws the typed LeaseEvidenceError: the poll reports it (first, then every 16th), keeps the refusal row and spends none of its attempts (it holds the row), and the sweep counts the execution as failed. The first successful read after an error opens a one-TTL quiet period (on the keeper’s monotonic clock) during which an expired lease is not acted on. A keeper acquire error in the claim is treated the same way.
• Where the veto lives (review). The veto is only as strong as the JUDGING node’s keeper: a node with heartbeat off (no keeper), or a sweep given isAbandoned on a keeper-less node, reads no lease and can still double-enter a live leased driver; run heartbeat on every node that polls or sweeps. The fence still guards the log (the loser is refused at its next append), so the cost is a duplicated in-flight effect, not a corrupted log. Defence in depth on the same node: a heartbeat-enabled sweep never re-drives an execution this node drives (drivesLocally; a keeper-less sweep keeps its power to supersede a hung in-process run), because the keeper answers true for a lease its own owner already holds. A keeper acquire error in the poll is handled like a judge-read error: reported (first, then every 16th), the row held, no attempt spent, the poll continues. A won conditional claim re-stamps the refusal row at the new head (onClaimed), so a later failure retries instead of retiring the row as stale; the retry still needs evidence, so an attempt that WON the claim and then does not succeed (a fence or resume error, a refusal) does not release its lease: LeaseKeeper.forget(id) (surfaced as DriverLease.forget()) stops renewing and drops the id from the held set (validNow false at once, no onLost, no store call) but leaves the row to EXPIRE after the TTL, and that expired row is the default evidence for the retry, with no host isAbandoned. Until it lapses the row is a live lease and vetoes any predicate, as it should. DriverLeases keeps the intent per id: with other holds still live it applies when the LAST goes, a deferred release (a pending acquire) honours it, and a lost lease clears it. Success, park and terminal still release. RunOptions.recoveryClaimed is an opaque per-runtime grant (recoveryClaimOptions()), not a boolean a caller can pass to skip the recovery marker.
• Residuals. (a) The conditional claim closes the window between the head read and the claim, not the decision (isAbandoned or an expired lease) before it: a driver stalled past its TTL that writes nothing meanwhile is still superseded, and its own gate and the fence then stop it. (b) A store with no get cannot tell an absent lease from a live one, so it cannot veto (it can list expired leases); the first-party stores all have get. (c) A dispatched re-drive’s marker is still written by the worker’s resume, after the dispatch.

Decision questions (user) — answered 2026-10-14: (a) for all ten

1. Which drivers lease, and when? (a) Every driver, eager acquire before the first append, awaited release at park/terminal (recommended). (b) Lazy acquire after a grace period (cheaper for short runs, but a crash inside the grace window is unevidenced). (c) Per-flow opt-in only.
2. Lease store default. (a) Co-located with the event store, derived by defineConfig; in-memory only when the event store is in-memory; otherwise disabled with a doctor finding (fail-closed) (recommended). (b) Always in-memory (evidence meaningless across nodes). (c) Required explicit config, no default.
3. TTL and cadence. (a) TTL 30 s, renew every 10 s, self-fence margin TTL/6 (recommended; changes today’s 60 s explicit-config default for heartbeat only). (b) Keep 60 s / 20 s (slower detection, more tolerance of stalls). (c) No default; the operator must choose.
4. Driver that cannot confirm a renewal. (a) Stop at its deadline: abort with LeaseLostError, append nothing, re-driven after the store returns (recommended; keeps “expired lease = abandoned” true; a store outage longer than TTL costs in-flight long steps a re-run). (b) Keep running, rely on the fence (no outage blast radius, but the lease stops being evidence and the poll’s default re-drive must stay off).
5. Lost-lease effect cancellation. (a) Ephemeral abort through the existing run AbortController, plus the dispatch/append validity gate; unabortable in-flight effects are an accepted, documented residual (recommended). (b) Abort only; no gate. (c) Treat a lost lease as a durable cancel (rejected in the analysis: another node owns the log).
6. Re-driver safety. (a) Expired-lease evidence, an unexpired lease vetoes any isAbandoned, and a conditional appendIfUnchanged(head, fence) takeover when the store supports it (recommended). (b) Evidence and veto only, plain acquireFence. (c) Evidence only.
7. Batching and SPI. (a) Per-node keeper with group-committed acquire/release and one renewMany per tick; LEASE_STORE_SPI 2.2.0 optional batch methods and steal; no EVENT_SCHEMA/EVENT_STORE_SPI change (recommended). (b) Single-id SPI only (11K statements/s at the ceiling; not viable). (c) Put the lease on the EventStore interface (an EVENT_STORE_SPI minor, ties lease and log into one SPI).
8. Cost gate. (a) Ship eager if the S7f bench shows no more than a 10% regression on the no-op ceiling and none at I/O-bound durations; otherwise fall back to lazy (1b) (recommended). (b) Ship eager regardless. (c) Ship lazy first.
9. Default re-drive. Should the D10 poll re-drive by default once S7 lands (lease evidence, no host predicate), or stay opt-in behind durability.heartbeat.redrive: true for one release? (a) On by default when heartbeat is enabled and the lease store is cluster-visible (recommended); enabled: false restores S6 behaviour. (b) Opt-in for one release.
10. Mixed-fleet “required” mode (a missing lease becomes evidence once every node runs S7). (a) Out of scope for S7; a follow-up if asked (recommended). (b) Include now.

• S7e implementation note: LeaseStore gains an optional scope (“process” | “host” | “cluster”; absent = unknown, NOT cluster-visible; documented under SPI 2.2.0) and app.heartbeat reports scope and clusterVisible (scope is “cluster”). S7d enables the default poll re-drive ONLY when clusterVisible; a host or unknown store keeps the host isAbandoned requirement, and loom doctor says so.

Amendment 2026-10-15 (S7g): the cost gate failed on Postgres, so the default is LAZY

Why. Decision question 8 was answered “(a) ship eager if the S7f bench shows no more than a 10% regression on the no-op ceiling and none at I/O-bound durations; otherwise fall back to lazy (1b)”. S7f measured eager at 25-30% no-op overhead on Postgres at concurrency 16, 64 and 256 (SQLite 1-7%, in-memory 19-24%; I/O-bound about 1-2% where the store is not saturated), recorded in docs/benchmarks.md. The gate failed on the durable backend, so the fallback applies and lazy acquisition (option 2 of section 1) is the default. durability.heartbeat.acquire: "eager" keeps the S7c behaviour.

What lazy means (precisely).

• When (user ruling 2026-10-15, after the Opus review: lazy applies ONLY to a fresh start). A fresh start takes a hold that costs nothing when it is created. “Fresh” is classified from DURABLE state, never from caller-supplied state, and when unsure the lease is eager: a minted execution id cannot have a stream or a leftover row; a caller-chosen id is lazy too, because a start onto an existing stream is refused at the sequence-0 bootstrap before the body runs, so no start ever drives an existing stream (no extra durable read is paid). The hold: no lease-store call, one unref’d timer. If the run parks, completes or fails first, the timer is cancelled and the lease store never heard of the run. If the run is still driving after lazyGraceMs (default the renew cadence, TTL/3 = 10 s) the hold acquires through the same per-node LeaseKeeper (group-committed, renewed, released awaited at the park/terminal commit, so a prompt send winner is not blocked by its own stale lease). From the acquire on, everything is exactly the S7c/S7d protocol: the local monotonic self-fencing deadline, the dispatch and append gate, LeaseLostError aborting the run with nothing appended.

• What is eager, in both modes (the entry-point table). Every entry where a predecessor may have left a lease row, so its row is stood down on (live) or overwritten (expired) exactly as in eager mode:

  Entry point	Mode	How it is detected
  start, minted id (incl. runAgentOnce, child starts)	lazy	a ulid minted here cannot have a stream or a row
  start, caller-chosen id (every one)	lazy	a start never drives an existing stream: writeBootstrap makes sequence 0 durable before the body, so an existing stream is refused with ExecutionAlreadyExistsError before any effect (the loser’s grace timer is cancelled, it writes no row). No extra read classifies the id.
  resume (the send/notify winner, a timer wake, a recovery drive)	eager	by entry point
  resumeAs (migration)	eager	by entry point
  restoreFromState	eager	by entry point (never trusts the caller’s state)
  continue-as-new successor	eager	by entry point: a handoff from a leased predecessor
  poll / sweep re-drive claim, TimerService.fire, a local worker attempt	eager	acquireDriverLease

  These keep “lease before fence” and the stand-down on a live foreign lease (null means drive nothing and take no fence), and the unexpired-lease veto on isAbandoned. A run that runs on top of a claimant’s lease hooks onto it immediately (no second store call, no grace) so it is told if the claimant’s lease is lost.

• The lazy acquire’s three outcomes. Granted: the run is leased from now on. The store confirms another live owner’s unexpired lease (null): this run is a duplicate of a driver that is live elsewhere, so it is told its lease is lost (LeaseLostError, reason taken-over): it stops and appends nothing more (the fence guards the log either way; the eager start would have stood down before driving). Only a store-confirmed foreign owner is taken-over: a late answer (the reply arrived after the grant’s own deadline), a cancelled one (a release overtook it) and a closed keeper have their own typed outcomes (closed ends the run closed; cancelled retries; late is treated as unknown, below). The outcome is UNKNOWN (a store error or timeout: the request may have been applied and the reply lost, so a row nobody confirmed may exist): under onStoreError: "stop" the run is treated as possibly held. It self-fences at the SEND time + ttl - margin (deadline, the same rule as a granted lease, anchored at the first unknown send because a later unknown attempt may not extend the earlier row’s expiry) unless a re-acquire confirms first; retries run every min(grace, renew cadence) and bypass the keeper’s quiet period for this id so one is guaranteed to land before the deadline. This keeps the invariant that an expired row is true evidence of a stopped driver: without it a run could outlive its own unconfirmed row and a sweep would re-enter the step beside a live driver. The acquire provably sent nothing (the keeper’s quiet period, a bad id): there is no row, the run drives unleased (no evidence, never false evidence) and retries after another grace. “stop” therefore binds a run that holds a lease OR whose acquire may have written one; a fresh start on a node partitioned from the lease store drives unleased within its grace (the fence protects the log). close() tells a hold still in its grace or acquiring closed, like an eager hold.

• Evidence invariants. (1) An absent lease is no evidence (D11 section 1, S7d): a driver in its grace window has no row, so it is never read as abandoned, and a ghost row (a lease with no stream) is still GC’d by the poll. (2) The S7d veto, takeover and ghost-GC logic is unchanged and sound when a lease appears mid-run: before the acquire a heuristic isAbandoned is not vetoed (there is nothing to read); from the acquire on an unexpired lease vetoes it and an expired one is evidence. (3) The fence remains the only write guard in both modes. (4) A row that MAY exist (unknown acquire outcome) never outlives its driver under “stop”: the self-fence above. The one case still open is the ghost-row residual below.

What lazy gives up (accepted by the ruling “otherwise lazy”; this is the cost the original option 2 named).

1. A driver that dies inside its grace window left no lease. There is no evidence, so that execution is re-driven only by a host isAbandoned (as before S7), never by the poll’s default re-drive. A crash after the grace is evidenced exactly as in eager mode (detection latency TTL plus the poll interval).
2. During the window no lease vetoes a heuristic: a live silent driver younger than the grace can be double-entered by an aggressive isAbandoned (the log is still protected by the fence; the effect may be paid twice). With TTL/3 = 10 s the window is short, and lazyGraceMs is tunable; acquire: "eager" closes both gaps at the measured cost.
3. A continue-as-new chain keeps “no instant at which a stream is running and unleased” for every successor (eager, leased before its EXECUTION_STARTED while the predecessor still holds its own); only the chain’s FIRST member, a fresh start, is lazy. (The review’s ruling removed this give-up.)
4. A run the self-fence stopped after a store outage whose acquire never wrote a row left no lease either: it is re-driven only by a host isAbandoned, like a grace-window crash. (Under “stop” the alternative, driving on past an unconfirmed row, was the double-entry the review reproduced.)
5. Ghost-row residual (documented, pinned by a test). A fresh start has no stream, so the lazy ruling cannot see a ghost row: a lease left by a failed start whose release also failed, reused by an immediate retry of the same caller-chosen id. The retry drives until its grace, finds the unexpired row at its delayed acquire: a row of ANOTHER owner loses it (taken-over) and the run stops, a row of the same owner is simply renewed; either way the row’s eventual expiry makes the execution recoverable, and the retry never stood down at its start. Pinned by lease.dst.test.ts “a ghost row under a caller-chosen start”. acquire: "eager" closes it.
6. The “no lease AND head older than grace + margin” recovery (option 2’s idea) stays rejected: it compares a node-clock event timestamp with a store-clock row and needs a fleet-wide “everyone leases” declaration.

Mixed fleets. Nothing is new on the wire: a lazily acquired lease is an ordinary lease row with the same owner format, so a node in either mode reads the other’s rows identically. A lazy node’s young runs are simply invisible (absent) to everyone, which is the “no lease is no evidence” rule that already protects legacy nodes.

Measured. With the lazy default (bench:lease, off vs eager vs lazy, N = 7, docs/benchmarks.md “S7g”): Postgres no-op lazy overhead -4.4%, 0.2%, -3.3% at concurrency 16/64/256 (eager 27.6%, 28.3%, 24.5% in the same run), SQLite -4.4..3.1%, the 10% gate is met on the durable backends; in-memory (the worst case for a ratio, not the gate) is 1-18%, dominated by the per-run grace timer against a ~10 microsecond run.

Surface. durability.heartbeat.acquire: "lazy" | "eager" (default "lazy") and lazyGraceMs (default the renew cadence, a finite number >= 0); the same two fields on HeartbeatOptions; HeartbeatStatus.acquire / lazyGraceMs; loom doctor names the mode and, when lazy, the grace-window caveat. No version bump: LEASE_STORE_SPI (2.2.0), EVENT_STORE_SPI, EVENT_SCHEMA and KERNEL_ABI are untouched (the lazy acquire uses the same store calls), and the config fields are additive and optional.

Tests. packages/workflow-runtime/src/lazyLease.test.ts (no store call for a short run, acquire after the grace, taken-over, loss after acquire, outage falls soft and retries, claimant hook-on, recovery drive stays eager, config validation); tools/dst/src/lease.dst.test.ts runs every S7f scenario in both modes (N-node chaos seeds, entitlement window, veto, poll default re-drive, ghost GC, mixed fleet, negative controls) plus lazy-only scenarios (a crash inside the grace window is not evidence, the lease appearing mid-run vetoes and becomes evidence, a store outage at the grace, a ghost row under a caller-chosen start, an acquire whose reply is lost, a resume over another owner’s leftover row, a successor leased eagerly) and, in both modes, resumeAs / restoreFromState standing down on a leased execution; the double-entry detector counts a run still in its grace window as entitled, the append audit stops skipping once a node has held the lease, and half of the lazy chaos seeds keep their runs inside the grace window while the chaos ops hit; packages/app/src/heartbeatConfig.test.ts and tools/cli/src/doctor.test.ts pin the config and the doctor finding.