Loom

Type to search across the guides, specification and decision records.

↑↓ navigate ↵ open esc close Pagefind

Decision records

ADR 0139 — Reserved streams archive what a trim or a delete removes

  • Status: Accepted (2026-11-01) — proposed 2026-10-31, implemented and merged; accepted by the owner
  • Date: 2026-10-31
  • Found by: GitHub issue #728, from the review of #727 (ADR 0138). It predates that PR.
  • Builds on: ADR 0138 (the ArchiveStore SPI, the erasure and tenant guards, the audit reads, the sweep), ADR 0110 (the inbox streams), ADR 0130 residuals D9 (the quarantine stream).
  • Owner rulings (2026-10-31, the review round of #729): erasure DELETES archived copies over a PLAINTEXT archive (a key’s destruction does not reach plaintext; eraseSubject never reports erased while archived plaintext copies remain; this also fixes the ADR 0138 compaction archive) and a message’s sender is sealed under the SENDER’s own subject (erasing the sender shreds their identity in other chains’ inboxes and archives, while the message body stays under the owner’s record; in a plaintext deployment their identity is redacted from other chains’ archived reserved segments by a recorded redaction). See D12, D13.
  • Owner rulings (2026-10-31): auditability stays intact (every fact the log held stays recoverable, byte-exact; a range that cannot be recovered is REPORTED, never silent); nothing in loom is pruned or deleted (a removal MOVES). Same rulings as ADR 0138, applied to the streams it did not reach.
  • Version impact: EVENT_STORE_SPI 2.5.0 -> 2.6.0 (additive: the optional deleteThrough, D8; a 2.5.0 store conforms and the caller refuses the delete it cannot make safe). ARCHIVE_STORE_SPI 1.0.0 -> 1.1.0 (second review round, additive: the optional replaceSegments compare-and-swap, D14, and the optional header field senderDigest, D19; a 1.0.0 archive conforms and erasure refuses what it cannot do). The reserved streams use the ArchiveStore of ADR 0138, keyed by the reserved stream id. No EVENT_SCHEMA, KERNEL_ABI, WIRE_ABI or SNAPSHOT_STORE_SPI change. New userspace API: InboxArchiveRequiredError, ReservedStreamDeleteUnsupportedError, EraseCoordinator’s archives / senderMatches, ArchivedCopiesRemainError, senderSubjectOf on the crypto stores, archiveChainOf / reservedLateKey / reservedLateOrigin / archivedReservedStreams, and (from the first round) inboxArchive / inboxArchiveTenant on the runtime deps, ReservedStreamArchiveRequiredError, LogLifecycle.readReservedWithArchive / reservedArchiveCoverage / readReservedArchive, SubjectExecutionExport.reservedStreams. Behaviour change (CHANGELOG): a trim or a delete of a reserved stream refuses without an archive.
  • Placement: the kernel half (@loom/workflow-runtime: trims and deletes), @loom/plugin-content (the sweep), @loom/plugin-crypto (the sealing subject), @loom/compliance (the export). Reviewed as kernel + storage + security boundary (it is the same boundary as ADR 0138).

Context

ADR 0138 made compaction MOVE the events it removes. Four other paths still removed envelopes from the log and kept nothing:

  1. trimInbox and trimTopicStream truncate the consumed prefix of a chain’s inbox (__loom_inbox__:<chain> and its per-topic streams). The payload survives in the owner’s MESSAGE_CONSUMED; the envelope (its id, timestamp, the sender metadata, a floor marker) does not.
  2. trimQuarantine frees the disposed copies of the chain’s quarantine stream (__loom_quarantine__:<chain>): verbatim copies of messages a node could not open.
  3. deleteInbox (the owner’s terminal, reclaimInbox, a send that raced the terminal) deletes every stream of a chain, including messages nobody ever consumed.
  4. The retention path (applyRetention) calls deleteInbox as part of deleting a whole execution.

The deletion-path inventory

Every path that removes envelopes from any stream, classified:

PathClass
LogLifecycle.compact (truncatePrefix(Keeping))(c) archives (ADR 0138)
LogLifecycle.applyRetention, retention archive configured (copy, then delete)(c) archives (the existing retention archive)
applyRetention, plain delete (deleteExecution, the compaction segments, the chain’s streams)(a) the user asked for the execution to go
trimInbox, trimTopicStream(b) fixed here
trimQuarantine(b) fixed here
deleteInbox from retireInbox (owner terminal), reclaimInbox, and the send-after-terminal re-delete(b) fixed here
applyRetention -> deleteInbox, retention archive configured(b) fixed here (moves to the compaction archive when one is configured)
eraseSubject -> purgeInboxStreams(a) a requested erasure; the cryptographic half is the key, which a shred destroys under the archive’s segments too
ShardedEventStore rebalance (source.deleteExecution after copyPrefix)(c) a move between shards, the destination holds every envelope
Failed-fork cleanup (deleteExecution(childQuarantine))(a) rollback of a stream the same call just created; the originals are untouched in the parent’s stream
deleteExecution / truncatePrefix inside store conformance kits and testsnot a product path

Decisions

D1 — the same ArchiveStore, keyed by the reserved stream id; archive first, then remove, refuse without one

ReservedStreamArchive (@loom/workflow-runtime) moves the exact envelopes a removal is about to take out of the stream into the archive (moveThrough for a trim, moveAll for a delete), then the caller removes. Crash order is ADR 0138’s: archive, then remove, so a crash leaves a duplicate (byte-exact, recognised by the next attempt), never a gap. Without an archive every trim and delete REFUSES with ReservedStreamArchiveRequiredError (typed, names the fix, op: "trim" | "delete"), before anything is read or removed. deleteInbox archives EVERY stream of the chain before deleting ANY (a refusal on the last leaves the first intact); a chain whose streams are all empty has nothing to lose and is deleted as before. Streams that do not exist cost one head read each, not a read.

D2 — the incarnation of a reserved stream is the chain OWNER’s bootstrap digest (amended in the review round)

ADR 0138 D11 binds a segment to the digest of the log’s bootstrap envelope. A reserved stream has none of its own, but its OWNER has: reservedIncarnationOf is the segmentDigest of sequence 0 of the execution named by the chain id (a late-run key resolves to the same owner), read with one point read per move. A chain id reused after a retention delete that kept the archive (a retention archive configured) therefore refuses the previous run’s segments (ArchiveIncarnationError, on every move, read and export) instead of leaking its envelopes into the new run’s history, exactly as ADR 0138 H2 does for compaction. The reads verify it AFTER the segments open, so a shredded owner reports the shred. Only an owner with no bootstrap (an orphan stream) falls back to the first round’s constant of the stream id (reservedIncarnation); the byte-for-byte comparison of every move still stands.

D3 — a re-created stream never reuses an archived sequence

The archive is append-only and contiguous per key, but a reserved stream is deleted at the owner’s terminal and may be re-created (a send that raced the delete, a re-opened chain, a quarantine trimmed to its anchor). Re-creation used to restart at the stream’s base. Now every writer that creates a stream consults the archive FIRST, only when the stream does not exist: appendInboxMessage (the first message of a stream), the floor seed of a fork or re-open (seedInboxFloors), and appendQuarantineCopy (a quarantine whose head is its anchor) start at archive's last through + 1 (ReservedStreamArchive.floorOf, ArchiveFloor). The common hot path (the stream exists) reads nothing from the archive. A node WITHOUT an archive in a fleet restarts at the base; D4 catches what it writes.

D4 — stragglers go to a late run, never dropped, never a wedged stream

An envelope at or below the archive’s last sequence that the archive does not hold byte-for-byte (sequence 0 included, D10) is a STRAGGLER (a send allocated its sequence before the owner’s delete and appended after it; a node with no archive floor). It is archived under the late key __loom_late__:<stream>@<n> (reservedLateKey, RESERVED_LATE_PREFIX; the first n that can take it; an exact repeat is recognised after a crash). The key is in the reserved namespace, not a suffix of a live id (review round, M4): start refuses every id beginning __loom_, so no live chain or execution can have that shape, and inboxChainOf / hasInboxStreamShape are UNCHANGED for live ids (the first round’s #late-<n> suffix changed what an existing chain job#late-1 meant). archiveChainOf (archive-key contexts only: the sealing subject, the listings) resolves a late key to the chain of its stream, so the retention delete, the sweep, the erasure and the export all see a late run as part of its chain. More than 64 late runs of one stream fails closed (ArchiveGapError). Without this, one raced send would leave a stream that can never be trimmed or deleted.

D5 — wiring: the compaction archive is the inbox archive

WorkflowRuntime deps take inboxArchive (and inboxArchiveTenant). defineConfig hands the SAME archive it hands LogLifecycle: storage: { kind: "memory" } and the sqlite/postgres presets get one automatically; a custom storage supplies storage.archiveStore or durability.retention.compactionArchive (the boot warning for a durable custom store with none now also says the inbox will not trim). It is resolved before the runtime is built. Review round (M1): no silent no-archive mode. With the inbox on and no inboxArchive, the constructor either DEFAULTS one (an in-memory event store gets an in-memory archive; when the log seals, the sealing event store offers sealArchive, which seals it under the same KeyStore, subject and sender mappings; a tenant-scoped or durable store gets none) or THROWS InboxArchiveRequiredError naming the fix. inboxArchive: "none" (kernel.inbox.archive: "none" in defineConfig) is the explicit opt-out: the runtime says so at boot through onWarn, does not trim and does not delete (maybeTrimInbox reports one ReservedStreamArchiveRequiredError per chain; a terminal’s delete is refused and reported the same way), and every refusal reaches onStorageError or, when none is wired, onWarn (once per chain and error kind). Sends keep working until inboxMaxRetained refuses them (InboxFullError now names the missing archive). The erasure guard (PlaintextArchiveError) and tenant guard (ArchiveTenantMismatchError, inboxArchiveTenant) of ADR 0138 D4/D12/D17 run before the first move.

D6 — the sealing subject of a reserved stream is the chain owner’s

EncryptingEventStore seals a chain’s inbox payloads under the OWNER’s subject, resolved from the chain id. EncryptingArchiveStore now does the same for a reserved stream’s segments (inboxChainOf, including a late run’s key), so a shred of the owner destroys the archived envelopes with the live ones (a read throws PayloadShreddedError). A custom subjectOf that reads workflowId resolves the owner’s from its first event through the new ownerWorkflowId option (defineConfig supplies it from the event store, as the event store does).

D7 — the reads, the sweep and the export see them

  • LogLifecycle.readReservedWithArchive(stream) returns { events, lateRuns }: archive u surviving stream, one entry per sequence, duplicates compared byte-exact (ArchiveLogMismatchError), ArchiveIncompleteError for a range nothing holds unless allowIncomplete; lateRuns are the late runs of D4, tagged by key and apart from the main run (review round, M3). reservedArchiveCoverage(stream) reports every unrecoverable range (why: "legacy" when the archive holds nothing for the stream, trimmed before it archived; "missing" when it holds some but not this range) and lists the late runs. Its extent is the surviving stream’s tip, or, once the stream is DELETED (nothing live), the archive’s own last sequence: a deleted stream whose archive skips a range is incomplete too. readReservedArchive(chain) returns every archived stream of a chain (late runs included, as their own entries without ranges) for the export.
  • The content sweep already enumerates the archive, so a blob only an archived inbox envelope cites stays pinned. Reserved streams were mis-classified by its compaction detection (a topic stream starts at INBOX_TOPIC_SEQ_BASE, so a bare gap test called every topic stream “compacted”; a reserved stream has no checkpoint). A reserved stream is now judged by its sequences from its first possible one (1, or the base), and an ERASING pass requires the archive’s headers to cover them (D24 of ADR 0138), else ArchiveStoresRequiredError before anything is erased. A DELETED reserved stream (archive only) is judged by the holes between its first possible sequence and its last archived one (review round, B2). Streams trimmed BEFORE this ADR (no archive) are not covered: acknowledgeUnarchivedLogs is the explicit statement that their history is already gone.
  • exportSubject returns reservedStreams per execution (the chain’s archived streams with their envelopes and every unarchived range) for the chain’s generation-0 execution.
  • applyRetention: a plain delete removes the chain’s archived reserved segments with the execution (the user asked for it to go, and the segments are keyed by a stream no sweep revisits once the log is gone); with a retention archive configured the execution is KEPT, so the chain’s streams MOVE to the compaction archive when there is one and its segments stay readable. Crash ordering, stated: the plain delete removes the archived segments FIRST and the log after, so a crash in between leaves the execution in place and the next sweep finishes the job (it over-retains for one sweep, never loses a fact the user wanted kept).

Decisions of the review round (#729)

D8 — the terminal delete is BOUNDED by what was archived: EventStore.deleteThrough (EVENT_STORE_SPI 2.6.0)

The first round archived a stream (moveAll) and then deleted the WHOLE stream: a message appended between the archive’s read and the delete was lost with no archive entry (the “residual” the first round stated). Fixed, not documented away: moveAll returns the highest sequence it archived and the delete is EventStore.deleteThrough(stream, through) - ONE atomic DELETE ... WHERE sequence <= through (sequence 0 included, unlike truncatePrefix), on the in-memory, SQLite and Postgres stores (Postgres takes the fence row first, like deleteExecution, and drops the inert token-0 row only when nothing remains), forwarded by every decorator and checked by the event-store conformance kit. A later append survives as the live remainder of the stream; the sender’s own re-check (it reads the owner after its append and re-runs the retirement) or reclaimInbox archives and deletes it the same bounded way. A store WITHOUT deleteThrough REFUSES the delete (ReservedStreamDeleteUnsupportedError, the stream is kept, reported loudly) and never falls back to a whole-stream delete. An EMPTY stream is not deleted at all (a point head read; nothing to lose, and an append since survives). Only the retention path’s explicit "discard" (the user asked for the execution to go) still uses deleteExecution.

D9 — segment ranges are tight (a gap is a reported gap)

A segment’s [from, through] is a claim that nothing in it is missing. The first round started the first segment at sequence 0 and abutted later ones, so a stream trimmed BEFORE the archive existed (1..5 gone) archived [0, 7] across the hole and read as covered. Now every segment covers exactly one run of consecutive sequences (sequence 0 is not special: it JOINS THE FIRST CONSECUTIVE RUN, so a stream 0,1,2,3 archives as ONE segment [0,3]; only a hole starts a new segment, and a stream whose 1..5 were trimmed before the archive existed archives 0 and 6..7 as two, [0,0] and [6,7]), and a later segment abuts the previous one only when it starts on an INBOX_FLOOR marker (a seeded floor skips sequences by design). The compaction path (appendMoved) starts each segment at its first removed event for the same reason. The lost range then shows as unarchived in the coverage, readReservedWithArchive throws ArchiveIncompleteError, the erasing sweep refuses (D24) and the export reports it.

D10 — sequence 0 is compared like any other

A re-created stream’s own sequence 0 (the quarantine’s clock-stamped anchor, a legacy stream restarted by a node without an archive floor) used to be dropped as “already archived”. It is compared byte-for-byte (the sender excepted, D12); a different one is a straggler and goes to a late run.

D11 — amended D5/D2/D4: see above (owner-bootstrap incarnation, no silent no-archive mode, reserved-namespace late keys).

D12 — the sender is sealed under the SENDER’s subject (owner ruling)

An inbox envelope carries two kinds of personal data: the message BODY (the owner’s data, sealed under the owner’s subject since ADR 0110) and the SENDER’s identity (payload.sender: actor, tenant, onBehalfOf) sitting in ANOTHER chain’s stream and, after a trim, in that chain’s archive. With senderSubjectOf set on EncryptingEventStore (the archive takes the same mapping: sealArchive, defineConfig’s sealArchiveLike), the sender is sealed on append under the key of senderSubjectOf(sender) as { $loomSender: 1, subject, ct } (the ciphertext is base64, deliberately carrying no __loomEnc key so the sealed-payload guard of ADR 0130 never mistakes it for an unopenable message; the subject is an OPAQUE handle stored in the clear, never the identity), and opened on read. Erasing the sender destroys that key only: every read of the inbox stream or of the archive then yields sender: { erased: true } (never an error: the OWNER’s message must stay readable) while the body, under the owner’s key, is untouched. The archive seals the sender inside the segment that is itself sealed under the chain owner’s subject. Because the erased identity changes the bytes of an archived envelope, a reserved segment’s digest masks payload.sender (reservedSegmentDigest, assertReservedSegmentDigest): every other byte stays covered, and sameReservedEnvelope still tells two different live senders apart. This is the definition of the recorded redaction: the one field an erasure may change in an archived reserved segment, changed only by an erasure that journals it (D13).

D13 — erasure deletes archived copies over a plaintext archive (owner ruling)

EraseCoordinator takes the archive(s) (archives; defineConfig wires the compaction archive, which is also the inbox archive). After the shred and the live-stream purge, over every PLAINTEXT archive (sealsSegments not true; a sealing archive is erased by the shred, unchanged) it:

  1. DELETES the compaction segments of each erased execution (ADR 0138) and the reserved segments of each erased chain (archivedReservedStreams, late runs included), and journals ERASURE_ARCHIVE_DELETED on the control stream (what was deleted is also archivedCopiesDeleted on the result);
  2. REFUSES BEFORE the shred (ArchivedCopiesRemainError) when such an archive holds the subject’s copies and cannot delete: eraseSubject never returns erased while plaintext copies remain;
  3. REDACTS the erased subject’s identity in OTHER chains’ archived reserved segments (senderMatches, default sender.actor === subjectId and the tenant agreeing): the segments are rewritten with sender: { erased: true } (the masked digest of D12 stays valid) through ONE atomic ArchiveStore.replaceSegments per key (D14), and journals it (ERASURE_SENDER_REDACTION_INTENDED first, then ERASURE_SENDER_REDACTED, D17). Cost: ONE pass over the archive’s reserved keys per erasure (a listExecutions prefix walk and a read of each key; not indexed, because an erasure is rare - an index would be one more thing to keep consistent with the segments). Stated limit: a live, not yet archived envelope of a plaintext deployment keeps its sender until it is trimmed (there is no key to shred and no store operation that rewrites a live envelope); a sealing deployment seals the sender BY DEFAULT (D20) and has no such window.

Decisions of the second review round (#729)

D14 — the redaction is ONE atomic compare-and-swap: ArchiveStore.replaceSegments (ARCHIVE_STORE_SPI 1.1.0)

The first round rewrote a key by reading its segments, deleteExecution(key), and re-adding them. A trim on ANOTHER node that appended a segment after the read was lost (the delete removed it; the re-add restored only what was read): an audit fact destroyed by an erasure that was only meant to change one field. Now the optional replaceSegments(key, expected, next) swaps a key’s whole segment list IF its headers are exactly expected (every field, senderDigest included), in one transaction on SQLite and Postgres (the same advisory lock as append), atomically in memory, forwarded by the tenant scope and re-sealed by the encrypting archive (next is sealed, expected is the inner store’s headers, which pass through verbatim), and checked by the conformance kit (including the in-place header change of a concurrent writer). A changed key rejects with ArchiveConflictError (new replaceReason) and writes NOTHING; the erasure re-reads the key and retries (16 rounds, then fails loud). An archive WITHOUT it is not erasable once it holds the subject as a sender (D16). After a swap or a delete the coordinator calls onArchiveKeyChanged (defineConfig wires WorkflowRuntime.invalidateInboxArchive), so a runtime forgets its remembered last archived sequence for the key. ARCHIVE_STORE_SPI is an ADDITION (a 1.0.0 archive conforms); the Postgres archive_segments table gains a nullable sender_digest column through the guarded add of timers.redrive (catalog first, one real ALTER under the migration lock timeout), SQLite adds it on open.

D15 — a stage left by the earlier build is recovered at BOOT, with an operator verb

Nothing writes a __loom_redact__:* stage any more (D14), but a build of this PR’s first round may have left one (a crash between its delete and its re-add: the only copy of a key). EraseCoordinator.recoverRedactions() restores a stage whose original is gone and drops one whose original is whole; it runs at loadParked() (boot, defineConfig), at the start of each redaction, and is exposed as app.recoverArchiveRedactions().

D16 — erasure refuses BEFORE the shred when it could not redact (fail closed)

assertArchivesErasable extends to the sender copies: a plaintext archive without replaceSegments that holds the subject as the sender of any archived envelope throws SenderRedactionUnsupportedError before the key is destroyed (the first round skipped such an archive silently and reported erased with the identity still in it). An archive that holds no such copy is fine without it; one that can swap is scanned once, after the shred.

D17 — the intent is journaled BEFORE the work, the completion after

deleteArchivedCopies journaled ERASURE_ARCHIVE_DELETED after deleting, so a crash between the two left a deletion nobody recorded. Now ERASURE_ARCHIVE_DELETE_INTENDED (the key list) is appended first and the completion (ERASURE_ARCHIVE_DELETED, carrying the intentId) last; a retry finds the open intent, finishes its deletes (idempotent) and writes the completion, even when nothing is left to delete. The sender redaction journals the same way (ERASURE_SENDER_REDACTION_INTENDED / ERASURE_SENDER_REDACTED).

D18 — an archived reserved stream stays readable after a KEPT retention (incarnation)

applyRetention deleted the owner’s log before deleteInbox, so reservedIncarnationOf (the owner’s bootstrap digest) fell back to the constant: the kept execution’s archived inbox became unreadable (ArchiveIncarnationError) and a live stream could not be moved. Now (a) the incarnation is derived BEFORE the owner is deleted and pinned on the ReservedStreamArchive that moves the chain’s live streams, and (b) with no bootstrap to derive from, reservedIncarnationOf(store, key, archive) adopts the incarnation the archive’s own segments for the stream already carry (a chain id REUSED after the delete has a bootstrap again, which differs, so the previous run is still refused); readReservedArchive no longer returns [] for an owner that is gone. Fail closed (N8): a KEPT execution (retention archive) with NO compaction archive used to pass "discard" to deleteInbox, dropping the chain’s unconsumed envelopes; it now refuses with ReservedStreamArchiveRequiredError BEFORE copying or deleting anything while any stream of the chain holds an envelope.

D19 — the sender has its own digest in a plaintext archive; a sealed sender is bound to its message

The segment digest MASKS payload.sender (an erasure may redact it), which left the field without bit-rot detection in a PLAINTEXT archive. A plaintext reserved segment now carries senderDigest (SHA-256 over the canonical JSON of the envelopes’ senders) beside digest, a header field persisted by every backend and verified by assertReservedSegmentDigest; a redaction recomputes it inside the same replaceSegments. A sealing archive does not store it (AEAD authenticates the sender, and a digest of a low-entropy identity would leak it). A sealed sender is bound by AEAD additional data to the MESSAGE it belongs to: chain|topic|inbox-sequence (senderBindOf), so a sealed sender pasted from one message into another fails to authenticate and never reads as that sender. The position is the envelope’s sequence, except for a quarantine copy (which a node that cannot open the message copies VERBATIM, at another sequence, stamped with the original’s inboxSeq): the stamp stands in, so the copy still opens. The ruling asked for executionId + sequence; those two change under a verbatim quarantine copy, so the chain, topic and logical position are bound instead (equally unique: a topic stream’s sequence is unique within the topic).

D20 — sender sealing is the DEFAULT of a sealing deployment: an opaque keyed handle that erasure makes unlinkable (owner ruling)

EncryptingEventStore and EncryptingArchiveStore seal the sender under SenderHandles unless senderSubjectOf is given (senderSubjectOf: false is the explicit opt-out; defineConfig warns at boot when a sealing event store disables it). The design, over the deployment’s one KeyStore:

id     = HMAC(master, [tenant, actor])               master = the KeyStore key "__loom_sender_master__"
seed   = KeyStore key "__loom_sender_seed__:<id>"    32 random bytes, minted on the sender's first message
handle = "sbj-" + HMAC(seed, "handle")[:32]          the subject the sender's sealing key lives under

The clear-text handle is a keyed function of KeyStore material, never the actor id and not a hash of it (another deployment’s, or anyone without master and the seed, cannot derive or confirm it). Erasing the sender (EncryptingEventStore.eraseSender, called by eraseSubject through defineConfig’s eraseSender, with the subject as the actor and the erasure’s tenant partition) destroys the handle’s sealing key and then the seed. The identity becomes unreadable everywhere (log and archive read { erased: true, ref: <handle> }) AND the handle becomes UNLINKABLE: it is a function of a seed that no longer exists, so even the operator holding master and the actor id cannot show which messages were theirs. A sender erased this way is BLOCKED by default (SenderErasedError: the KeyStore refuses to re-mint the seed) until an operator reinstates them (D24, owner ruling 2026-10-31). The identity is [tenant, actor]: a deployment whose erasure runs in the DEFAULT partition while senders carry a tenant must supply senderSubjectOf (the default partition cannot enumerate tenants). Cost: two extra key lookups (the seed, and the handle’s key, which surfaces an erasure that crashed between its two destroys as SenderErasedError) per send, beside the sealing key’s. Review round 3 (D22-D24) amends the design: the mint is atomic, the master is guarded, an erased sender can be reinstated. Limit, stated: a node WITHOUT the KeyStore reads the sender still sealed and keys it by its handle (dedupeKeyOf), which differs from the identity-based key an opening node computes for the same message.

D21 — an erased sender stays a distinct sender

dedupeKeyOf collapsed an erased sender to ["", "", "", messageId], so two DIFFERENT erased senders’ m1 deduped against each other (one was dropped). The erased marker now carries a stable non-identifying ref (the sealed handle; a plaintext redaction mints a random one per erased subject) and the key hashes ["erased", ref, messageId]; a sealed sender a keyless node reads is keyed by its handle.

Hash-chained stores

hashChainedEventStore offers deleteThrough only with allowCompaction (the same gate as truncatePrefix): a delete that leaves a remainder leaves a survivor chained to a deleted event. Without the opt-in the reserved-stream delete refuses loudly (ReservedStreamDeleteUnsupportedError) and keeps the stream; with it, verifyChain(stream, { archived }) verifies across the gap.

Decisions of the third review round (#729)

D22 — the seed mint is atomic: KeyStore.getOrCreate MUST be a create-if-absent (KeyStore contract)

SenderHandles.subjectOf mints the sender’s seed with keys.getOrCreate. A KeyStore whose getOrCreate is get -> isDestroyed -> wrap -> write (a KMS store over a table whose write is an upsert) lets two nodes (or two concurrent sends) each mint a seed and the last writer win: the first sender’s messages are sealed under a handle whose seed nobody can find, eraseSender destroys one handle, reports success, and the other copy of the identity stays readable. Three layers close it: (a) SenderHandles shares ONE in-flight resolution per key on a node (the master, the marker, each sender’s seed: concurrent first sends cannot mint twice); (b) the KeyStore contract now REQUIRES getOrCreate to be an atomic create-if-absent across every caller and node (a conditional write: INSERT ... ON CONFLICT DO NOTHING then re-read, or the vault’s create-if-not-exists / compare-and-set), documented on the interface; the in-repo InMemoryKeyStore is atomic by construction (no await between the lookup and the set); (c) runKeyStoreConformance gains a concurrent-getOrCreate case, so a third-party store that is not atomic fails its conformance run; (d) erasure VERIFIES after destroying (the handle’s key and the seed are re-read; a seed a concurrent writer re-minted is destroyed in a bounded loop; anything still live throws SenderEraseIncompleteError, retryable and loud). The in-repo conformance kit only proves the contract; a store in a sibling repo must be changed to meet it (the KMS plugin’s KmsKeyStore and its backing stores).

D23 — a lost master is loud; the error is accurately named; the sender AAD binds the message id

erase returned false (“never sent”) when the master key was gone, and a re-minted master derives different sender ids, so an erase on a node that lost the master silently erased nothing. The first sealed sender now also mints a marker key (__loom_sender_marker__, BEFORE the first seed); a master that is missing while the marker exists, or that was destroyed, is SenderMasterLostError (a distinct, accurately named error, never SenderErasedError, which is one sender’s erasure) on erase, reinstate AND send (a send must not mint a new master over live sealed senders). A store that never sealed a sender still answers false. The crash window between erase’s two destroys (handle key gone, seed alive) surfaces as SenderErasedError on a send (the handle’s key is looked up with the seed), and the erase retry finishes it. The sealed sender’s AEAD context now also binds the envelope’s messageId (chain|topic|sequence|messageId), so a sender cannot be pasted onto a different message at the same position. Stated limits: the quarantine copy’s metadata.quarantine.inboxSeq stamp is NOT authenticated (whoever can write the quarantine stream can relabel a copy’s position; that actor could equally replace the copy, so it is out of the threat model); the segment senderDigest is verified only when present (a segment written before ARCHIVE_STORE_SPI 1.1.0 has none: the pre-1.1.0 compatibility path).

D24 — an erased sender stays blocked until an operator REINSTATES them, recorded (owner ruling 2026-10-31)

Erasing a sender does not make the person unable to ever write again by accident, and it must not make a later message silently attributable under a recomputable handle. Default: an erased sender is BLOCKED (SenderErasedError). The explicit operator verb reinstateSender(actor, tenant?, { by }) (app.reinstateSender, EraseCoordinator.reinstateSender, EncryptingEventStore.reinstateSender) lifts the block: it mints an EPOCH authorization key in the KeyStore (__loom_sender_epoch__:<id>:<n>), and the sender’s next message mints a FRESH seed (__loom_sender_seed__:<id>:<n>), so a NEW handle. The old handle stays destroyed, the old messages stay erased ({ erased: true, ref: <old handle> }) and cannot be linked to the new handle (different seeds, nothing derives one from the other). Repeated erase/reinstate advance the epoch; the verb is idempotent and reports reinstated: false for a sender that is live or never sent. It is JOURNALED on the control stream, intent first (D17): SENDER_REINSTATE_INTENDED { tenant?, by, at } then SENDER_REINSTATED { intentId, by, at, reinstated, ref, epoch } where ref is the keyed non-identifying sender id - never the actor. There is no CLI verb (the CLI has no erasure command family; app.reinstateSender is the surface).

D25 — a failing archive at boot never skips the parked set

EraseCoordinator.loadParked ran recoverRedactions() before reading the parked erasures, so an archive that threw at boot aborted the load and left the key-window guard’s parked set EMPTY (fail open). The parked set is now loaded first; the recovery runs in a try/catch that warns loudly through onWarn (defineConfig wires it) and leaves the recovery pending for app.recoverArchiveRedactions() and the next redaction.

Cost on the hot path

Trimming already ran wherever the inbox advances (maybeTrimInbox), once per advance, bounded by the entries it removes. The archive adds, per TRIM: one ranged read of the entries being removed (the scan already read them once), one archive append, and nothing else in steady state (the archive’s last sequence is remembered per stream, bounded; a stale memory is answered once with a header read after the archive’s own conflict error). Per message consumed in a burst the added cost is zero (one trim covers the burst). A creation of a stream adds one headers read. Nothing reads the archive on the resume path (tests pin read at zero for a trim burst and for a fresh runtime that replays, sweeps and receives a send). Review round: each move adds ONE point read of the chain owner’s sequence 0 (D2: the incarnation is never cached across moves, because a cached one could outlive a retention delete and an id reuse). archivedReservedStreams is a keyset prefix scan (200 per page) per prefix: cheap for a chain’s own prefixes, and the late-key prefix walks every late run of every chain (they exist only after a raced send or a node without an archive floor). The Postgres trim cost is measured in tools/bench-throughput (LOOM_BENCH_ARCHIVE=postgres; numbers in the PR). Measured on the in-memory stores, 40 messages consumed one at a time (the worst case: one trim per message): store calls 1521 -> 1654 (+133, 8.7%), archive calls 39 (37 appends, 2 header reads), with the trim itself unchanged. The terminal delete adds one head read per stream and one read + append per stream that holds anything.

Consequences

  • trimInbox / trimTopicStream / trimQuarantine / deleteInbox take an optional ReservedStreamArchive; called without one they refuse (deleteInbox accepts "discard" for the explicit whole-execution delete of the retention path). Apps going through defineConfig need nothing; a hand-built WorkflowRuntime with inboxEnabled needs inboxArchive.
  • The archive grows by exactly what the streams lose. Every node of a fleet must configure the same archive (D3/D4 degrade, not corrupt, when one does not).
  • A message appended in the instant between the terminal’s archive read and its delete is no longer a residual: the delete is bounded (D8), the message survives as a live remainder and is archived by the sender’s re-check or reclaimInbox. A message sent to a TERMINAL chain is refused with InboxClosedError regardless.
  • A hand-built runtime with the inbox on must pass inboxArchive (or "none"); an in-memory event store defaults one. Sibling repos that build new WorkflowRuntime({ inboxEnabled: true ... }) over a durable store need it.
  • A seeded floor (a fork or a re-open) leaves a gap below it that never held a message; the erasing sweep reports it as uncovered until acknowledgeUnarchivedLogs (before this ADR the same stream stopped the sweep outright).

Open questions

  1. loom doctor does not yet check that an inbox-enabled deployment has an archive (the boot warning and the per-chain onStorageError are the guards).
  2. (Answered by D13: erasure deletes the archived copies of a plaintext archive; a plaintext deployment’s LIVE envelopes keep the sender until trimmed.)
  3. ShardedEventStore rebalancing (migrateExecution) copies and deletes whole streams and does not use deleteThrough; it is a move between shards, the destination holds every envelope.

Source: docs/adr/0139-reserved-streams-archive-before-removal.md in the loom repository.