implement-crash-safe

Implement crash-safe candidate finalization and merge

Metadata

Statusdone
Assignedoperator-recovery
Created2026-07-26T11:49:33.634948627+00:00
Started2026-07-27T20:46:52.511274919+00:00
Completed2026-07-27T20:46:55.848908439+00:00
Tokens33911803 in / 54738 out

Description

Objective

Implement the ratified docs/design-candidate-finalization-transaction.md as the single crash-safe candidate finalization path. No isolated-worktree output may disappear or be evaluated/merged from the wrong bytes merely because Pi omitted wg_done, an old false-stall path fired, the daemon/evaluator crashed, main changed, or the worker never pushed.

Prerequisites and strict ownership boundary

  • Implement only after implement-authoritative-lifecycle, implement-pi-stalled, and design-candidate-checkpoint. Use the authoritative lifecycle kernel/ledger and the Pi watchdog's ratified continuation/quiescence receipts; do not add direct task-status writes or an alternate coordinator state path.
  • This task owns rescue/candidate checkpointing, candidate-bound deterministic validation/evaluation handoff, merge/repair transaction, retention/GC safety, replay, and their operator surfaces.
  • It does not own Pi meaningful-progress/stall classification, same-session proof, continuation budgets, process-epoch launch/resume, provider handling, or generic dead-agent reaping. It may request/consume the watchdog's typed completion-probe and quiescence results only. Do not duplicate its timer, PID killer, or session launcher.
  • Lazy evaluation work must consume this task's immutable candidate event/descriptor; do not reintroduce eager satellites or implement their selection here. The graph already serializes implement-lazy-evaluation after this task.
  • If a required lifecycle/watchdog seam differs from the ratified designs, stop production edits, log the mismatch, and amend the design contract rather than silently creating a second authority.

Required state machine and behavior

Implement the design's one mapped finalization state machine from current source attempt through: Active -> Suspect/ContinuationProbe | TerminalIntent | QuiescentNoTerminal -> NeedsFinalization -> RescueCheckpointed -> CandidateCheckpointed -> Validating -> [Evaluating] -> MergePending -> Merged | RepairNeeded | FailedPreserved. These are projections over the authoritative lifecycle/attempt/process/worktree/evaluation domains, not new ad-hoc task statuses.

  • False stall: never infer progress/stall from main contents, branch visibility, missing push, wall-clock silence alone, or ordinary messages. If the exact current process can still write, finalization does not start.
  • No-terminal Pi exit: use the watchdog-owned same-session completion-probe/continuation protocol. The structured probe inspects the pinned contract, candidate diff/status, tests, and messages, then explicitly requests done, continues the same attempt, explicitly fails with evidence, or leaves an operator hold for ambiguous replay. This task consumes the resulting terminal intent/quiescence receipt; it never launches/resumes Pi itself.
  • Generic no-terminal exit: record exact process evidence and follow the design's typed lifecycle classification. Never infer success from prose or from files in main.
  • Fence before stable bytes: done/fail/cancel/abort are intents while a current writer may live. Require exact PID + start identity/nonce + process-group reap proof, current attempt fence, and matching worktree lease epoch before treating bytes as stable. Ambiguity quarantines/holds the tree; no checkpoint promotion, merge, reuse, terminal cleanup, or deletion may race a possible writer.
  • Rescue before release/failure: after quiescence, atomically create a durable rescue ref/object and canonical content manifest for every exit/fence/failure/cancel path before worktree release or terminal cleanup. Preserve explicit worker commits and checkpoint remaining tracked/untracked/deleted content per the design. Rescue retention preserves bytes without claiming correctness.
  • Candidate promotion: only completion-ready rescue bytes become an immutable, append-only candidate version containing task/generation/attempt/fence/lease/base, commit/tree OID, canonical manifest CID/digest, policy snapshot, and creation event. A later repair creates a new version; never mutate or retag an evaluated candidate.
  • Worker/finalizer contract: workers edit/test/message and issue explicit done/fail/wait intents. They never need to push. The WG finalizer owns local checkpoint/promotion and requests downstream actions; evaluator is read-only; merge authority alone integrates main.
  • Validation/evaluation binding: deterministic validation and evaluator/FLIP materialize the exact immutable candidate in a detached read-only view, never main or the mutable retained worktree. Verdict/evidence records must include candidate commit/tree + manifest digest/CID + policy/route identity and are rejected if stale/mismatched.
  • Ordering: required policy runs deterministic validation, then evaluates the exact candidate, then merges that exact accepted candidate, then records acceptance/Done. Evaluator infrastructure failure holds without changing source success. Advisory policy may merge after deterministic validation while asynchronous evaluation remains pinned to the same descriptor; later evidence cannot reopen/replace the accepted generation. Implement the design's None/manual behavior without alternate paths.
  • Content-bound exactly-once merge: merge/apply the exact candidate commit/tree under a stable idempotency key, bind the receipt to candidate CID and main base/result, and re-verify content. A changed main or conflict produces an explicit retained RepairNeeded/merge-conflict state; never validate/evaluate/merge a 6KB main substitute for a 28KB candidate. Duplicate requests/restarts yield one merge receipt.
  • Failure/repair: explicit fail becomes lifecycle failure only after rescue retention is durable. Evaluation reject or merge conflict retains the candidate worktree/session for repair or links a lifecycle-authorized repair attempt. New bytes require a new candidate/version/verdict. First-terminal-wins makes late contradictory process/terminal/evaluation events evidence only.
  • Replay/cleanup: drive fence receipt, rescue creation, candidate promotion, validation, evaluation, merge, acceptance, retention, and cleanup through durable idempotent ledger/outbox actions. Startup replay converges at every boundary. Cleanup is ancillary and cannot erase unmerged/source-bearing/rescue/rejected/conflicted bytes or roll back semantic acceptance.

Diagnostics and operator surface

Implement the ratified status/commands/reason codes showing task generation/attempt/fence; watchdog process/continuation epoch by reference; exact PID identity and quiescence receipt; worktree path/lease epoch; rescue and candidate commit/tree/manifest IDs; validation/evaluation policy and binding; merge request/receipt/conflict; retained reason/expiry; replay action; and safe next command. Operators must be able to inspect/materialize rescue/candidate bytes, resume linked repair, resolve/retry merge, preserve/archive, and explicitly clean only eligible objects.

RED-first permanent validation

Write failing fixtures before implementation and retain them permanently:

  1. Cross-repo 28KB/6KB incident, real flow: through the installed wg, real daemon/service, wrapper, isolated worktree, lifecycle/finalizer, and operator/status surfaces (Fake-Pi is allowed only as the credential-free provider), plant a 6KB file in main and a conflicting 28KB candidate in the worker worktree. The silent worker crosses the historical false-stall boundary, writes late, exits without wg_done, and never pushes. Prove the exact writer is fenced, same-session completion probe occurs when authorized, rescue/candidate survives, evaluator reads the 28KB immutable candidate, and merge integrates it exactly once or retains it for repair. The 6KB main bytes must never be substituted.
  2. Content binding: compare candidate commit/tree/manifest digest to validator input, evaluator verdict binding, merge request/receipt, and resulting tree. Path/branch equality is insufficient. Mutating the retained tree after checkpoint cannot change the candidate; it creates a new candidate before re-evaluation.
  3. Restart matrix: kill/restart before and after terminal intent, probe handoff receipt, fence, reap proof, rescue write/ref publication, candidate promotion, validation request/result, evaluation request/verdict, merge request/conflict/receipt, acceptance, and cleanup. Replay yields one charge/action/receipt and no missing bytes.
  4. Race/retention matrix: false stall, late writer, PID reuse/start mismatch, duplicate/contradictory done/fail, explicit fail with useful WIP, cancel/abort, dirty tracked/untracked/deleted files, evaluator crash/reject/stale verdict, main edit/merge conflict, and cleanup/archive failure all retain recoverable evidence and preserve first-terminal-wins.
  5. Authority isolation: worker push is never invoked/required; evaluator has no candidate/worktree/main mutation authority; candidate finalizer does not classify Pi progress or launch/kill/resume Pi; watchdog does not checkpoint/evaluate/merge.

Add a grow-only permanent smoke scenario owned by implement-crash-safe in tests/smoke/manifest.toml. It must run the installed binary and actual daemon/worktree/finalizer human-visible terminal flow (PTY/tmux/expect or equivalent), first fail on pre-change main, and cover the 28KB/6KB no-terminal incident plus explicit fail rescue, evaluation binding, conflict repair, daemon restart, status/inspection, and exactly-once merge. A helper/library-only or main-worktree-only test is insufficient.

Validation

  • RED fixtures fail on the pre-change implementation and pass after the fix; the real installed-binary daemon/worktree smoke is permanently registered with this task as owner.
  • One mapped state machine covers false stall, no-terminal completion probe, rescue/candidate checkpoint, deterministic validation/evaluation, merge/repair, explicit failure, first-terminal-wins, and retention.
  • Candidate descriptor, validator input, verdict, merge request/receipt, and resulting tree are content-identical/bound; repair produces a new immutable version and main is never substituted.
  • Exact PID/start-identity/process-group fencing, attempt fence, worktree lease epoch, durable rescue refs, and crash replay prove no late-writer/data-loss race and exactly one merge.
  • No agent push requirement; WG finalizer, worker, watchdog, lifecycle kernel, validator, evaluator, merge authority, and operator responsibilities remain mechanically separated.
  • Explicit fail, eval reject/crash, merge conflict, dirty tree, cleanup failure, stale events, and abandoned attempts retain inspectable/recoverable evidence under the documented GC rules.
  • Focused/model/property tests, cargo fmt --check, cargo clippy, cargo build, cargo test, cargo install --path . --locked, and owned smoke pass.

Depends on

Required by

Messages 2 messages (2 unread)

  1. #1quality-pass-candidate2026-07-26T11:56:26.358288942+00:00read
    Quality-gated scope/deps: implement only after lifecycle core + Pi watchdog + candidate design. Own rescue/candidate checkpoint, candidate-bound validation/evaluation handoff, exactly-once merge/repair, replay/retention; do not implement Pi detection/session/process epochs. Worker push is never required; permanent real daemon/worktree 28KB-vs-6KB RED smoke is mandatory.
  2. #2implement-crash-safe2026-07-26T19:24:31.854336305+00:00read
    Acknowledged quality-pass scope: I will implement only the candidate finalization transaction atop lifecycle/watchdog receipts, keep push unnecessary, and add the permanent installed-daemon 28KB/6KB flow.

Log