Metadata
| Status | done |
|---|---|
| Assigned | operator-recovery |
| Created | 2026-07-26T11:49:33.634948627+00:00 |
| Started | 2026-07-27T20:46:52.511274919+00:00 |
| Completed | 2026-07-27T20:46:55.848908439+00:00 |
| Tokens | 33911803 in / 54738 out |
Description
Objective
Implement the ratified docs/design-candidate-finalization-transaction.md as the single crash-safe candidate finalization path. No isolated-worktree output may disappear or be evaluated/merged from the wrong bytes merely because Pi omitted wg_done, an old false-stall path fired, the daemon/evaluator crashed, main changed, or the worker never pushed.
Prerequisites and strict ownership boundary
- Implement only after
implement-authoritative-lifecycle,implement-pi-stalled, anddesign-candidate-checkpoint. Use the authoritative lifecycle kernel/ledger and the Pi watchdog's ratified continuation/quiescence receipts; do not add direct task-status writes or an alternate coordinator state path. - This task owns rescue/candidate checkpointing, candidate-bound deterministic validation/evaluation handoff, merge/repair transaction, retention/GC safety, replay, and their operator surfaces.
- It does not own Pi meaningful-progress/stall classification, same-session proof, continuation budgets, process-epoch launch/resume, provider handling, or generic dead-agent reaping. It may request/consume the watchdog's typed completion-probe and quiescence results only. Do not duplicate its timer, PID killer, or session launcher.
- Lazy evaluation work must consume this task's immutable candidate event/descriptor; do not reintroduce eager satellites or implement their selection here. The graph already serializes
implement-lazy-evaluationafter this task. - If a required lifecycle/watchdog seam differs from the ratified designs, stop production edits, log the mismatch, and amend the design contract rather than silently creating a second authority.
Required state machine and behavior
Implement the design's one mapped finalization state machine from current source attempt through:
Active -> Suspect/ContinuationProbe | TerminalIntent | QuiescentNoTerminal -> NeedsFinalization -> RescueCheckpointed -> CandidateCheckpointed -> Validating -> [Evaluating] -> MergePending -> Merged | RepairNeeded | FailedPreserved.
These are projections over the authoritative lifecycle/attempt/process/worktree/evaluation domains, not new ad-hoc task statuses.
- False stall: never infer progress/stall from main contents, branch visibility, missing push, wall-clock silence alone, or ordinary messages. If the exact current process can still write, finalization does not start.
- No-terminal Pi exit: use the watchdog-owned same-session completion-probe/continuation protocol. The structured probe inspects the pinned contract, candidate diff/status, tests, and messages, then explicitly requests done, continues the same attempt, explicitly fails with evidence, or leaves an operator hold for ambiguous replay. This task consumes the resulting terminal intent/quiescence receipt; it never launches/resumes Pi itself.
- Generic no-terminal exit: record exact process evidence and follow the design's typed lifecycle classification. Never infer success from prose or from files in main.
- Fence before stable bytes: done/fail/cancel/abort are intents while a current writer may live. Require exact PID + start identity/nonce + process-group reap proof, current attempt fence, and matching worktree lease epoch before treating bytes as stable. Ambiguity quarantines/holds the tree; no checkpoint promotion, merge, reuse, terminal cleanup, or deletion may race a possible writer.
- Rescue before release/failure: after quiescence, atomically create a durable rescue ref/object and canonical content manifest for every exit/fence/failure/cancel path before worktree release or terminal cleanup. Preserve explicit worker commits and checkpoint remaining tracked/untracked/deleted content per the design. Rescue retention preserves bytes without claiming correctness.
- Candidate promotion: only completion-ready rescue bytes become an immutable, append-only candidate version containing task/generation/attempt/fence/lease/base, commit/tree OID, canonical manifest CID/digest, policy snapshot, and creation event. A later repair creates a new version; never mutate or retag an evaluated candidate.
- Worker/finalizer contract: workers edit/test/message and issue explicit done/fail/wait intents. They never need to push. The WG finalizer owns local checkpoint/promotion and requests downstream actions; evaluator is read-only; merge authority alone integrates main.
- Validation/evaluation binding: deterministic validation and evaluator/FLIP materialize the exact immutable candidate in a detached read-only view, never main or the mutable retained worktree. Verdict/evidence records must include candidate commit/tree + manifest digest/CID + policy/route identity and are rejected if stale/mismatched.
- Ordering: required policy runs deterministic validation, then evaluates the exact candidate, then merges that exact accepted candidate, then records acceptance/Done. Evaluator infrastructure failure holds without changing source success. Advisory policy may merge after deterministic validation while asynchronous evaluation remains pinned to the same descriptor; later evidence cannot reopen/replace the accepted generation. Implement the design's
None/manual behavior without alternate paths. - Content-bound exactly-once merge: merge/apply the exact candidate commit/tree under a stable idempotency key, bind the receipt to candidate CID and main base/result, and re-verify content. A changed main or conflict produces an explicit retained
RepairNeeded/merge-conflict state; never validate/evaluate/merge a 6KB main substitute for a 28KB candidate. Duplicate requests/restarts yield one merge receipt. - Failure/repair: explicit fail becomes lifecycle failure only after rescue retention is durable. Evaluation reject or merge conflict retains the candidate worktree/session for repair or links a lifecycle-authorized repair attempt. New bytes require a new candidate/version/verdict. First-terminal-wins makes late contradictory process/terminal/evaluation events evidence only.
- Replay/cleanup: drive fence receipt, rescue creation, candidate promotion, validation, evaluation, merge, acceptance, retention, and cleanup through durable idempotent ledger/outbox actions. Startup replay converges at every boundary. Cleanup is ancillary and cannot erase unmerged/source-bearing/rescue/rejected/conflicted bytes or roll back semantic acceptance.
Diagnostics and operator surface
Implement the ratified status/commands/reason codes showing task generation/attempt/fence; watchdog process/continuation epoch by reference; exact PID identity and quiescence receipt; worktree path/lease epoch; rescue and candidate commit/tree/manifest IDs; validation/evaluation policy and binding; merge request/receipt/conflict; retained reason/expiry; replay action; and safe next command. Operators must be able to inspect/materialize rescue/candidate bytes, resume linked repair, resolve/retry merge, preserve/archive, and explicitly clean only eligible objects.
RED-first permanent validation
Write failing fixtures before implementation and retain them permanently:
- Cross-repo 28KB/6KB incident, real flow: through the installed
wg, real daemon/service, wrapper, isolated worktree, lifecycle/finalizer, and operator/status surfaces (Fake-Pi is allowed only as the credential-free provider), plant a 6KB file in main and a conflicting 28KB candidate in the worker worktree. The silent worker crosses the historical false-stall boundary, writes late, exits withoutwg_done, and never pushes. Prove the exact writer is fenced, same-session completion probe occurs when authorized, rescue/candidate survives, evaluator reads the 28KB immutable candidate, and merge integrates it exactly once or retains it for repair. The 6KB main bytes must never be substituted. - Content binding: compare candidate commit/tree/manifest digest to validator input, evaluator verdict binding, merge request/receipt, and resulting tree. Path/branch equality is insufficient. Mutating the retained tree after checkpoint cannot change the candidate; it creates a new candidate before re-evaluation.
- Restart matrix: kill/restart before and after terminal intent, probe handoff receipt, fence, reap proof, rescue write/ref publication, candidate promotion, validation request/result, evaluation request/verdict, merge request/conflict/receipt, acceptance, and cleanup. Replay yields one charge/action/receipt and no missing bytes.
- Race/retention matrix: false stall, late writer, PID reuse/start mismatch, duplicate/contradictory done/fail, explicit fail with useful WIP, cancel/abort, dirty tracked/untracked/deleted files, evaluator crash/reject/stale verdict, main edit/merge conflict, and cleanup/archive failure all retain recoverable evidence and preserve first-terminal-wins.
- Authority isolation: worker push is never invoked/required; evaluator has no candidate/worktree/main mutation authority; candidate finalizer does not classify Pi progress or launch/kill/resume Pi; watchdog does not checkpoint/evaluate/merge.
Add a grow-only permanent smoke scenario owned by implement-crash-safe in tests/smoke/manifest.toml. It must run the installed binary and actual daemon/worktree/finalizer human-visible terminal flow (PTY/tmux/expect or equivalent), first fail on pre-change main, and cover the 28KB/6KB no-terminal incident plus explicit fail rescue, evaluation binding, conflict repair, daemon restart, status/inspection, and exactly-once merge. A helper/library-only or main-worktree-only test is insufficient.
Validation
- RED fixtures fail on the pre-change implementation and pass after the fix; the real installed-binary daemon/worktree smoke is permanently registered with this task as owner.
- One mapped state machine covers false stall, no-terminal completion probe, rescue/candidate checkpoint, deterministic validation/evaluation, merge/repair, explicit failure, first-terminal-wins, and retention.
- Candidate descriptor, validator input, verdict, merge request/receipt, and resulting tree are content-identical/bound; repair produces a new immutable version and main is never substituted.
- Exact PID/start-identity/process-group fencing, attempt fence, worktree lease epoch, durable rescue refs, and crash replay prove no late-writer/data-loss race and exactly one merge.
- No agent push requirement; WG finalizer, worker, watchdog, lifecycle kernel, validator, evaluator, merge authority, and operator responsibilities remain mechanically separated.
- Explicit fail, eval reject/crash, merge conflict, dirty tree, cleanup failure, stale events, and abandoned attempts retain inspectable/recoverable evidence under the documented GC rules.
-
Focused/model/property tests,
cargo fmt --check,cargo clippy,cargo build,cargo test,cargo install --path . --locked, and owned smoke pass.
Depends on
Required by
Log
- 2026-07-26T11:49:33.601672859+00:00 Task paused
- 2026-07-26T11:49:49.107155172+00:00 Task published
- 2026-07-26T19:24:12.114921431+00:00 Spawn preparation deferred before launch permit; rollback is complete and no circuit-breaker charge was recorded. Repair the reported checkout/configuration condition and retry: spawn transaction for agent-898 rolled back (task remains dispatchable; rollback diagnostics: complete): failed to establish isolated-worktree observer baseline: observer state already exists; use open_at for restart reconciliation
- 2026-07-26T19:24:23.125719643+00:00 Spawned by coordinator --executor pi --model openai-codex:gpt-5.6-sol --isolation required-worktree
- 2026-07-26T19:24:31.918549352+00:00 Starting: inspecting prior WIP, ratified design, lifecycle/watchdog seams, and existing finalization/evaluation/worktree code.
- 2026-07-26T19:30:31.559683235+00:00 RED confirmed: permanent candidate finalization integration fixture fails to compile because worksgood::finalization is absent; cases bind 28KB vs 6KB, immutable mutation, exactly-once merge/conflict, dirty failure rescue, and fail-closed quiescence.
- 2026-07-26T20:25:01.288706220+00:00 Implemented content-addressed rescue/candidate descriptors, private-index dirty/untracked/deleted snapshots, immutable refs/manifests, candidate-bound validation/evaluation request, CAS merge/result refs/receipts/conflict retention, crash replay (including post-CAS receipt reconstruction), v2 repair versions, fail-closed GC/status/materialize/repair surfaces, Pi terminal intent/reap handoff, wrapper settle/retention, and daemon startup replay. RED suite and installed 28KB/6KB smoke are green.
- 2026-07-26T20:38:16.260565426+00:00 Validation: RED fixture became green (5 tests); lifecycle 45/45 and Pi watchdog 12/12; cargo fmt --check, cargo build/install, and cargo clippy pass (repository warnings only); installed-binary candidate finalization smoke and updated Pi watchdog smoke pass. Full cargo test reached 3000 passed/38 ignored with only the same 8 pre-existing dispatch/profile failures documented by both prerequisite tasks.
- 2026-07-26T20:38:16.390632580+00:00 Committed: 23f05b26 — pushed to remote
- 2026-07-26T20:39:27.762195560+00:00 Reconciliation recorded one fenced lost attempt; explicit retry is required
- 2026-07-27T15:41:59.784232570+00:00 Task reset for retry from failed (attempt #2) — reason: Operator recovery: implementation, validation, commit 23f05b26, and owned smoke completed before a missing terminal receipt caused Lost; preserve exact Pi session and worktree to finalize safely
- 2026-07-27T15:42:00.806731190+00:00 Spawn preparation deferred before launch permit; rollback is complete and no circuit-breaker charge was recorded. Repair the reported checkout/configuration condition and retry: cannot resume exact Pi session '019f9fe2-b62b-7f92-bc91-23b3a2b51d2d': found 2 matching session files under /home/bot/wg/.wg/agents
- 2026-07-27T15:43:12.465290387+00:00 Operator session reconciliation: preserved duplicate WG bootstrap header at agent-898/pi-session-bootstrap-archive/wg_019f9fe2-b62b-7f92-bc91-23b3a2b51d2d.jsonl (sha256=a409861c90aa9c0be571bc7bcad06c01b1f5da3b17ad02ea73f84e1d94a3679d); selected the sole substantive Pi journal 2026-07-26T19-24-23-925Z_019f9fe2-b62b-7f92-bc91-23b3a2b51d2d.jsonl (sha256=09fbe4bb5461d552c9c5e99a7637c19fa0676072dbfad669e36747a910b422cc, cwd=exact retained worktree). No session bytes deleted; exact session ID unchanged.
- 2026-07-27T15:43:16.228779247+00:00 Spawned by coordinator --executor pi --model openai-codex:gpt-5.6-sol --isolation required-worktree
- 2026-07-27T15:43:34.215106581+00:00 Resuming prior committed implementation after terminal-intent handoff; inspecting task/finalization status and branch before any edits.
- 2026-07-27T15:46:18.461990949+00:00 Continuation review: prior branch commit 23f05b26 is clean, pushed, and contains the registered installed-binary candidate transaction smoke plus focused RED-first fixtures. No new production edits required on retry.
- 2026-07-27T15:52:35.790001638+00:00 Retry validation complete: reinstalled exact branch binary (global wg had drifted to another agent build), cargo fmt --check, cargo build --locked, focused finalization 5/5, and permanent installed-binary candidate_finalization_transaction smoke all pass. Branch remains clean at pushed 23f05b26.
- 2026-07-27T15:53:01.539671111+00:00 Reconciliation recorded one fenced lost attempt; explicit retry is required
- 2026-07-27T18:01:39.987439300+00:00 Recovered immutable candidate 23f05b26 onto current main in merge commit 90fd288a (origin/wg/agent-918/recover-implement-crash-safe-candidate). Exact candidate history preserved as second parent; historical Lost events retained unchanged. Focused tests and installed candidate/lifecycle/watchdog/observer/spawn/waiting smokes pass.
- 2026-07-27T20:46:52.452767553+00:00 Task reset for retry from failed (attempt #3) — reason: Operator acceptance after exact candidate 23f05b26 was integrated as second parent of validated main commit 90fd288a; rerun owned smoke on installed integrated build before unblocking dependents
- 2026-07-27T20:46:52.511275490+00:00 Task claimed by @operator-recovery
- 2026-07-27T20:46:52.566886586+00:00 Post-bootstrap operator acceptance: origin/main=90fd288a, exact immutable candidate 23f05b26 is preserved as ancestor/second parent, and the installed daemon binary was rebuilt from that tree. Running the task-owned installed-binary smoke now; historical Lost attempts remain immutable evidence.
- 2026-07-27T20:46:55.848908669+00:00 Task marked as done