fix-exited-worker-finish-convergence-deadlock

Metadata

Status	done
Agent identity	`fbb2d89fe44d6b761bcb4b67748ab3d417f29df70a65f8d9403308f5bb2bab78`
Created	2026-08-01T12:43:55.933329053+00:00
Started	2026-08-02T05:27:39.750280082+00:00
Completed	2026-08-03T09:42:51.413168926+00:00
Tokens	14909997 in / 26649 out

Description

The task-owned finish simplification still deadlocks across process epoch, task lifecycle, and finish-transaction state. Exact production incident on fix-candidate-wg-control-plane-destruction generation 0 attempt attempt-0-1: the Pi worker completed substantial validated WIP and emitted agent_settled; its task wrapper then failed stale_process_identity: terminal caller is not descended from current epoch 1 PID 3913691; cleanup next failed finish transaction missing; ProcessObserver classified the task NeedsFinalization with reason needs_finalization_settled; the service left it there for ~1h47m with zero live agents; wg why-blocked misleadingly said “has no blockers.” A human had to run wg retry --preserve-session, which correctly released the dead owner and resumed the exact session/worktree. This violates completed-means-cleaned, deterministic post-lifecycle convergence, and the simplified task-owned finish contract.

Remove the state dance rather than adding another state. Establish one durable completion handoff before/at child exit that is authenticated by the attempt fence and wrapper/process identity without requiring the wrapper to be a descendant of the native Pi child it owns. If an exact owner exits after settlement with WIP and no finish transaction, service convergence must deterministically either (a) finish the already-receipted completion, or (b) release the proven-dead owner exactly once and resume the same session/worktree/intent automatically. It must never sit indefinitely in NeedsFinalization, never infer semantic success from silence, never discard WIP, and never weaken stale-writer fencing. wg why-blocked/status must report the concrete pending convergence action and deadline. Expected terminal-wrapper/epoch topology must be modeled explicitly, not rejected as stale ancestry.

Validation

Candidate-binary regression reproduces the exact topology: wrapper owns native Pi child; child settles/exits; wrapper terminal call occurs after process-epoch observation; no finish transaction exists initially. Assert automatic bounded convergence to a live same-session continuation or completed Land transaction with no operator command.
Assert one session ID, one worktree, monotonically fenced generations/epochs, no competitor, no WIP loss, no duplicate promotion, and zero breaker charges.
Kill/crash at every handoff boundary (before settlement receipt, after receipt/before tx, after tx/before promotion, after promotion/before cleanup); restart service and converge exactly once.
A genuinely stale unrelated process remains rejected and cannot terminalize or write.
NeedsFinalization cannot persist with a proven-dead owner and no scheduled action/deadline; wg why-blocked must never say unblocked for this condition.
cargo fmt --check, focused lifecycle/watchdog/finalization tests, cargo check, cargo clippy, owned smoke.

The task-owned finish simplification still deadlocks across process epoch, task lifecycle, and finish-transaction state. Exact production incident on `fix-candidate-wg-control-plane-destruction` generation 0 attempt `attempt-0-1`: the Pi worker completed substantial validated WIP and emitted `agent_settled`; its task wrapper then failed `stale_process_identity: terminal caller is not descended from current epoch 1 PID 3913691`; cleanup next failed `finish transaction missing`; ProcessObserver classified the task `NeedsFinalization` with reason `needs_finalization_settled`; the service left it there for ~1h47m with zero live agents; `wg why-blocked` misleadingly said “has no blockers.” A human had to run `wg retry --preserve-session`, which correctly released the dead owner and resumed the exact session/worktree. This violates completed-means-cleaned, deterministic post-lifecycle convergence, and the simplified task-owned finish contract.

Remove the state dance rather than adding another state. Establish one durable completion handoff before/at child exit that is authenticated by the attempt fence and wrapper/process identity without requiring the wrapper to be a descendant of the native Pi child it owns. If an exact owner exits after settlement with WIP and no finish transaction, service convergence must deterministically either (a) finish the already-receipted completion, or (b) release the proven-dead owner exactly once and resume the same session/worktree/intent automatically. It must never sit indefinitely in `NeedsFinalization`, never infer semantic success from silence, never discard WIP, and never weaken stale-writer fencing. `wg why-blocked`/status must report the concrete pending convergence action and deadline. Expected terminal-wrapper/epoch topology must be modeled explicitly, not rejected as stale ancestry.

## Validation
- Candidate-binary regression reproduces the exact topology: wrapper owns native Pi child; child settles/exits; wrapper terminal call occurs after process-epoch observation; no finish transaction exists initially. Assert automatic bounded convergence to a live same-session continuation or completed Land transaction with no operator command.
- Assert one session ID, one worktree, monotonically fenced generations/epochs, no competitor, no WIP loss, no duplicate promotion, and zero breaker charges.
- Kill/crash at every handoff boundary (before settlement receipt, after receipt/before tx, after tx/before promotion, after promotion/before cleanup); restart service and converge exactly once.
- A genuinely stale unrelated process remains rejected and cannot terminalize or write.
- `NeedsFinalization` cannot persist with a proven-dead owner and no scheduled action/deadline; `wg why-blocked` must never say unblocked for this condition.
- cargo fmt --check, focused lifecycle/watchdog/finalization tests, cargo check, cargo clippy, owned smoke.

Depends on

done .assign-fix-exited-worker-finish-convergence-deadlock

Required by

✉ Messages 10 messages (10 unread)

#1user2026-08-01T12:46:44.720273638+00:00read

A parallel formalization task, formalize-lifecycle-finish-lean4, now models your exact production deadlock trace. Please keep the runtime repair's transition semantics small/pure and document the wrapper/native-child capability and convergence rank so Lean/Rust conformance can bind to it; do not wait for the proof task before fixing the urgent production bug.

#2formalize-lifecycle-finish-lean42026-08-01T13:33:38.486921878+00:00read

Formal model constraints: please expose a versioned pure reducer where exact wrapper capability = (task, generation, attempt_id, fence, wrapper_epoch, child_epoch/owned-child link). Wrapper handoff is authorized by wrapper identity owning current native child, NOT descendant-of-child ancestry; unrelated/stale tuple rejects inertly. Convergence state should carry explicit pending action+deadline and rank (receipt/no-tx -> tx -> promoted -> cleanup), with contention breaker-neutral. Please send final event/field names; I will mirror byte-for-byte fixtures without blocking your repair.

#3formalize-lifecycle-finish-lean42026-08-02T05:29:51.309731570+00:00read

Formal WIP currently uses schema_version=1 and ProcessCapability fields task_id, attempt_id, generation, fence, wrapper_epoch, child_epoch; events child_settled, wrapper_handoff{candidate,base,disposition}, owner_proven_dead, continue_same_session, begin_finish, promote, commit_cleanup, message, ownership_contention. Please send final runtime names/semantics when stable; I will update fixtures byte-for-byte, without blocking your repair.

#4formalize-lifecycle-finish-lean42026-08-02T05:45:29.901455938+00:00read

Please list both owners = ["fix-exited-worker-finish-convergence-deadlock", "formalize-lifecycle-finish-lean4"] on your candidate-binary incident restart smoke. Formal fixtures now include finish_lease plus rank cuts; I will not duplicate your urgent OS/runtime scenario.

#5formalize-lifecycle-finish-lean42026-08-02T05:45:42.183073232+00:00read

Received. I will mirror those final constants/types/field names exactly in the incident fixture/conformance once confirmed. Please message when tests stabilize; current formal broader state can retain its abstract rank while embedding your runtime capability/action wire byte-for-byte.

#6formalize-lifecycle-finish-lean42026-08-02T05:55:38.866616268+00:00read

I mirrored your exact capability fields and reducer version/rank/action names in Lean, plus formal/fixtures/runtime/v1/exited_worker_finish.json with all cuts. Once your branch lands I will make Rust conformance deserialize that fixture into your production FinishConvergenceSnapshot and call reduce_exited_worker_finish directly.

#7formalize-lifecycle-finish-lean42026-08-02T05:56:33.429720788+00:00read

Acknowledged final correction: I will use  (not attempt_fence) byte-for-byte in the runtime capability fixture. Please send final decision JSON/commit when landed.

#8formalize-lifecycle-finish-lean42026-08-02T05:56:44.311597019+00:00read

Correction: acknowledged. I will use fence, not attempt_fence, byte-for-byte. Please send the final decision JSON/commit when landed.

#9formalize-lifecycle-finish-lean42026-08-02T06:09:05.892437940+00:00read

I see your native child exited after full validation with WIP uncommitted. Please preserve WIP and commit/push when your wrapper resumes; formal conformance is ready to bind immediately.

#10formalize-lifecycle-finish-lean42026-08-02T06:27:18.249112405+00:00sent

Integrated your landed runtime reducer. Production conformance now deserializes formal/fixtures/runtime/v1/exited_worker_finish.json into the real WrapperChildCapability/FinalizationPhase, invokes reduce_exited_worker_finish for all crash cuts, and byte-compares decisions; 5/5 focused tests pass.

Log

2026-08-01T12:43:55.883918442+00:00 Task paused
2026-08-01T12:43:56.686526293+00:00 Task published
2026-08-01T12:44:06.242224531+00:00 Lightweight assignment: agent=Pi Lifecycle Hotfix Programmer (fbb2d89f), exec_mode=full, context_scope=graph, reason=Pi Lifecycle Hotfix Programmer best matches the complex process-epoch, finish-transaction, watchdog convergence, and stale-fencing implementation requirements.
2026-08-01T12:44:06.997859817+00:00 Spawned by coordinator --executor pi --model openai-codex:gpt-5.6-sol --isolation required-worktree
2026-08-01T12:44:12.888951548+00:00 Starting incident reproduction and lifecycle convergence analysis
2026-08-02T05:25:45.424967639+00:00 Task fenced for retry from in-progress (attempt #1) — waiting for exact owner release — reason: Operator-approved recovery: exact process epoch is proven absent; preserve WIP, worktree, route, and attested Pi session
2026-08-02T05:27:40.256162695+00:00 Spawned by coordinator --executor pi --model openai-codex:gpt-5.6-sol --isolation required-worktree
2026-08-02T05:27:52.138996857+00:00 Starting implementation: inspect prior WIP and lifecycle/process-epoch/finish convergence paths; will model wrapper-owned child capability and bounded exact-once convergence.
2026-08-02T05:55:03.361265332+00:00 Implemented durable wrapper/native-child binding with pre-exec bootstrap gate (closing fast-child exit race), settlement handoff, exact capability reducer, dead-owner same-session reopen, receipted transaction replay/cleanup, and concrete convergence rank/action/deadline surfaces. Candidate smoke now passes including stale-writer rejection and 28KiB WIP continuity.
2026-08-02T06:22:17.336526069+00:00 Validated: candidate-binary exited_worker_finish_convergence passed (fast child held behind pre-exec gate; wrapper owns child; settlement/no-tx; automatic exact same-session/worktree continuation; 28KiB WIP lands once; stale unrelated writer rejected; why-blocked action/deadline; restart no-op; zero spawn failures).
2026-08-02T06:22:18.658386916+00:00 Validated: cargo fmt --check, cargo build, cargo check, cargo clippy, integration_pi_watchdog (19), integration_candidate_finalization (9), pure crash-boundary reducer/wire tests (2), and generated-wrapper test pass. Full cargo test reached 3850 passes but 12 unrelated pre-existing route/TUI assertions fail; representative isolated failure passes, and none touch lifecycle changes.
2026-08-02T06:22:20.855427301+00:00 Committed and pushed: 0acfc1ff
2026-08-02T06:23:25.756219915+00:00 Bounded evaluator infrastructure state error[WG-EVAL-INSUFFICIENT-EVIDENCE]: bounded evidence insufficient [candidate-manifest:candidate-manifest:truncated,candidate-source:candidate-source:truncated,declared-artifact:declared-artifact-000:truncated,declared-artifact:declared-artifact-001:truncated,declared-artifact:declared-artifact-002:truncated,declared-artifact:declared-artifact-003:truncated,declared-artifact:declared-artifact-004:truncated,declared-artifact:declared-artifact-005:truncated,declared-artifact:declared-artifact-006:truncated,declared-artifact:declared-artifact-007:truncated,declared-artifact:declared-artifact-008:truncated,declared-artifact:declared-artifact-009:truncated,declared-artifact:declared-artifact-010:truncated,declared-artifact:declared-artifact-011:truncated] without semantic rejection or cross-executor fallback; source remains in-progress
2026-08-02T06:24:02.760961223+00:00 Deep FLIP infrastructure failed closed without source/config/repository mutation: error[WG-DEEP-UNOBSERVED-CITATION]: deep finding cites evidence the tool audit did not observe
2026-08-02T06:24:06.487152713+00:00 Bounded evaluator infrastructure state error[WG-EVAL-EVIDENCE-UNAVAILABLE]: automatic immutable evidence assembly failed without semantic rejection or cross-executor fallback; source remains in-progress
2026-08-02T06:24:22.030917829+00:00 Final promoted branch pushed at d133fdf1; wrapper-owned synchronous cleanup will expose Done after this process exits.
2026-08-02T06:24:44.558533443+00:00 Bounded evaluator infrastructure state error[WG-EVAL-EVIDENCE-UNAVAILABLE]: automatic immutable evidence assembly failed without semantic rejection or cross-executor fallback; source remains in-progress