fix-spawn-failures

Metadata

Description

## Objective
The per-task `spawn_failures` counter, once it hits the threshold (5), blocks all dispatch for that task PERMANENTLY until manually cleared — and NO `wg` CLI clears it. `wg retry` / `wg reset` only touch status/failure_reason/retry_count, not `spawn_failures`. Today the only remedy is a surgical `graph.jsonl` edit (pause dispatcher, back up the graph, set `spawn_failures: 0` on the one line, resume) — which is what unblocked `resolve-prophage-source` after the registry/key fix landed.

## Root cause
The spawn circuit breaker (`spawn_failures` on the task; threshold `dispatcher.max_spawn_failures = 5`) trips and stays tripped. There is no CLI surface to reset it, and it does not self-heal (no time-based decay, no auto-clear on a successful spawn or on `wg retry`).

## Work
1. Make `wg retry <task>` (or a dedicated `wg reset-spawn-breaker <task>`) clear `spawn_failures` so dispatch resumes WITHOUT a graph edit.
2. Make the breaker self-healing: decay after a cooldown, and/or auto-clear on the next successful spawn, so a transient burst (e.g. caused by the layer-1 registry failure) does not permanently brick a task.
3. The breaker must be per-task and must never block the WHOLE dispatcher (only the affected task).
4. Surface a tripped breaker clearly in `wg show`/`wg status` (not just a silent "spawned=0" in the daemon log).

## Validation
- [ ] Force 5 spawn failures on a task; `wg retry <task>` clears `spawn_failures` and the worker dispatches immediately, with no graph.jsonl edit.
- [ ] A tripped breaker self-heals (cooldown decay or clear-on-success).
- [ ] A tripped breaker on one task never blocks dispatch of other tasks.
- [ ] A tripped breaker is visible in `wg show`/`wg status`.

## Reference
Root-cause layer 3 from the resolve-prophage-source incident (spawn_failures:5 stuck; required graph.jsonl surgery).

Depends on

• (none)

Required by

• done .flip-fix-spawn-failures
• open bricked
• open healthy

Log

• 2026-07-25T14:34:58.639160900+00:00 Task paused
• 2026-07-25T14:35:05.169392573+00:00 Task published
• 2026-07-25T15:29:14.792550555+00:00 Spawned by coordinator --executor pi --model zai:glm-5.2 --isolation required-worktree
• 2026-07-25T15:29:36.286981732+00:00 Task pending eval (agent reported done; awaiting `.evaluate-*` to score)
• 2026-07-25T15:32:21.412597083+00:00 Repaired evaluation pipeline drift for authoritative source attempt 1 (evalp-57a71512a153088791938a0d)
• 2026-07-25T15:35:07.203413260+00:00 error[WG-EVAL-PIPELINE-REPAIR-EXHAUSTED]: bounded repair already ran for evalp-57a71512a153088791938a0d; terminal satellites still lack evidence: .flip-fix-spawn-failures
• 2026-07-25T19:10:25.303834454+00:00 Reset by `wg recover` — reason: eval-stuck after spawn-breaker fallout; retrying on zai
• 2026-07-25T19:10:46.558913750+00:00 Spawned by coordinator --executor pi --model zai:glm-5.2 --isolation required-worktree
• 2026-07-25T19:15:26.235698203+00:00 Plan: (1) add last_spawn_failure_at + spawn_failure_cooldown_secs config; (2) wg retry + wg reset clear spawn_failures; (3) clear-on-success in record_dispatch; (4) cooldown-decay reconcile pass resurrects tripped Incomplete tasks; (5) surface in wg show + wg status; (6) unit + smoke tests
• 2026-07-25T19:26:35.590932817+00:00 Reconciliation: task recovered from orphaned state (was InProgress, agent: agent-782)
• 2026-07-25T19:26:35.590950057+00:00 Minted evaluation pipeline evalp-91e364efe73b305e96f58b90 for source attempt 2 (coordinator orphan reconciliation retry)
• 2026-07-25T19:26:36.525162829+00:00 Spawned by coordinator --executor pi --model zai:glm-5.2 --isolation required-worktree
• 2026-07-25T19:28:33.247021254+00:00 Task reset for retry from in-progress (attempt #2) — reason: cancelled per user; clean retry at max_agents=2
• 2026-07-25T19:28:33.247060695+00:00 Minted evaluation pipeline evalp-a6eb5e22e721cbd8fa9a59ad for source attempt 3 (resume-in-place in-progress retry)
• 2026-07-25T19:40:04.139543834+00:00 Spawned by coordinator --executor pi --model zai:glm-5.2 --isolation required-worktree
• 2026-07-25T19:43:32.324062743+00:00 Plan: (1) wg retry clears spawn_failures; (2) self-heal via cooldown-decay + clear-on-success; (3) per-task already (verify); (4) surface in wg show/status. Adding last_spawn_failure_at field + spawn_failure_cooldown config.
• 2026-07-25T20:19:29.966423925+00:00 Implementation complete: (1) wg retry clears spawn_failures+last_spawn_failure_at (both failed/incomplete and in-progress paths); (2) self-heal via cooldown-decay (new spawn_failure_cooldown config, default 5m) + clear-on-success in record_dispatch; (3) breaker is per-task (was already; now also surfaced in status); (4) tripped breaker visible in wg show (human+json) and wg status (attention section). Added last_spawn_failure_at field. New smoke scenario spawn_circuit_breaker_self_heal (4 phases incl. live daemon dispatch-resumption). cargo build/clippy clean; all breaker/retry/show/status tests pass.
• 2026-07-25T20:19:59.429399672+00:00 Committed: 334fbf66 — pushed to remote (branch wg/agent-757/fix-spawn-failures)
• 2026-07-25T20:22:01.112220201+00:00 Task pending eval (agent reported done; awaiting `.evaluate-*` to score)
• 2026-07-25T20:22:07.439976393+00:00 DONE — merged to main (3415323d). Smoke gate passed. Validation: [x] wg retry clears spawn_failures (smoke phase 3 + unit test); [x] self-heal via cooldown-decay (phase 4a) + clear-on-success (unit test record_dispatch_clears_breaker_on_success); [x] per-task isolation never blocks whole dispatcher (phase 2+4b + unit test); [x] visible in wg show/status (phase 1). cargo build/clippy clean. All new breaker/retry/show/status tests pass; pre-existing Pi-migration failures (edit/recover/profile) unaffected.
• 2026-07-25T20:24:55.628850700+00:00 Consumed durable verdict verdict-evalp-a6eb5e22e721cbd8fa9a59ad-evaluate-c16c41ee99b69929 exactly once: score=0.85, outcome=done