← graph

deep-survey-of

Deep survey of task lifecycle and coordinator state machine

Metadata

Statusdone
Assignedagent-872
Agent identity603daf069cd2ae4a26357aa66e8c6124154b648711a0816ffd8b660ee3b23815
Created2026-07-26T10:23:02.359307417+00:00
Started2026-07-26T10:30:54.406210119+00:00
Completed2026-07-26T11:09:28.745386344+00:00
Tokens35590176 in / 52534 out

Description

Objective

Take a complete, evidence-backed inventory of WG task lifecycle management before further redesign. The observed system repeatedly reopens/retries tasks, lets messages or helper machinery affect scheduling in surprising ways, creates evaluation satellites eagerly, conflates resource deferral with failure, and has multiple reconciliation/recovery mechanisms that sometimes amplify faults. Determine exactly what is happening rather than patching another symptom.

Survey scope

  • Enumerate the complete persisted Task/attempt state machine: every status, transition, guard, side effect, timestamp, assignment/session/worktree field, evaluation lifecycle field, and terminal/nonterminal distinction.
  • Identify every transition authority and caller: CLI commands, worker wrapper, dispatcher tick, reconciler, message send/read/poll, wait conditions, dead-agent reaper, spawn/admission/circuit breaker, retry/reset/recover, cleanup, profile/service reload, agency assign/evaluate/FLIP/rescue, verifier, and legacy migration.
  • Trace message semantics end-to-end. Reproduce whether sending an irrelevant message can reopen, resume, make eligible, refresh liveness, prevent reaping, or otherwise alter a task. Distinguish ordinary inbox messages from explicit wait --until message and chat-control traffic.
  • Map source attempt identity, exact route/session pinning, worktree ownership/merge-back, and how daemon restart/concurrent writers/replay affect them.
  • Catalogue every helper intended to improve reliability and its failure direction: eager scaffolding, auto-rescue, auto-reconcile, status repair, historical gate migration, cleanup/reaping, breaker cooldown, implicit completion, and downstream resume propagation.
  • Compare documented intent, tests, and actual live behavior/history. Identify duplicate sources of truth, cyclic transitions, non-idempotent handlers, temporal coupling, and impossible/ambiguous states.
  • Build a transition/invariant matrix and sequence diagrams for normal completion, message delivery, crash/retry, resource deferral, evaluation, worktree merge, daemon restart, and operator recovery.
  • Use credential-free live graph fixtures/model-based traces to falsify key assumptions. Do not mutate this production graph except through disposable isolated fixtures.

Deliverable

Produce docs/studies/task-lifecycle-coordinator-deep-survey.md with:

  1. current state machine and transition-authority map,
  2. ranked failure/foot-gun catalogue with source and live evidence,
  3. helpers classified as keep/simplify/remove,
  4. minimal invariants a rebuilt core must enforce,
  5. staged simplification/rebuild options with migration risk,
  6. concrete regression/model-based test plan.

No production implementation changes.

Validation

  • Every status and every writer/caller has source references.
  • Message-triggered reopen/resume behavior is reproduced or conclusively disproved with a disposable live flow.
  • At least normal, crash, deferral, evaluation, retry, message, merge, and restart traces are modeled.
  • Recent incidents map to specific violated invariants rather than generic explanations.
  • Recommendations reduce transition authorities and implicit side effects measurably.
  • Unknowns and unproven hypotheses are labeled explicitly.
  • Artifact is sufficient to derive a ratified simplified lifecycle design without another source audit.

Depends on

Required by

✉ Messages 6 messages (6 unread)

  1. #1user2026-07-26T10:51:32.282639246+00:00read
    Cross-graph requirement: diagnose an open/retried middle-of-chain task that will not spawn. Inventory every readiness/admission gate and require a deterministic -style diagnostic (dependency, pause/publication, wait/not-before, assignment/capacity, stale claim/worktree, route/profile, service/provider breaker, resource admission, evaluation hold, cycle). Generic 'possible causes' output is insufficient; identify the exact blocking gate and originating transition.
  2. #2user2026-07-26T10:51:40.356724097+00:00read
    Clarification: require a deterministic proposed `wg why-not-ready TASK` diagnostic. It must name the exact blocking gate and originating transition; generic possible-causes output is insufficient.
  3. #3deep-survey-of2026-07-26T11:01:38.410567619+00:00read
    Incorporated the cross-graph requirement: the survey now inventories the full readiness/admission pipeline, identifies why existing why-blocked is insufficient, and specifies deterministic wg why-not-ready TASK output with exact first gate + originating transition, JSON contract, exit codes, and retried middle-chain example.
  4. #4user2026-07-26T11:05:55.727890830+00:00read
    Concrete cross-graph nightmare trace to incorporate: source attempt failed for stall, wrapper later emitted done, eval reconciled low score to Done, coordinator immediately reopened terminal source because one unrelated pending message existed, respawn hit the still-owned Done-attempt worktree five times, charged each ownership refusal to the spawn breaker, task was manually marked Done, same message resurrected it again, and reset left task Ready with stale attempt/worktree ownership so daemon reported spawned=0. Required invariants: first terminal attempt event wins; later contradictory terminal events are evidence only; messages are data and only an explicit Waiting(Message) transition may resume; evaluation cannot reopen source; terminalization atomically fences/reaps process and releases or deliberately retains a named worktree lease; stale ownership is one breaker-neutral recovery/deferral, never five launch failures; reset/retry cannot leave a Ready task with an unavailable owner; loop detector must stop repeated transition cycles with an exact why-not-ready diagnosis.
  5. #5user2026-07-26T11:47:17.617734191+00:00read
    Second concrete cross-graph trace: stall detector inferred failure from no visible main-tree file while isolated Pi worker was still alive/thinking. Source became Failed at 11:11; fenced state did not stop the old process; worker wrote a 28KB artifact in its worktree at 11:14; later abandoned; no candidate checkpoint/merge occurred; evaluator/main-tree observer missed the actual work. Required survey findings: main-tree file visibility is never progress evidence for isolated attempts; watchdog cannot terminalize a live attempt without fence/reap; late fenced writes must remain recoverable; exit/no explicit wg_done|wg_fail enters NeedsFinalization and same-session completion probe, not Failed/Done; worktree remains a durable candidate until checkpoint -> validate/evaluate candidate -> merge or repair; evaluator must inspect the candidate snapshot, never assume main; abandonment cannot silently strand superior work.
  6. #6.respond-to-deep-survey-of2026-07-26T11:47:36.169426704+00:00sent
    Acknowledged all pending lifecycle traces and requirements. I will verify the existing survey covers the deterministic why-not-ready gate/origin inventory and first nightmare invariants, then extend it with the second isolated-worker trace: no main-tree visibility progress signal, live-attempt fence/reap before terminalization, recoverable late writes, NeedsFinalization plus same-session probe, durable candidate checkpoint/validation/evaluation/merge-or-repair flow, candidate-snapshot evaluation, and no silent abandonment of superior work.

Log