← graph

impl-supervisor-hard-agent

Impl: long-lived supervisor hard-agent for graph health + auto-reset

Metadata

Statusabandoned ‖ paused
Created2026-07-25T20:57:07.605943960+00:00
Tagssupervisor, lifecycle, daemon, config
Failure reasonSuperseded by the operator-published post-lifecycle synthesis: deterministic convergence is service-owned; no supervisor persona, adaptive outage controller, or composition authority remains.

Description

Objective

Build the long-lived "supervisor" hard-agent — a slow-tick (180s) stateful persona that scans the graph for tasks stuck FOR DUMB REASONS and resets/requeues them, with persistent memory so it neither flaps nor double-acts with the existing reaper/sweep/reconciler. This is the long-lived-graph-health layer that the existing reactive layer (reaper/sweep/reconciler) leaves open.

Context: docs/studies/supervisor-hard-agent-design.md (the full design — read it; this task implements §2-§10). docs/studies/roadmap-rate-limit-and-distribution.md §1/§4(T3). The supervisor CONSUMES the T2 telemetry signal and is a peer of the T4 controller (the composition contract is wired in T5).

Boundary (do not cross — study §6)

The supervisor acts ONLY on tasks the reactive layer has stopped touching, and ONLY on conditions the reactive layer has no policy for. It: reopens terminal Failed/FailedPendingEval tasks for reversible/dumb reasons (C1/C6/C7); stops unbounded re-arm loops the throttle missed (C2/C3/C8); cleans stranded agency satellites (C5). It NEVER reaps PIDs, NEVER runs wg recover, NEVER deletes worktrees, NEVER touches .wg/agents/, and NEVER touches max_agents (that is the controller). It skips any task that is InProgress or touched by triage/sweep/reconciler within react_settle (90s).

What to build (study §10 code map)

  1. New module src/supervisor/:
    • mod.rs — the Supervisor persona: run_pass(dir, graph_path, config, dry_run), the per-class matcher table, loop-prevention guards (study §5), and a hard pass timeout (pass_timeout, 30s, deadline-checked not thread-killed).
    • memory.rs — the sidecar journal (.wg/supervisor/journal.jsonl, append-only, bounded, audit trail) + state rollup (.wg/supervisor/state.json, atomic-write mirroring coordinator-state-N.json). load_state, record_action, snapshot_health (the producer for the controller — writes the bounded health snapshot from study §7).
    • policy.rs — the 8-class (C1-C8) policy table AS DATA (study §3), so adding a class is a table row not a code branch. Reuse FailureClass allow/deny lists from src/graph.rs:129. For C6, call eval_lifecycle::begin_source_attempt inside the supervisor's own modify_graph transaction exactly as the reaper does (triage.rs:~510). For C5, mirror recover's AbandonFollowup path (src/commands/recover.rs).
  2. Per-task memory on the graph row (src/graph.rs:~640-720): add supervisor_reset_count: u32 and last_supervisor_action: Option<String> (serde default/skip-if), plus a needs_human: bool marker. These ride the existing atomic graph transaction.
  3. Recognition memory (C3): a bucketed content hash of (failure_class, failure_reason[:120], exit_code, died_within_secs_bucket) keyed by task id in the journal; 3 identical hashes => C3.
  4. Config (src/config.rs): new [supervisor] section — enabled (default false), dry_run (default true), interval (180), min_interval (60), min_task_interval (900), max_resets_per_task (1), max_actions_per_pass (3), react_settle (90), urgent_kick (false), pass_timeout (30), classes (Vec). Plus default_* fns mirroring default_poll_interval/default_reaper_grace_seconds.
  5. Daemon-loop wiring (src/commands/service/mod.rs:~3138, the last_coordinator_tick.elapsed() block): add a parallel last_supervisor_pass check + supervisor::run_pass(...) call, gated by config.supervisor.enabled and the paused flag. Reuse the SAME daemon loop / SAME graph lock — NOT a separate process or thread (study §2).
  6. CLI (new src/commands/supervisor_cmd.rs, register in src/commands/mod.rs): wg supervisor {run [--dry-run], status, pause, resume, revert <task>}. run --dry-run reuses Stage 0/1 logic.
  7. Status surface (src/commands/status.rs or wherever wg status renders): surface needs_human count + "supervisor stalled" (last_pass older than 3x interval with daemon alive) + last-pass summary from state.json.
  8. Kill switch (study §8.4): [supervisor] enabled = false default; WG_SUPERVISOR_DISABLE=1 env override; wg supervisor pause/resume live toggle.

Rollout state for THIS task

Ship at Stage 0 (observer, default-on) + Stage 1 (dry-run, opt-in) per study §9: the supervisor wakes, scans, writes the journal + state.json health snapshot, and computes what it WOULD do for each class, but mutates no graph row by default. The live-mutation CODE PATH is implemented and gated behind dry_run/classes config, but promotion to Stage 2 (limited live: C1/C5/C6) and Stage 3 (full live) is an explicit operator decision audited via the journal — note that as a follow-on, NOT part of this task.

File scope

  • new src/supervisor/{mod,memory,policy}.rs
  • src/graph.rs (Task fields near :640; needs_human)
  • src/config.rs ([supervisor])
  • src/commands/service/mod.rs (daemon-loop timer block near :3138)
  • new src/commands/supervisor_cmd.rs, src/commands/mod.rs
  • src/commands/status.rs
  • new smoke tests/smoke/scenarios/supervisor_loop_prevention.sh + tests/smoke/manifest.toml (grow-only owners list)

Implement directly. The supervisor is large but cohesive (one module + a policy table); only decompose if you hit genuine context pressure, and keep edits within the listed file scope.

Validation

  • Unit test per class matcher (C1-C8) against fixture tasks — reuse the Task builder helpers in src/eval_lifecycle.rs:~2790 tests. C1 reopens a FailedPendingEval tar pit once then escalates; C3 recognizes 3 identical crash signatures and leaves for human; C5 abandons a stranded satellite; C6 resets a verdict-less PendingEval source once.
  • Unit test: loop-prevention bounds hold — per-task cap (1 for C1/C6, 0 for C2/C3/C8), per-task min-interval (15 min), global per-pass cap (3). A supervisor pass over 200 resettable tasks reopens at most 3.
  • Unit test: the supervisor SKIPS InProgress tasks and anything touched by triage/sweep/reconciler within react_settle (90s) — proves it does not race the reactive layer.
  • Unit test: Stage 0 / dry_run=true writes the journal + state.json health snapshot but mutates NO graph row.
  • Unit test: the health snapshot (study §7) is written each pass with the right counts (needs_human, backoff, storm_cooldown, stranded_satellites, pending_eval_stuck, failed_revertible).
  • Kill switch: [supervisor] enabled=false skips the whole subsystem at config load; WG_SUPERVISOR_DISABLE=1 overrides to off; wg supervisor pause/resume toggle live.
  • Pass timeout: a pathological scan is aborted at pass_timeout without leaving the graph lock held.
  • wg supervisor run --dry-run against seeded C1/C2/C3 fixtures emits human-readable diffs (task X: Failed -> Open (class C1, reason ...)) with zero graph mutations.
  • wg status surfaces the needs_human count + last-pass summary; "supervisor stalled" appears when last_pass is stale.
  • Smoke scenario tests/smoke/scenarios/supervisor_loop_prevention.sh added + listed in owners of tests/smoke/manifest.toml (grow-only): seed a C1 tar pit and a C2 storm, run passes, assert bounded resets + escalation; assert dry-run writes no graph mutations; assert the kill switch.
  • cargo build + cargo test pass; cargo fmt --check + cargo clippy clean.
  • No model pin; routes through the active zai profile.

Depends on

Required by

Log