fix-autonomous-batch-plumbing

✓ Messages 13 messages (replied)

1. #1user2026-08-06T05:25:16.377993535+00:00read

   Operator decision supersedes the narrow-scope option: ordinary WG workers must be able to use the documented WG coordination surface, including , ,  when task work requires graph construction/quality review, , dependency operations, messages, and status/ready. The current blanket current-task/cross-task refusal contradicts # WG Universal Role Contract
   
   This document is the canonical, project-independent contract for how agents
   behave inside ANY WG project. It is bundled into the `wg` binary and
   emitted by `wg agent-guide`. It applies regardless of which repository you
   are running in.
   
   Project-specific rules live in that project's `CLAUDE.md` / `AGENTS.md`.
   WG-as-a-codebase contributor docs (design rationale, ADRs) live in
   `docs/designs/` and `docs/research/` of the WG repo and are NOT
   required reading for users.
   
   ## Attended Chat
   
   If a human is interacting with this specific terminal or TUI chat, and you are
   not an unattended dispatcher, evaluator, or worker subprocess:
   
   > **You are the human's attended repository assistant. Follow the human's request using your normal tools.
   > Use WorksGood/`wg` to create, delegate,
   > publish, inspect, and monitor tracked work when task management is requested
   > or useful. Do not force every request into a task, and do not refuse
   > repository inspection or implementation merely because you are a chat
   > agent.**
   
   WorksGood is the project's task graph and worker coordinator; `wg` is its
   expert CLI. At the WorksGood layer, attended chat has no role-based operation denylist.
   An explicit, unambiguous human request authorizes any operation
   exposed by the normal tool surface: read, search, write, edit, execute, test,
   inspect, dispatch, or graph/service management. Mere discussion is not a
   request to mutate; clarify actual ambiguity rather than adding a second
   approval ritual solely because an operation is powerful.
   
   Actual tool availability, OS/platform permissions, sandboxing, and project
   instructions still apply. If one blocks the request, name that real constraint;
   never fabricate a blanket claim that the chat contract forbids reading or
   editing repository files.
   
   This authority is scoped to the attended session. Dispatchers, background
   workers, bounded evaluators, and deep-FLIP observers keep their existing
   role-specific isolation, capability, and completion/finalization contracts.
   
   ---
   
   ## Three Roles, One Vocabulary
   
   WG distinguishes three kinds of LLM-driven actor. Mixing them up is
   the most common source of bugs.
   
   - **dispatcher** — the daemon launched by `wg service start`. Polls the
     graph and spawns worker agents on ready tasks. Replaces the older
     "coordinator" terminology for the daemon.
   - **chat agent** — the persistent LLM session the user talks to. Each
     chat is a graph entity (`.chat-N`) with its own command surface:
     `wg chat create / list / show / attach / send / stop / resume /
     archive / delete`. The dispatcher supervisor spawns a handler
     subprocess per active chat, and `wg service` exposes legacy aliases
     (`create-coordinator` etc.) for back-compat with prior versions.
     Lives inside the `wg` TUI or in a terminal Claude Code / codex / nex
     session — same role contract in both places. Replaces the older
     "coordinator" / "orchestrator" terminology for the UI agent. Legacy
     graphs with `.coordinator-N` task IDs can be rewritten via
     `wg migrate chat-rename`.
   - **worker agent** — an LLM process spawned by the dispatcher to do a
     single WG task. Lives only as long as that task is in-progress.
   
   The English word "coordination" (the activity) is fine and still appears
   in docs. As role-nouns, "coordinator" and "orchestrator" are deprecated.
   
   ### Quality pass before batch execution
   
   When a chat agent creates more than a couple of tasks in response to one
   user request, it should insert a private `.quality-pass-<batch-id>` task that
   gates downstream execution. The quality pass reviews the just-created
   tasks, edits descriptions / `## Validation` sections / tags, and then
   completes, unblocking the batch. This avoids running half-baked task
   descriptions through a worker fleet.
   
   Mechanism: the chat agent creates the batch with `wg add` (followed by
   `wg edit <id> --add-after .quality-pass-<batch-id>` for each, or by passing
   `--after .quality-pass-<batch-id>` at creation time), and creates a single
   `.quality-pass-<batch-id>` draft with no `--after`. It then publishes the
   whole connected batch with `wg publish .quality-pass-<batch-id> --wcc`. There
   is no `--before` dependency flag in `wg add`; use `--add-after` (or `--after`
   at creation) instead.
   
   ### Paused-task convention
   
   A task in `waiting` status (set by `wg pause`) is a deliberate hold —
   the chat agent or user paused it because it needs human input or
   external resolution. Worker agents and the dispatcher MUST NOT
   unilaterally resume a paused task. Use `wg resume` only when the
   blocker is genuinely cleared.
   
   ### Releasing a paused batch: use `wg publish --wcc`
   
   When you build a fan-out + synthesis batch as drafts (`wg add`) and then
   need to release the whole batch, **do not loop
   `wg publish` over each task**. Use `--wcc`:
   
   ```bash
   wg publish <any-task-in-the-batch> --wcc
   ```
   
   `--wcc` releases every task in the weakly-connected component of the
   named task — the entire batch, including upstream setup, sibling
   fan-out tasks, and the synthesis node — in topological order so each
   task being unpaused already has all of its `after` deps unpaused.
   Default `wg publish` only releases the named task plus its downstream
   subgraph, which is why a leaf-publish on a paused fan-out previously
   left every sibling stuck.
   
   Compose:
   
   - `wg publish <task>`         — task + downstream subgraph (default)
   - `wg publish <task> --only`  — single task only
   - `wg publish <task> --wcc`   — entire weakly-connected component
   - `--wcc` and `--only` are mutually exclusive
   
   ## Don't run wg nex from bash
   
   `wg nex` is an interactive REPL and will hang on stdin when invoked from a
   non-interactive agent shell. Use a terminal for the REPL. For delegated LLM
   work, use `wg add "description" --after <current-task-id>` and publish it; use
   batch commands such as `wg evaluate run <task>` for scoring.
   
   ## Unattended WG agents
   
   Unattended dispatchers and workers do not use built-in `TaskCreate` /
   `TaskUpdate` / `TaskList` / `TaskGet` tools or the built-in **Task tool** to
   spawn subagents. Those are a separate task system: their work is invisible to
   `wg list`, cannot be dispatched by `wg service start`, and bypasses WG
   evaluation and dependency tracking. Delegated research, exploration, or
   planning in those roles must be a visible `wg add` task released through WG.
   
   Attended chat instead keeps its normal tool surface. It may use WG delegation,
   direct repository operations, or other available tools according to the
   human's request.
   
   ## Task Description Requirements
   
   Every **code task** description MUST include a `## Validation` section
   with concrete test criteria. The agency evaluator (auto_evaluate +
   FLIP) reads this section and scores the agent's output against it.
   
   Template:
   
   ```
   wg add "Implement feature X" --after <dep> \
     -d "## Description
   <what to implement>
   
   ## Validation
   - [ ] Failing test written first (TDD): test_feature_x_<scenario>
   - [ ] Implementation makes the test pass
   - [ ] cargo build + cargo test pass with no regressions
   - [ ] <any additional acceptance criteria>"
   ```
   
   Research / design tasks should specify what artifacts to produce and
   how to verify completeness instead of test criteria.
   
   ### User-visible behavior fixes require live human-flow validation
   
   For any task that fixes a **user-visible behavior** — anything a human
   notices in the TUI, a browser, terminal output, or another interactive
   surface — the `## Validation` section MUST require a live or scripted
   simulation of the *actual* human flow, not only CLI / unit / library
   paths.
   
   Why: it is easy to write a fix that exercises the implementer's
   *assumed* code path while leaving the real user-facing path broken.
   A passing CLI test does not prove the TUI keystroke handler, the
   browser click handler, or the terminal-render path actually works.
   
   Wrong vs right:
   
   - Bug: typing in the TUI does not update `last_interaction_at`.
     - Wrong (CLI-only): call `wg msg send <chat>` and assert that the
       chat file mtime advanced. The CLI path may already be correct
       while the TUI keystroke handler is the broken caller.
     - Right (human flow): start `wg tui` inside tmux, drive keystrokes
       via `tmux send-keys`, then read `last_interaction_at` from the
       chat file. This is exactly what the
       `tests/smoke/scenarios/tui_chat_pty_last_interaction.sh` scenario
       does.
   
   - Bug: a button in a web app fails to submit.
     - Wrong: POST directly to the form endpoint.
     - Right: drive the click via a headless browser so the real event
       handler runs.
   
   - Bug: a cancellation key in an editor view does the wrong thing.
     - Wrong: call the `cancel()` function in a unit test.
     - Right: feed keystrokes through the real keymap dispatcher and
       observe the resulting view state.
   
   Validation checklist for user-visible fixes:
   
   - [ ] Reproducer is a live or scripted simulation of the real human
         flow (TUI via tmux/PTY, browser via headless driver, terminal
         via `expect` or equivalent), not only a CLI / unit substitute
   - [ ] The reproducer fails on `main` and passes after the fix
   - [ ] A scenario is added to `tests/smoke/scenarios/` and listed in
         `owners` of `tests/smoke/manifest.toml` so future regressions
         are caught by the smoke gate (the manifest is grow-only)
   
   If you are tempted to validate a user-visible fix with only a CLI or
   unit test "because it exercises the same code", stop. The
   `fix-chat-tasks` regression shipped green for exactly this reason: the
   CLI path was already correct and the TUI caller was the broken one.
   Add the human-flow simulation.
   
   ## Cycles
   
   Historical graphs may contain cycle metadata, but completion has no
   `--converged` bypass and failure never authorizes an automatic restart. Model
   each new pass as explicit operator-created work. Every pass that becomes Done
   must independently satisfy the universal manifest, FLIP, eval, and publication
   protocol below.
   
   ## Universal Completion Valve
   
   `wg done` never infers success from process exit, cleanup, or an editable
   worktree. Every Land, Report, and Explore task must first publish an immutable
   completion manifest and receive exact FLIP then eval pass receipts for that
   same manifest and requirements digest.
   
   ### Smoke Gate
   
   If `tests/smoke/manifest.toml` owns scenarios for the task, those live scenarios
   are mandatory validation. Run them against the candidate binary before review,
   capture their exact output as immutable validation evidence, and repair every
   failure. Exit 77 is a loud environmental skip, not a semantic pass. Never
   substitute a unit test merely because it exercises similar code, and never omit
   the owned smoke evidence from the manifest.
   
   A worker agent assigned to `<task-id>` follows this sequence:
   
   1. **Check messages and log progress**:
      ```
      wg msg read <task-id> --agent $WG_AGENT_ID
      wg log <task-id> "Starting implementation..."
      ```
   
   2. **Work and validate.** Run the declared tests/checks. Save their output as a
      validation log. Stage only your files; never use `git add -A` or `git add .`.
      A Land task must commit its candidate. Do not push or merge the root checkout
      yourself.
   
   3. **Snapshot outputs and evidence.** Report/Explore outputs become immutable
      object references. Validation evidence is mandatory for every contract:
      ```
      wg completion-object report.md --media-type text/markdown > output-ref.json
      wg completion-object validation.log --media-type text/plain \
          --evidence-kind validation > evidence-ref.json
      printf '%s\n' 'Implemented and validated the requested result.' > summary.txt
      ```
   
   4. **Build the exact task-bound manifest.** For Report/Explore:
      ```
      wg completion-manifest <task-id> --summary summary.txt \
          --output-ref output-ref.json --evidence-ref evidence-ref.json > manifest.json
      ```
      For Land, first merge current local `main` into the worker branch, resolve
      conflicts in this same worktree, rerun validation, and commit. Then use:
      ```
      wg completion-manifest <task-id> --summary summary.txt --git \
          --evidence-ref evidence-ref.json > manifest.json
      ```
      The command supplies the exact task, generation, contract, requirements
      digest, Git tree/diff identity, and summary digest. Do not hand-invent them.
   
   5. **Submit for mandatory review**:
      ```
      wg submit <task-id> --manifest manifest.json --summary summary.txt
      ```
      FLIP always runs before eval. A rejection returns findings to this worker;
      repair, revalidate, rebuild the manifest, and resubmit. Missing evidence is
      `IncompleteEvidence`; reviewer/provider failure is `Unavailable`. Neither
      creates a replacement source worker or authorizes fallback.
   
   6. **Publish and derive Done.** After exact review passes:
      ```
      wg land <task-id>   # Land only: short-lock compare-and-fast-forward
      wg done <task-id>   # every contract: re-verifies review + publication
      ```
      If `wg land` reports moved main or a conflict, integrate current main in this
      same worker, revalidate, rebuild, and resubmit. Report/Explore skip `wg land`
      but never skip review. Check messages once more before `wg done`.
   
   7. **Visible non-success outcomes**:
      ```
      wg fail <task-id> --reason "genuine blocker after attempting the work"
      wg wait <task-id> --until <condition>
      ```
      A failed or dead attempt remains visibly failed and blocks dependents until
      an operator explicitly retries it. There is no hidden retry, source
      replacement, legacy finalizer, evaluation child task, or cleanup-gated Done.
   
   ### Anti-pattern: Explain-and-Bail
   
   DO NOT: read a task → write an explanation of why it's hard →
   `wg fail`.
   
   DO: read the task → attempt the work → if genuinely stuck after
   trying, `wg fail` with what you tried.
   
   The system has retry logic and model escalation. A failed attempt with
   partial progress is more valuable than a long explanation of why you
   didn't try.
   
   ### Decompose vs implement
   
   Fanout is a tool, not a default.
   
   **Stay inline (default)** when:
   - Task is straightforward, even if it touches multiple files
     sequentially
   - Each step depends on the previous
   - The task is hard but single-scope — difficulty alone is NOT a reason
     to decompose
   
   **Fan out** when:
   - 3+ independent files / components need changes that can genuinely
     run in parallel
   - You hit context pressure (re-reading files, losing track of changes)
   - Natural parallelism exists (e.g., 3 separate test files, N
     independent modules)
   
   When you decompose, every parallel join MUST have an integrator task
   (`wg add 'Integrate' --after part-a,part-b`). Never leave parallel
   work unmerged.
   
   ### Same files = sequential edges
   
   NEVER parallelize tasks that modify the same files — one will
   overwrite the other. When unsure, default to pipeline.
   
   ## Git Hygiene (Shared Repo Rules)
   
   Worker agents share a working tree (or worktrees off the same repo).
   
   - **Surgical staging only.** NEVER use `git add -A` or `git add .`.
     Always list specific files: `git add src/foo.rs src/bar.rs`.
   - **Verify before committing.** Run
     `git diff --cached --name-only` — every file must be one YOU
     modified for YOUR task. Unstage others' files with
     `git restore --staged <file>`.
   - **Commit early, commit often.** Don't accumulate large uncommitted
     deltas.
   - **NEVER stash.** Do not run `git stash`. If you see uncommitted
     changes from another agent, leave them alone.
   - **NEVER force push.** No `git push --force`.
   - **Don't touch others' changes.** If `git status` shows files you
     didn't modify, do not stage, commit, stash, or reset them.
   - **Handle locks gracefully.** `.git/index.lock` or cargo target
     locks mean another agent is working. Wait 2-3 seconds and retry.
     Don't delete lock files.
   
   ## Worktree Isolation (Worker Agents)
   
   A worker agent runs inside a WG-managed worktree. Its working
   directory is already isolated.
   
   NEVER use the `EnterWorktree` or `ExitWorktree` tools. Using them will:
   
   1. Create a SECOND worktree in `.claude/worktrees/`, abandoning this
      one
   2. Switch the session CWD away from the WG branch
   3. Cause ALL commits to go to the wrong branch
   4. Result in work being LOST — the merge-back will find no commits
   
   If you see those tools available, ignore them. WG already
   provides full git isolation.
   
   ### Prior WIP from a previous attempt
   
   A worktree may contain prior work-in-progress from an earlier agent
   attempt (rate-limit, crash, or signal-induced exit, then `wg retry`).
   **Before starting fresh, inspect what's already there**:
   
   - `git status` — uncommitted changes (the prior agent's in-flight
     edits)
   - `git log --oneline main..HEAD` — commits the prior agent made on
     this branch
   - `git diff main...HEAD` — full delta vs `main`
   
   If prior work is present and on-track, **continue from where it left
   off** rather than redoing it. If it's broken or wrong, commit a clean
   reset and start over from there. Either way, do not blindly overwrite
   the prior agent's commits — they may contain valuable progress.
   
   ## Exec Modes
   
   Workers run with an `exec-mode` that limits available tools:
   
   - `full` (default) — all tools (read / write / shell / web)
   - `light` — read-only tools; cannot write files
   - `bare` — wg CLI only (coordination tasks)
   - `shell` — no LLM; the task runs a shell command (set with `--exec`)
   
   `exec-mode` is set at task creation (`wg add ... --exec-mode <mode>`)
   and is exposed to the worker via `$WG_EXEC_MODE`. Workers in `light`
   mode that try to write files will fail.
   
   ## Environment Variables
   
   - `$WG_TASK_ID` — the task you are working on
   - `$WG_AGENT_ID` — your unique agent identifier
   - `$WG_EXECUTOR_TYPE` — handler kind (`claude`, `codex`, `nex`,
     `shell`, ...)
   - `$WG_MODEL` — the resolved model spec
   - `$WG_TIER` — your quality tier (fast, standard, premium)
   - `$WG_EXEC_MODE` — exec mode for this task (`full` / `light` / `bare`
     / `shell`)
   - `$WG_USER` — current user identity
   - `$WG_WORKTREE_PATH` / `$WG_BRANCH` / `$WG_PROJECT_ROOT` /
     `$WG_WORKTREE_ACTIVE` — set when worktree isolation is active; the
     spawn wrapper uses these to detect worktree escape
   
   Tiers control capability and cost: **fast** for triage / routing /
   compaction, **standard** for typical implementation, **premium** for
   complex reasoning, verification, evolution.
   
   ## Where Project-Specific Rules Live
   
   - `CLAUDE.md` (read by Claude Code) and `AGENTS.md` (read by Codex CLI)
     at the repo root — project-specific conventions, smoke gate scope,
     glossary. These two files MUST be kept in lock-step; any drift
     between them is a bug, not an intentional difference. Both should
     be layer-2-only (project specifics) and point at this guide for the
     universal contract.
   - `docs/designs/` and `docs/research/` (WG repo only) —
     contributor docs for people hacking on WG itself; not
     required reading for users
   - `wg quickstart` — command cheat sheet for the current binary
   - `wg agent-guide` — this document and caused this outage. Preserve only real lifecycle/ownership fences (e.g. a worker cannot impersonate another worker's terminal receipt or mutate another active attempt's owned runtime); do not retain a generic cross_task_refused policy for graph inspection/coordination. Update task requirements/tests accordingly.

2. #2user2026-08-06T05:25:25.880904275+00:00read

   Operator decision supersedes the narrow-scope option: ordinary WG workers must be able to use the documented WG coordination surface, including `wg show <other-task>`, `wg add`, `wg edit` when task work requires graph construction/quality review, `wg publish`, dependency operations, messages, status, and ready. The current blanket current-task/cross-task refusal contradicts `wg agent-guide` and caused this outage. Preserve real lifecycle/ownership fences (for example, a worker cannot impersonate another worker's terminal receipt or mutate another active attempt's owned runtime); do not retain a generic `cross_task_refused` policy for graph inspection and coordination. Update task requirements and tests accordingly.

3. #3user2026-08-06T05:26:03.177446312+00:00read

   Task description is now authoritative: the old WCC-only cross-task restriction bullet was removed. Implement broad documented graph coordination for workers; keep only runtime/terminal/admin fences.

4. #4user2026-08-06T06:13:47.109539043+00:00read

   New incident evidence is now in the authoritative description. Agent-8 was falsely reaped at 06:08 despite a verified-live wrapper and active Pi output: heartbeat-watch had exited after one IPC response failure; triage then used the Claude translator on Pi raw_stream and missed fresh liveness. pi-stream-observe also retained exited Pi children/wrappers and repeatedly processed 400–700MB streams. Fix heartbeat retry, Pi-native incremental liveness, observer follow-PID exit, and arbitrary JSON IPC response serialization (`#[serde(flatten)]` currently errors on arrays). Preserve your existing WIP; operator raised heartbeat_timeout 5→60 temporarily and will revert after validation.

5. #5fix-autonomous-batch-plumbing2026-08-06T06:14:33.902224951+00:00read

   Acknowledged all operator updates: I will restore the documented worker graph-coordination surface broadly while retaining terminal/runtime/ownership/completion/admin fences, and align tests plus agent-guide behavior.

6. #6user2026-08-06T06:42:35.102548276+00:00read

   Architectural acceptance, not a patch pile: encode and test the common failure model across this incident. Required invariants are (1) transport timeout/response loss never changes the meaning of a durable idempotent operation; reconcile by request identity, (2) advisory heartbeat loss alone cannot revoke an exact live process with fresh normalized stream progress, (3) provider-specific streams normalize before shared liveness policy, (4) supervised helpers cannot die permanently on one transient error or outlive a terminal child, (5) synthetic system plumbing has its own idempotent terminal transition rather than impersonating a user Land completion, and (6) every zero-ready state explains an actionable root blocker. The installed-binary smoke must inject failures at these seams and complete a quality-gated batch with zero operator commands. Refactor toward these invariants rather than accumulating special cases.

7. #7fix-autonomous-batch-plumbing2026-08-06T07:28:56.279443841+00:00read

   Acknowledged the explicit broad-surface requirement: implementation now brokers cross-task show/context plus list/ready/status, add/edit/contract/publish, dependency operations, and cross-task messages for every normal worker; no WCC-only authorization remains.

8. #8fix-autonomous-batch-plumbing2026-08-06T07:28:56.528040612+00:00read

   Acknowledged the fence requirement: terminal receipts stay current-attempt-only, edits to another active task are refused, completion bypasses remain refused, and secret/service/lifecycle administration has explicit denials.

9. #9user2026-08-06T07:54:33.103781428+00:00read

   New incident for the core robustness fix: design agent-11 completed normally (`stream.jsonl` Result success=true; raw tail ends stop/agent_end) but wrapper classified it as provider-overloaded and failed it. `wg classify-failure` matched the phrase "daemon may be overloaded" from a WG IPC tool-result error in Pi raw output. Provider failure classification must consume only structured provider error fields/events, never arbitrary assistant/tool-result/source text. Add a RED regression proving a successful Pi turn containing that phrase or an IPC tool error cannot become provider-overloaded. This joins the arbitrary-array IPC response bug that prevented the worker from reading the operator correction message.

10. #10fix-autonomous-batch-plumbing2026-08-06T07:56:28.474650411+00:00read

    Implementation committed as 7ce753d1. Installed-binary autonomous batch and normal-worker broker smokes both PASS; typed Plumbed assignment receipts are terminal/idempotent across parent lifecycle advancement, and restart reconciliation never touches failed semantic source tasks.

11. #11user2026-08-06T09:25:52.083036437+00:00delivered

    Release-blocking mixed-version/schema regression: the candidate introduced serialized CompletionContract::Plumbing and CompletionDisposition::Plumbed, then publish/reconciliation wrote those values into 45 graph tasks while the running service/TUI and subsequently reinstalled main binary only knew land/deliver/report/explore. Every old client failed to load the entire graph (`unknown variant plumbing`), making all user tasks invisible. I emergency-migrated those 45 records under graph.lock to the wire-compatible existing report/reported representation; backup is `.wg/graph.jsonl.visibility-repair-20260806T092440Z.bak`. Amend the implementation: do not emit a new non-backward-compatible enum value. Reuse an old-readable representation plus typed presentation/origin/receipt, or add a truly backward-compatible envelope—but an N-1 binary must load every candidate-written graph. Add a mixed-version installed-flow test: N daemon/TUI stays running while N+1 worker/CLI performs assignment plumbing; graph remains readable and visible to N. Also test tagged and untagged primary user tasks remain visible; tags must never exclude them absent an explicit user filter. Do not re-emit plumbing/plumbed before cutover.

12. #12user2026-08-06T09:26:49.052393840+00:00delivered

    Correction to emergency wire form: old readiness requires an ordinary after-edge predecessor to project Landed, so assignment records are now old-wire `land/landed` (not report/reported), while typed presentation/origin plus the authoritative assignment receipt distinguish plumbing semantically. The fix must keep wire compatibility and make completion/preflight exempt typed assignment plumbing by presentation+origin+receipt, not by a new enum. Mixed-version test must prove both graph readability and parent readiness/completion.

13. #13user2026-08-06T11:59:57.631318665+00:00read

    Recovery context: incompatible writer removed. Commit e5b1ffae replaces Plumbing/Plumbed completion enums with N-1-readable Land/Landed while retaining typed presentation+origin+receipt semantics. Focused completion/query tests and installed autonomous_batch_plumbing smoke pass. Do not reintroduce new wire enum values; validate, submit exact current HEAD, and finish.