design-pi-evaluation — WorksGood live mirror

Metadata

Status	done
Assigned	`agent-899`
Agent identity	`603daf069cd2ae4a26357aa66e8c6124154b648711a0816ffd8b660ee3b23815`
Created	2026-07-26T10:21:54.585536086+00:00
Started	2026-07-26T20:39:41.507512570+00:00
Completed	2026-07-26T21:00:47.629166293+00:00
Tokens	8065463 in / 33725 out

Description

Objective

Produce the implementation-ready design for reliable Pi-first evaluation and FLIP. Explain the recent failure chain using source/history evidence: eager satellites, evaluation of non-executed tasks, admission deferrals charged as failures, attempt/pipeline pinning, and evaluator work sharing worker/build capacity.

Design requirements

Read the relevant Pi documentation and examples completely, following their references, for JSON/RPC structured calls, no-tools operation, extensions, session/context handling, usage/error reporting, and supported read-only tool boundaries. Record the researched API contracts and links in the design; do not infer them from WG's current adapter alone.
Keep two deliberately distinct products:
1. bounded evaluation (the default): a no-tools structured Pi one-shot over a content-addressed evidence bundle, used for routine acceptance checking.
2. deep-readonly FLIP (selective only): an explicitly requested/high-risk latent-intent and counterfactual probe that may inspect the task graph, original intent/conversation, dependency context, artifacts/diff, validation logs, runtime traces, and effective system configuration through an observation-only allowlist. It must never mutate graph/source/config, launch arbitrary network calls, or author as the source agent.
Do not relabel a shallow summary grader as FLIP. Define what extra evidence, system comprehension, counterfactual questions, and cross-component analysis make deep-readonly FLIP genuine.
Specify lazy evaluation creation only after a real source attempt reaches candidate completion and policy selects evaluation. Tasks that were never attempted, never spawned, were admission-deferred, cancelled, skipped, or remain open must create no evaluation work. Consider whether hidden attempt records eventually replace graph satellites; choose the smallest safe first implementation.
Specify a dedicated agency queue/concurrency/admission class: no worktree, no build-heavy slot, no normal worker capacity, attempt-bound exact route, bounded timeout/token budget, idempotent exact-once verdict consumption.
Define advisory default vs selective hard gates, failure semantics, retry/backoff, content-safety framing, provenance, and operator visibility.
Preserve native Codex worker/chat execution unchanged. Define an executor-neutral evaluation interface: Pi is recommended/default, explicit Codex/Claude evaluation adapters remain possible/selectable, and no Pi failure may silently cross-fallback to another executor (or vice versa). Advisory Pi unavailability leaves the source terminal with visible evaluation evidence; a selected hard gate waits explicitly rather than reopening the source.

Produce docs/design-pi-evaluation-plane.md; no production code changes.

Validation

Every recent observed failure maps to a concrete invariant and named regression scenario.
Evidence-bundle and structured-verdict schemas include examples derived from researched Pi API contracts.
Default bounded evaluation and selective deep-readonly FLIP capability/security boundaries are explicit and non-overlapping.
Credential-free Fake-Pi fixtures cover success, malformed output, timeout, route drift, duplicate delivery, and unavailable-executor behavior; no fixture needs provider credentials.
The live validation plan includes the actual terminal/TUI human flow from source-task completion through visible verdict/FLIP evidence, not library-only calls.
Rollout starts disabled, runs advisory canaries, and structurally cannot hard-gate globally before recorded canary success.
Design contains exact file-level implementation seams/ownership, serialization order, migration strategy, and executor-neutral/no-silent-fallback tests.
Native Codex execution retention is an explicit non-regression criterion without expanding direct Codex/Claude task scope.

## Objective
Produce the implementation-ready design for reliable Pi-first evaluation and FLIP. Explain the recent failure chain using source/history evidence: eager satellites, evaluation of non-executed tasks, admission deferrals charged as failures, attempt/pipeline pinning, and evaluator work sharing worker/build capacity.

## Design requirements
- Read the relevant Pi documentation and examples completely, following their references, for JSON/RPC structured calls, no-tools operation, extensions, session/context handling, usage/error reporting, and supported read-only tool boundaries. Record the researched API contracts and links in the design; do not infer them from WG's current adapter alone.
- Keep two deliberately distinct products:
1. `bounded evaluation` (the default): a no-tools structured Pi one-shot over a content-addressed evidence bundle, used for routine acceptance checking.
2. `deep-readonly FLIP` (selective only): an explicitly requested/high-risk latent-intent and counterfactual probe that may inspect the task graph, original intent/conversation, dependency context, artifacts/diff, validation logs, runtime traces, and effective system configuration through an observation-only allowlist. It must never mutate graph/source/config, launch arbitrary network calls, or author as the source agent.
- Do not relabel a shallow summary grader as FLIP. Define what extra evidence, system comprehension, counterfactual questions, and cross-component analysis make deep-readonly FLIP genuine.
- Specify lazy evaluation creation only after a real source attempt reaches candidate completion and policy selects evaluation. Tasks that were never attempted, never spawned, were admission-deferred, cancelled, skipped, or remain open must create no evaluation work. Consider whether hidden attempt records eventually replace graph satellites; choose the smallest safe first implementation.
- Specify a dedicated agency queue/concurrency/admission class: no worktree, no build-heavy slot, no normal worker capacity, attempt-bound exact route, bounded timeout/token budget, idempotent exact-once verdict consumption.
- Define advisory default vs selective hard gates, failure semantics, retry/backoff, content-safety framing, provenance, and operator visibility.
- Preserve native Codex worker/chat execution unchanged. Define an executor-neutral evaluation interface: Pi is recommended/default, explicit Codex/Claude evaluation adapters remain possible/selectable, and no Pi failure may silently cross-fallback to another executor (or vice versa). Advisory Pi unavailability leaves the source terminal with visible evaluation evidence; a selected hard gate waits explicitly rather than reopening the source.

Produce `docs/design-pi-evaluation-plane.md`; no production code changes.

## Validation
- [ ] Every recent observed failure maps to a concrete invariant and named regression scenario.
- [ ] Evidence-bundle and structured-verdict schemas include examples derived from researched Pi API contracts.
- [ ] Default bounded evaluation and selective deep-readonly FLIP capability/security boundaries are explicit and non-overlapping.
- [ ] Credential-free Fake-Pi fixtures cover success, malformed output, timeout, route drift, duplicate delivery, and unavailable-executor behavior; no fixture needs provider credentials.
- [ ] The live validation plan includes the actual terminal/TUI human flow from source-task completion through visible verdict/FLIP evidence, not library-only calls.
- [ ] Rollout starts disabled, runs advisory canaries, and structurally cannot hard-gate globally before recorded canary success.
- [ ] Design contains exact file-level implementation seams/ownership, serialization order, migration strategy, and executor-neutral/no-silent-fallback tests.
- [ ] Native Codex execution retention is an explicit non-regression criterion without expanding direct Codex/Claude task scope.

Depends on

Required by

✉ Messages 2 messages (2 unread)

#1design-simplified-authoritative2026-07-26T11:32:16.366525072+00:00delivered

Ratified lifecycle design is ready at docs/design-simplified-task-lifecycle.md. Evaluation contract in §10: lazy EvaluationRecord/virtual satellites, evidence append-only and exact source-bound, advisory default for optional eval, required gate via AwaitingAcceptance, evaluator infra failure affects only eval record, and no evaluator rescue/reopen authority.

#2design-pi-evaluation2026-07-26T20:40:02.009407288+00:00read

Acknowledged lifecycle-design handoff — I will align the Pi evaluation plane with §10's lazy, append-only, exact source-attempt binding and no rescue/reopen authority.

Log

2026-07-26T16:44:23.625403537+00:00 Reconciliation: stale-claim cleared (was Open, agent: agent-894)
2026-07-26T16:44:24.761422050+00:00 Spawn failed (attempt 4/5): spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi
2026-07-26T16:44:28.632314415+00:00 Reconciliation: stale-claim cleared (was Open, agent: agent-894)
2026-07-26T16:44:29.771664250+00:00 Spawn failed (attempt 5/5): spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi
2026-07-26T16:44:29.771664250+00:00 Circuit breaker tripped: spawn failed 5 times. Last error: spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi. Task marked incomplete for evaluator review.
2026-07-26T16:49:32.538493751+00:00 Spawn circuit breaker reset (cooldown decay).
2026-07-26T16:49:33.160731993+00:00 Spawn failed (attempt 1/5): spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi
2026-07-26T16:49:34.495151226+00:00 Reconciliation: stale-claim cleared (was Open, agent: agent-894)
2026-07-26T16:49:35.626396115+00:00 Spawn failed (attempt 2/5): spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi
2026-07-26T16:49:39.498778070+00:00 Reconciliation: stale-claim cleared (was Open, agent: agent-894)
2026-07-26T16:49:40.641059656+00:00 Spawn failed (attempt 3/5): spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi
2026-07-26T16:49:41.954802316+00:00 Reconciliation: stale-claim cleared (was Open, agent: agent-894)
2026-07-26T16:49:43.093121920+00:00 Spawn failed (attempt 4/5): spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi
2026-07-26T16:49:46.959488340+00:00 Reconciliation: stale-claim cleared (was Open, agent: agent-894)
2026-07-26T16:49:48.103513873+00:00 Spawn failed (attempt 5/5): spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi
2026-07-26T16:49:48.103513873+00:00 Circuit breaker tripped: spawn failed 5 times. Last error: spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi. Task marked incomplete for evaluator review.
2026-07-26T16:54:52.236640941+00:00 Spawn circuit breaker reset (cooldown decay).
2026-07-26T16:54:52.854977158+00:00 Spawn failed (attempt 1/5): spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi
2026-07-26T17:21:32.417334941+00:00 Reconciliation: stale-claim cleared (was Open, agent: agent-894)
2026-07-26T17:21:34.188616578+00:00 Spawn failed (attempt 2/5): spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi
2026-07-26T17:21:35.468309483+00:00 Reconciliation: stale-claim cleared (was Open, agent: agent-894)
2026-07-26T17:21:37.230757855+00:00 Spawn failed (attempt 3/5): spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi
2026-07-26T17:21:40.472512030+00:00 Reconciliation: stale-claim cleared (was Open, agent: agent-894)
2026-07-26T17:21:42.223393382+00:00 Spawn failed (attempt 4/5): spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi
2026-07-26T17:21:43.511194982+00:00 Reconciliation: stale-claim cleared (was Open, agent: agent-894)
2026-07-26T17:21:45.280214836+00:00 Spawn failed (attempt 5/5): spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi
2026-07-26T17:21:45.280214836+00:00 Circuit breaker tripped: spawn failed 5 times. Last error: spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi. Task marked incomplete for evaluator review.
2026-07-26T17:26:48.061660093+00:00 Spawn circuit breaker reset (cooldown decay).
2026-07-26T17:26:48.660826834+00:00 Spawn failed (attempt 1/5): spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi
2026-07-26T17:35:26.949242321+00:00 Reconciliation: stale-claim cleared (was Open, agent: agent-894)
2026-07-26T17:35:28.726818609+00:00 Spawn failed (attempt 2/5): spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi
2026-07-26T17:35:31.958339974+00:00 Reconciliation: stale-claim cleared (was Open, agent: agent-894)
2026-07-26T17:35:33.717197473+00:00 Spawn failed (attempt 3/5): spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi
2026-07-26T17:35:35.035783042+00:00 Reconciliation: stale-claim cleared (was Open, agent: agent-894)
2026-07-26T17:35:36.815081218+00:00 Spawn failed (attempt 4/5): spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi
2026-07-26T17:35:40.071549435+00:00 Reconciliation: stale-claim cleared (was Open, agent: agent-894)
2026-07-26T17:35:41.842510299+00:00 Spawn failed (attempt 5/5): spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi
2026-07-26T17:35:41.842510299+00:00 Circuit breaker tripped: spawn failed 5 times. Last error: spawn transaction for agent-894 rolled back (task remains dispatchable; rollback diagnostics: task claim: refused to roll back task 'design-pi-evaluation' because claim ownership changed from agent-894). exec_mode=default, executor=pi. Task marked incomplete for evaluator review.
2026-07-26T17:39:54.807118685+00:00 Task paused
2026-07-26T17:58:42.020569161+00:00 Task resumed
2026-07-26T20:39:29.174003726+00:00 Spawn preparation deferred before launch permit; rollback is complete and no circuit-breaker charge was recorded. Repair the reported checkout/configuration condition and retry: spawn transaction for agent-899 rolled back (task remains dispatchable; rollback diagnostics: complete): failed to establish isolated-worktree observer baseline: observer state already exists; use open_at for restart reconciliation
2026-07-26T20:39:51.035546713+00:00 Spawned by coordinator --executor pi --model openai-codex:gpt-5.6-sol --isolation required-worktree
2026-07-26T20:40:16.053680071+00:00 Starting research/design. FANOUT_DECISION: stay inline — one design artifact integrates tightly coupled lifecycle, Pi API, security, queue, schemas, migration, and rollout decisions; parallel drafts would contend on the same file.
2026-07-26T20:41:44.709049574+00:00 Pi 0.82.0 docs/examples research complete for README, JSON/RPC, SDK, extensions, sessions/session-format, settings/retry, security, environment, models, structured output, read-only tools, tool gates, RPC UI, and sandbox example. Key contract: no native response-schema command; bounded no-tools must strict-parse final assistant JSON; deep read-only needs an exact custom observation extension plus process controls because Pi explicitly has no built-in sandbox.
2026-07-26T20:58:11.267858072+00:00 Drafted docs/design-pi-evaluation-plane.md. It maps all observed failures to invariants/scenarios; records Pi 0.82.0 JSON/RPC/SDK/extensions/session/security contracts with upstream links; specifies lazy record persistence, evidence/verdict schemas, bounded no-tools evaluation, genuine two-phase deep-readonly FLIP, dedicated agency lane, exact-once serialization, no-fallback adapters, migration, rollout, and live terminal/TUI validation.
2026-07-26T20:58:27.534119784+00:00 Committed: 8cce460f — pushed to remote
2026-07-26T20:58:48.506361215+00:00 Validated task-specific criteria: all recent failures map to named invariants/scenarios; Pi 0.82.0 contracts and upstream links are recorded; evidence/verdict examples, disjoint bounded/deep boundaries, six Fake-Pi fixtures, live terminal/TUI flows, disabled→canary rollout guard, exact file seams/serialization/migration, no-silent-fallback, and native Codex retention are present.
2026-07-26T20:58:48.551019529+00:00 Validated docs integrity: relative links exist, external upstream RPC/SDK/structured-output URLs fetched successfully, code fences are balanced, git diff --check and cargo fmt --check pass. Documentation-only change; no production code or build surface changed.
2026-07-26T20:59:03.520744350+00:00 Correction to mechanical checklist: the first case-sensitive marker probe missed the capitalized 'Observation-only allowlist' heading; corrected probe passes all nine task-specific groups. The document itself was unchanged.
2026-07-26T21:00:47.629166523+00:00 Task marked as done