fix-bounded-eval-evidence-sufficiency

Metadata

Status	done
Assigned	`agent-941`
Model	`pi:openai-codex:gpt-5.6-sol`
Created	2026-07-30T21:26:45.535683673+00:00
Started	2026-07-31T06:44:59.137406844+00:00
Completed	2026-07-31T07:05:15.081268052+00:00
Tags	`evaluation`, `safety`, `candidate`, `regression`
Tokens	22959671 in / 42484 out

Description

Current source exposes a structural evidence-sufficiency bug in the dedicated bounded evaluation lane. BoundedCapabilities::no_authority() correctly sets worktree=false, but EvidenceManifest::artifact_diff_summary primarily carries declared artifact strings plus candidate/delta manifest paths, sizes, and hashes; it does not provide a full candidate checkout and may not provide changed file contents. The prompt nevertheless forces a closed pass|fail semantic verdict. A bounded model can therefore reject a valid coding/structural candidate merely because required evidence was absent from its automatically generated manifest. Users/agents must not be required to hand-curate manifest entries. Deep read-only FLIP already materializes the exact immutable candidate commit into its observation-only repository bundle; preserve that boundary and never expose the mutable/live worker worktree.

Implement an explicit evidence-sufficiency contract. Before invoking or consuming a bounded verdict, deterministically decide whether the bounded bundle can support the selected policy/task class. Extend the bounded response/protocol if needed so the evaluator can return insufficient_evidence as a structured non-semantic outcome. Missing/truncated/unreadable required candidate, delta, validation, declared artifact, or source evidence must become EvidenceUnavailable/InsufficientEvidence infrastructure state, never AcceptanceRejected, never source retry/reopen, and never a score averaged with or used to unlock required deep FLIP. Retain the immutable candidate in AwaitingAcceptance/PendingEval and consume only the bounded evaluation retry budget. Surface bounded diagnostics naming safe evidence IDs/categories rather than attacker-controlled text. Permit deterministic rebundling/evidence-locator expansion within budgets or route the decision to the already-required deep read-only FLIP; do not silently waive required evaluation and do not fall back across executors/models.

Keep bounded grading secondary/advisory under the FLIP-first policy. If bounded evidence is sufficient, existing exact candidate/route binding and pass/fail semantics remain. If the task is coding/structural and semantic correctness requires source context beyond bounded evidence, bounded must not be a required rejection authority. Manifest construction remains automatic from immutable candidate/finalization/validation/dependency records. Never mount the source worktree or use mutable main as evaluator input.

Validation

Add a regression candidate whose correctness depends on changed source bytes absent from a deliberately bounded/truncated manifest; assert bounded evaluation records insufficient/unavailable and cannot emit AcceptanceRejected or fail/retry the source.
Assert candidate bytes, source generation/attempt, and AwaitingAcceptance state remain unchanged across bounded infrastructure retries/exhaustion.
Assert deep FLIP receives a read-only materialization of the exact candidate commit and can decide independently.
Assert a genuinely sufficient bounded manifest can still produce candidate-bound advisory/pass/fail evidence without diluting required FLIP.
Run focused evaluation/finalization/lifecycle tests, relevant smoke scenarios, cargo fmt --check, and cargo clippy.
Document the bounded-vs-deep evidence boundary and operator-visible insufficient-evidence state.

Current source exposes a structural evidence-sufficiency bug in the dedicated bounded evaluation lane. `BoundedCapabilities::no_authority()` correctly sets worktree=false, but `EvidenceManifest::artifact_diff_summary` primarily carries declared artifact strings plus candidate/delta manifest paths, sizes, and hashes; it does not provide a full candidate checkout and may not provide changed file contents. The prompt nevertheless forces a closed `pass|fail` semantic verdict. A bounded model can therefore reject a valid coding/structural candidate merely because required evidence was absent from its automatically generated manifest. Users/agents must not be required to hand-curate manifest entries. Deep read-only FLIP already materializes the exact immutable candidate commit into its observation-only repository bundle; preserve that boundary and never expose the mutable/live worker worktree.

Implement an explicit evidence-sufficiency contract. Before invoking or consuming a bounded verdict, deterministically decide whether the bounded bundle can support the selected policy/task class. Extend the bounded response/protocol if needed so the evaluator can return `insufficient_evidence` as a structured non-semantic outcome. Missing/truncated/unreadable required candidate, delta, validation, declared artifact, or source evidence must become EvidenceUnavailable/InsufficientEvidence infrastructure state, never `AcceptanceRejected`, never source retry/reopen, and never a score averaged with or used to unlock required deep FLIP. Retain the immutable candidate in AwaitingAcceptance/PendingEval and consume only the bounded evaluation retry budget. Surface bounded diagnostics naming safe evidence IDs/categories rather than attacker-controlled text. Permit deterministic rebundling/evidence-locator expansion within budgets or route the decision to the already-required deep read-only FLIP; do not silently waive required evaluation and do not fall back across executors/models.

## Validation
- Add a regression candidate whose correctness depends on changed source bytes absent from a deliberately bounded/truncated manifest; assert bounded evaluation records insufficient/unavailable and cannot emit AcceptanceRejected or fail/retry the source.
- Assert candidate bytes, source generation/attempt, and AwaitingAcceptance state remain unchanged across bounded infrastructure retries/exhaustion.
- Assert deep FLIP receives a read-only materialization of the exact candidate commit and can decide independently.
- Assert a genuinely sufficient bounded manifest can still produce candidate-bound advisory/pass/fail evidence without diluting required FLIP.
- Run focused evaluation/finalization/lifecycle tests, relevant smoke scenarios, cargo fmt --check, and cargo clippy.
- Document the bounded-vs-deep evidence boundary and operator-visible insufficient-evidence state.

Depends on

(none)

Required by

done fix-lifecycle-retry-watchdog-fixture

Log

2026-07-30T21:26:45.482271126+00:00 Task paused
2026-07-30T21:26:51.334308765+00:00 Task published
2026-07-30T21:26:52.601772316+00:00 Spawned by coordinator --executor pi --model openai-codex:gpt-5.6-sol --isolation required-worktree
2026-07-30T21:26:58.580408573+00:00 Starting implementation; inspecting prior work and bounded/deep evaluation paths.
2026-07-30T21:30:54.414884932+00:00 FANOUT_DECISION: stay inline — protocol, bundle sufficiency, lifecycle consumption, tests, and docs share the same evaluation schemas/files and must be changed sequentially.
2026-07-30T21:30:54.497284411+00:00 Research complete: bounded manifest currently omits candidate bytes and silently treats missing CAS objects as empty; deep lane materializes exact candidate commit into an isolated bundle. Implementing preflight sufficiency + structured insufficient outcome + advisory-only source-change authority.
2026-07-30T21:46:33.875183068+00:00 Implemented bounded evidence preflight, structured insufficient_evidence, retry-only infrastructure handling, coding advisory/deep authority routing, exact-commit source excerpts, safe diagnostics, and read-only deep materialization. Focused evaluation/finalization tests pass; one unrelated task-lifecycle test hit a missing watchdog fixture under concurrent test execution and will be rerun standalone.
2026-07-31T05:05:30.690017511+00:00 Task reset for retry from in-progress (attempt #1) — reason: Recover preserved implementation after Pi provider overload; continue exact session/worktree and complete validation/checkpoint.
2026-07-31T05:05:34.427691158+00:00 Spawned by coordinator --executor pi --model openai-codex:gpt-5.6-sol --isolation required-worktree
2026-07-31T05:05:51.195974811+00:00 Resumed preserved WIP after provider overload; reviewing implementation/tests, then completing validation and checkpoint.
2026-07-31T05:19:52.309600654+00:00 Focused evaluation/finalization/lazy tests pass. Found pre-existing integration_task_lifecycle fixture failure (missing attempt-bound Pi watchdog state) standalone; published follow-up fix-lifecycle-retry-watchdog-fixture with isolated test scope.
2026-07-31T05:32:40.123929305+00:00 Implemented explicit bounded evidence locators/preflight, schema-v2 insufficient_evidence response, evidence-only retries, safe diagnostics, coding advisory/deep authority selection, exact-commit patch/excerpts, persisted validation receipts, and read-only deep materialization. Added terminal/tmux regression for bounded truncation + independent deep acceptance.
2026-07-31T05:32:58.834854824+00:00 Committed: 51f4b775 — pushed to remote
2026-07-31T05:33:09.792244473+00:00 Validated: focused evaluation/finalization/lazy suites pass (39 tests across dedicated bounded, deep FLIP, candidate finalization, lazy selection plus evaluation unit filters). Regression covers three bounded retries/exhaustion, immutable candidate/source attempt state, and exact read-only deep materialization.
2026-07-31T05:33:09.875189689+00:00 Validated human flow: dedicated_pi_bounded_evaluation_lane, deep_readonly_flip_human_flow, and new bounded_insufficient_deep_authority smoke scenarios pass with candidate binary; terminal/TUI show safe insufficient evidence diagnostics.
2026-07-31T05:33:09.959804300+00:00 Validated: cargo build, cargo fmt --check, cargo clippy, cargo install --path . --locked all pass. Full cargo test exposed unrelated pre-existing parallel profile fixture poisoning (standalone failing profile test passes); integration_task_lifecycle has standalone missing-watchdog fixture failure tracked by published fix-lifecycle-retry-watchdog-fixture.
2026-07-31T06:44:54.288012834+00:00 Task reset for retry from in-progress (attempt #2) — reason: repair candidate wgcid:v1:blake3:b0d54a3edb3007b5badf21fa3283d7873c448e9cb46d7a70da8664c2c83f8edc after merge.target_moved
2026-07-31T06:45:00.581326137+00:00 Spawned by coordinator --executor pi --model openai-codex:gpt-5.6-sol --isolation required-worktree
2026-07-31T06:45:14.817968705+00:00 Resuming prior implementation; inspecting committed delta and validation status
2026-07-31T06:51:45.176172653+00:00 Merged current main after target moved; focused bounded/deep/finalization suites pass. Strengthened exact-candidate regression by advancing mutable main before deep materialization.
2026-07-31T07:00:31.094892230+00:00 Validated after main integration: bounded/deep/finalization integration suites (24 tests), evaluation unit tests (4), eval lifecycle unit tests (33), cargo build, fmt --check, and cargo clippy pass; three owned candidate-binary terminal/tmux smoke flows pass. Full cargo test reached 3037 pass/36 ignored but hit the known parallel profile HOME-fixture poisoning (first failure passes standalone).
2026-07-31T07:00:31.184918005+00:00 Committed main integration and strengthened exact-candidate proof: 230f6eae + 1ece05fa; pushed to remote
2026-07-31T07:04:11.563635020+00:00 Validated: global binaries refreshed with cargo install --path . --locked; branch is clean and contains current main
2026-07-31T07:05:15.081268262+00:00 Task marked as done