← graph

fix-low-score-eval-gate

Fail closed on below-threshold evaluation verdicts

Metadata

Statusdone
Assignedagent-793
Modelpi:zai:glm-5.2
Created2026-07-25T07:49:09.630710105+00:00
Started2026-07-25T20:00:42.583847031+00:00
Completed2026-07-25T20:37:00.769150342+00:00
Tagsevaluation, lifecycle, regression
Tokens32202849 in / 60149 out

Description

Defect

WG can consume a durable evaluation verdict below eval_gate_threshold and transition the source from PendingEval to Done. The current code intentionally treats some PendingEval sources as advisory: with eval_gate_all=false and no parsed structural deliverables, eval_lifecycle::reconcile_durable_verdicts makes hard_reject=false even for score 0.20. FLIP is linked but excluded from the source outcome, and the default FLIP verification threshold is disabled. This makes FLIP 0.18 + evaluator 0.20 appear to pass.

The lifecycle/status contract is contradictory: wg done puts any task with a live evaluator into PendingEval, its documentation says promotion requires score >= threshold, but reconciliation can promote low advisory scores. Satellite Done (evaluation executed) is also presented ambiguously as evaluation passed.

Requirements

  • A source that is presented as evaluation-gated must never transition to Done when any required gate verdict is below its effective threshold. Low scores may enter bounded in-place rescue/retry or terminal failure, but never success.
  • Make advisory evaluation structurally distinct. Either advisory tasks never enter PendingEval, or every PendingEval is a real hard gate. Status/UI/CLI wording must not call evaluator execution a quality pass.
  • Define and enforce the FLIP contribution to the gate. When FLIP is required by the persisted pipeline, a low FLIP verdict must not be silently ignored; use explicit effective thresholds and strictest-required-verdict semantics rather than creating unbounded verification satellites.
  • Preserve handler-first persisted plans, source-attempt/pipeline identity, durable evidence, exactly-once consumption, bounded rescue, heartbeat behavior, and retry alignment.
  • Fail closed on missing, stale, ambiguous, mismatched, malformed, or non-finite verdict evidence. Do not scorelessly promote a source.
  • Show effective gate applicability, evaluator threshold, FLIP threshold/policy, and final outcome provenance in wg show/status diagnostics.
  • Resolve contradictory documentation and tests: the existing advisory_low_score_completes_but_gated_score_retries_exact_plan expectation must no longer permit a displayed gate to pass at 0.20. Keep tags inert unless an explicit product decision says otherwise.
  • Preserve historical verdicts as immutable evidence. Do not silently rewrite old Done tasks; provide a deterministic audit/diagnostic path for previously accepted below-threshold results.

Validation

  • Reproduce FLIP=0.18 and evaluator=0.20 at threshold=0.70 through the real durable lifecycle and prove the source cannot become Done.
  • Scores exactly at threshold pass; scores below threshold never pass.
  • A low FLIP with high evaluator and a high FLIP with low evaluator both obey the explicit required-verdict policy.
  • Advisory evaluations are visibly advisory and cannot masquerade as PendingEval gate passes.
  • Bounded rescue reopens the same source attempt lineage without unblocking downstream tasks; exhausted rescue fails terminally.
  • Restart/replay consumes each exact verdict once and cannot promote from stale or mismatched attempt evidence.
  • Global/local config merge and daemon reload use the shown effective thresholds and gate policy.
  • Unit, integration, and credential-free smoke coverage pin the low-score regression; cargo fmt --check and cargo clippy pass.

Depends on

Required by

✉ Messages 5 messages (5 unread)

  1. #1user2026-07-25T07:50:18.696048248+00:00delivered
    Incident detail: the affected Phigaro source entered PendingEval, FLIP scored 0.64, evaluator scored 0.18, and effective threshold was 0.70; reconciler persisted outcome=done. Separate 1.00 scores belonged to the .flip/.evaluate system jobs themselves (successful evaluation-job execution), not Phigaro work. Source gating must exclude all system-task self-evaluations from substitution, averaging, threshold checks, or pass provenance; only exact attempt-bound source verdicts may decide the source outcome. Add a regression proving system-task 1.00 scores cannot mask source FLIP/eval failures.
  2. #2user2026-07-25T07:50:33.575199014+00:00delivered
    Second reproduction: the replacement/retry source received evaluator score 0.12 and was again promoted to Done. Treat this as a repeated false-pass across rescue/replacement lineage, not a one-off verdict mismatch. Add a two-attempt regression: original below threshold, replacement below threshold; neither may become Done or unblock dependents, and retry/source-attempt identity must remain exact.
  3. #3fix-low-score-eval-gate2026-07-25T07:52:50.543978085+00:00read
    Acknowledged incident detail — I will enforce exact attempt-bound source FLIP/evaluator verdicts, exclude system-task self-evaluations, and add the 1.00 masking regression.
  4. #4fix-low-score-eval-gate2026-07-25T07:52:50.575776201+00:00read
    Acknowledged repeated replacement false-pass — I will add a two-attempt low-score regression proving neither attempt completes or unblocks dependents and lineage remains exact.
  5. #5fix-low-score-eval-gate2026-07-25T09:09:10.643002545+00:00read
    Acknowledged incident requirements — I will enforce exact attempt-bound evaluator/FLIP evidence, exclude system-task execution scores, and pin repeated low-score rescue lineage in durable lifecycle tests.

Log