← graph

impl-rate-limit-telemetry

Impl: rate-limit/cost telemetry detector + rolling window (shared substrate)

Metadata

Statusdone
Assignedagent-915
Created2026-07-25T20:57:03.399838970+00:00
Started2026-07-27T15:53:50.515123338+00:00
Completed2026-07-27T16:48:16.009548677+00:00
Tagstelemetry, failure-classification, openrouter, pi
Tokens28222877 in / 51191 out

Description

Objective

Build the shared failure-detection substrate that BOTH the supervisor (T3) and the adaptive-parallelism controller (T4) consume. The headline finding (docs/studies/ratelimit-cost-telemetry-design.md §4.2): the pi CLI handler's error event shape is NEVER classified today — pi emits {"type":"error",...} / {"type":"response","success":false,...} but translate_pi_stream drops them and classify_from_raw_stream never looks for them, so OpenRouter 429/402/overload failures fall through to AgentExitNonzero. This task closes that gap and adds the persisted rolling telemetry window.

Context: docs/studies/ratelimit-cost-telemetry-design.md §4 (where failures surface today + the gap), §5 (detector spec), §6 (telemetry persistence), §7 (exact code map). docs/studies/roadmap-rate-limit-and-distribution.md §2.1.

What to build

  1. Normalized signal. Add FailureReason enum + FailureSignal struct to src/graph.rs (alongside the existing FailureClass at :129). FailureClass stays the retry-policy axis; FailureReason is the provider-telemetry axis. Full field set + confidence ladder (1.0 status+error_type, 0.8 status-only, 0.5 substring, 0.2 exit-only) per study §5.1.
  2. Close the pi gap. Extend translate_pi_stream (src/stream_event.rs:464) to FORWARD pi type:"error" and type:"response"(success:false) events as canonical StreamEvent::Error { message, status, error_type } (today the trailing match arm _ => {} drops them). Reuse the field extraction already proven in RpcTurnAccumulator::ingest (src/commands/pi_handler.rs:182-235).
  3. Classifier parse. Teach classify_from_raw_stream (src/commands/spawn/raw_stream_classifier.rs:34) the new pi error-event parse + the 402 arm (today 402 falls through to AgentExitNonzero) + the OpenRouter body-envelope parse, emitting a FailureSignal. Add the substring ladder (credit-exhausted / rate-limit / overloaded / unavailable / auth / timeout) per study §5.2(b), order matters.
  4. Shared parse fn. Extract parse_openrouter_error_envelope(body: &str) -> Option<ParsedProviderError> as a pure fn that BOTH the native executor (src/executor/native/openai_client.rs, which already has the deep parse at :1759 parse_openrouter_provider_error) and the subprocess classifier call. Today they do not share code.
  5. Telemetry persistence. New append-only, size-bounded .wg/service/provider-telemetry.jsonl (one record per FAILED ATTEMPT, keyed by (executor, route-bucket), keep last 1000 records OR 24h, prune on append, atomic append). New src/telemetry/mod.rs module. Add ProviderHealth (with cooled_until_ms — the single field the controller thresholds on) to SessionCostTracking (src/commands/service/mod.rs:~663).
  6. Recording sites. Emit the signal at: src/commands/fail.rs (when recording failure_class), the spawn wrapper src/commands/spawn/execution.rs:~2855-2928 (a new wg record-telemetry --task T --exit-code N --raw-stream $RAW_STREAM mirroring wg classify-failure), and the native executor terminal-retry-exhaustion path (src/executor/native/openai_client.rs).
  7. CLI surfaces. wg classify-failure --json emits the full FailureSignal (not just the kebab); wg recover gains an exact reason=<kebab> filter on the new field (today error~credit is a fragile substring match, src/commands/recover.rs:197).
  8. Mid-stream trap. A pi worker that 429s AFTER the first token exits with the stream truncated, possibly exit 0 — wire the detector into the existing "no operational output" / "agent-no-work" gate (src/commands/spawn/execution.rs:~2880) so a mid-stream rate-limit still produces a RateLimit signal instead of NoOperationalOutput.

File scope

  • src/graph.rs (new enum/struct near :129; field on Task near :502/596)
  • src/stream_event.rs (:464 forward errors)
  • src/commands/spawn/raw_stream_classifier.rs (:34, :197)
  • src/executor/native/openai_client.rs (extract shared parse fn; terminal-retry emit)
  • new src/telemetry/mod.rs
  • src/commands/service/mod.rs (ProviderHealth on SessionCostTracking)
  • src/commands/fail.rs, src/commands/spawn/execution.rs
  • src/commands/classify_failure.rs, src/commands/recover.rs, src/cli.rs

Implement directly. The 8 sub-points above are a build order, not separate tasks.

Validation

  • Unit test: translate_pi_stream forwards a pi {"type":"error","error":"API error 402: Insufficient credits"} line as a StreamEvent::Error (today it is dropped). Test that the canonical stream.jsonl now contains the error.
  • Unit test: classify_from_raw_stream on a pi raw stream containing a 402 body envelope emits FailureReason::CreditExhausted (today: AgentExitNonzero). Use the in-repo calibration fixture at src/executor/native/openai_client.rs:5167 as a second input.
  • Unit test: 429 → RateLimit with retry_after_secs populated; 401 → Auth; 529 → ProviderOverloaded; the substring ladder order is correct.
  • Unit test: shared parse_openrouter_error_envelope is called by BOTH the native path and the subprocess classifier (no duplicated parse logic).
  • Unit test: telemetry window prunes to the last 1000 records / 24h on append; atomic append does not corrupt under concurrent writers.
  • Unit test: ProviderHealth.cooled_until_ms is set from the max of observed Retry-After and the exponential backoff seeded by consecutive rate-limits.
  • wg classify-failure --json on a fixture raw stream prints the full FailureSignal.
  • wg recover --filter reason=credit-exhausted matches exactly (no substring fragility).
  • cargo build + cargo test pass with no regressions; cargo fmt --check + cargo clippy clean.
  • No model pin; routes through the active zai profile.

Depends on

Required by

Log