study-rate-limit — WorksGood live mirror

Metadata

Status	done
Assigned	`agent-738`
Created	2026-07-25T13:23:45.757563955+00:00
Started	2026-07-25T14:08:28.994310343+00:00
Completed	2026-07-25T14:20:23.954056145+00:00
Tags	`research`, `design`, `telemetry`, `openrouter`
Tokens	3951420 in / 23734 out

Description

Objective

Study how to detect rate-limit / quota / cost-exhaustion failures by parsing the ACTUAL outputs of failed pi-based (LLM) requests, and catalog OpenRouter's real billing/rate-limit surface (free-endpoint requests/day caps, how limits are represented, exact error messages). Output a detector spec + a parsed-failure signal the supervisor and adaptive-parallelism controller can consume.

Research questions

Failure taxonomy: enumerate the distinct failure shapes pi/openrouter returns — HTTP 402/429, rate-limit headers (Retry-After, X-RateLimit-*), quota/credit-exhausted bodies, 5xx, timeouts, provider-specific messages. For each: how to detect from the raw response/stream.
Where in WG do these surface today? Survey pi-handler stream parsing (src/stream_event.rs, src/graph.rs parse_token_usage, the spawn wrapper in src/commands/spawn/execution.rs), agent exit logs, raw_stream.jsonl. What is currently captured vs lost.
OpenRouter specifics: requests/day limits on free endpoints (which models), how the limit is communicated (headers vs body vs docs), credit/balance errors, 429 vs 402 distinction, keyed vs keyless. Use live docs/API.
Signal design: a normalized "failure reason" classification (rate-limit / credit-exhausted / auth / transient / hard) emitted from parsed output, with confidence, that the controller can threshold on.
Telemetry persistence: where the signal is stored so the controller has history (rolling window of recent failures per provider/model).

Inputs

src/stream_event.rs, src/commands/spawn/execution.rs (pi wrapper), src/graph.rs (usage parsing)
OpenRouter docs (requests/day, rate limits, errors) — web research
Recent real failures in this graph's logs as calibration examples

Deliverable

docs/studies/ratelimit-cost-telemetry-design.md

Validation

Design doc committed at docs/studies/ratelimit-cost-telemetry-design.md
Concrete taxonomy of pi/openrouter failure outputs with detection rules (string/header/code)
Documents OpenRouter free-endpoint requests/day limits and how they are represented, with citations
Specifies a normalized failure-reason signal + where it is emitted and stored
Maps to the exact WG source locations that would parse/emit it

## Objective
Study how to detect rate-limit / quota / cost-exhaustion failures by parsing the ACTUAL outputs of failed pi-based (LLM) requests, and catalog OpenRouter's real billing/rate-limit surface (free-endpoint requests/day caps, how limits are represented, exact error messages). Output a detector spec + a parsed-failure signal the supervisor and adaptive-parallelism controller can consume.

## Research questions
- Failure taxonomy: enumerate the distinct failure shapes pi/openrouter returns — HTTP 402/429, rate-limit headers (Retry-After, X-RateLimit-*), quota/credit-exhausted bodies, 5xx, timeouts, provider-specific messages. For each: how to detect from the raw response/stream.
- Where in WG do these surface today? Survey pi-handler stream parsing (src/stream_event.rs, src/graph.rs parse_token_usage, the spawn wrapper in src/commands/spawn/execution.rs), agent exit logs, raw_stream.jsonl. What is currently captured vs lost.
- OpenRouter specifics: requests/day limits on free endpoints (which models), how the limit is communicated (headers vs body vs docs), credit/balance errors, 429 vs 402 distinction, keyed vs keyless. Use live docs/API.
- Signal design: a normalized "failure reason" classification (rate-limit / credit-exhausted / auth / transient / hard) emitted from parsed output, with confidence, that the controller can threshold on.
- Telemetry persistence: where the signal is stored so the controller has history (rolling window of recent failures per provider/model).

## Inputs
- src/stream_event.rs, src/commands/spawn/execution.rs (pi wrapper), src/graph.rs (usage parsing)
- OpenRouter docs (requests/day, rate limits, errors) — web research
- Recent real failures in this graph's logs as calibration examples

## Deliverable
docs/studies/ratelimit-cost-telemetry-design.md

## Validation
- [ ] Design doc committed at docs/studies/ratelimit-cost-telemetry-design.md
- [ ] Concrete taxonomy of pi/openrouter failure outputs with detection rules (string/header/code)
- [ ] Documents OpenRouter free-endpoint requests/day limits and how they are represented, with citations
- [ ] Specifies a normalized failure-reason signal + where it is emitted and stored
- [ ] Maps to the exact WG source locations that would parse/emit it

Depends on

(none)

Required by

Log

2026-07-25T13:23:45.747497519+00:00 Task paused
2026-07-25T13:24:22.932806623+00:00 Task published
2026-07-25T14:08:29.065123399+00:00 Spawned by coordinator --executor pi --model zai:glm-5.2 --isolation required-worktree
2026-07-25T14:08:47.601399379+00:00 Starting research: surveying pi stream parsing (stream_event.rs, graph.rs, execution.rs), then OpenRouter docs
2026-07-25T14:15:47.499909010+00:00 Research complete: surveyed FailureClass enum (graph.rs:129), classify_from_raw_stream (raw_stream_classifier.rs), native openai_client error parsing (max_retries_for_status, parse_retry_after_oai, parse_openrouter_provider_error), pi_handler error events, SessionCostTracking. Fetched canonical OpenRouter limits + errors docs. Writing design doc.
2026-07-25T14:20:02.804612276+00:00 Committed: ecddd55e — pushed to remote (docs/studies/ratelimit-cost-telemetry-design.md)
2026-07-25T14:20:23.954058535+00:00 Task marked as done; scheduled evaluator is advisory evidence only (execution is not a quality pass)