← graph

impl-adaptive-parallelism-controller

Impl: adaptive parallelism + cost/time-budget controller (writes max_agents)

Metadata

Statusabandoned ‖ paused
Created2026-07-25T20:57:11.933950303+00:00
Tagscontroller, parallelism, dispatch, budget
Failure reasonSuperseded by the operator-published post-lifecycle synthesis: deterministic convergence is service-owned; no supervisor persona, adaptive outage controller, or composition authority remains.

Description

Objective

Build the adaptive-parallelism + cost/time-budget controller — the control loop that turns the T2 telemetry signal into the max_agents knob. It replaces the binary "spawn / global-pause" response to rate pressure with a graduated knob, while leaving the existing global-outage breaker (src/commands/service/zero_output.rs) as a hard last-resort floor.

Context: docs/studies/adaptive-parallelism-budget-design.md (the full design — read it; this task implements §4-§7). docs/studies/roadmap-rate-limit-and-distribution.md §1/§4(T4). PREREQUISITES: T1 (the authority fix — the controller writes CoordinatorState.runtime_max_agents, which must survive a flagless reload) and T2 (the telemetry FailureSignal + ProviderHealth.cooled_until_ms the controller thresholds on). This task implements BOTH the rate-only spark (study §11 phase 2) AND the budget model (phase 3) as one cohesive controller, since they share the same module/loop/CLI.

Boundary (study §9 — peer of the supervisor, never cross)

The controller NEVER resets tasks; the supervisor NEVER touches max_agents. The controller writes ONLY CoordinatorState.runtime_max_agents; the existing slots_available = max_agents - alive_count math (src/commands/service/coordinator.rs:4850) does the rest. The one-way composition contract with the supervisor (supervisor reads effective_max_agents to politely delay a reset burst; controller reads the supervisor reset log to discount dumb-failures from its throughput denominator) is WIRED + smoke-tested in T5, but the read-method stubs can live here.

What to build

  1. New module src/budget/mod.rs:
    • The knob: effective_max_agents ∈ [floor, ceiling] is the ONLY thing the controller writes.
    • Policy (study §4.2): subtractive-down (rate guardrail: sustained-429 rate ≥10% OR provider-wide quorum => E - step_down where step_down = max(1, ceil(E/4)); spend guardrail; request-rate guardrail) and additive-up (ALL of: busy-fraction ≥0.95, sustained-429 ≤1%, spend ≤0.8x budget, cooldown elapsed, E < min(ceiling, spend_cap, rpm_cap) => E + step_up where step_up = 1). One cut per control interval, never two.
    • Hysteresis (study §4.3/§5.1): busy-fraction band (never cut because slots are empty) + cooldown band (cooldown=90s cut-to-grow, cut_lockout=30s cut-to-cut). Prove bounded-rate by construction (study §5.1).
    • Sustained-vs-blip (study §3.1): a single 429 is a blip; ≥sustain_count (3) rate-limit signals within W (5 min) OR a quorum across ≥2 distinct agents is sustained. Asymmetry: slow to add, faster to cut.
    • Cost/time budget model (study §6): projected_spend_per_hour(E) = E x (c-bar / d-bar_hours); spend_cap and rpm_cap derived from [budget] config + model-registry pricing + OpenRouter free-tier defaults (50 req/day <10 credits, 1000/day ≥10 credits, 20 RPM). Free models: only rpm_cap binds.
    • Safety envelope (study §7): floor (default 1), ceiling (default = static [coordinator].max_agents), clamp after every move. Kill-switches: budget.enabled=false/pause=true freezes; credit-exhausted (402) hard-stop sets effective_max_agents = floor/0 until a non-zero balance confirmed; daily-wall (usd_per_day) hard-stop. Human pin wg budget pin <N> (recorded with reason + TTL, default 1h) + unpin.
    • Composition with the global breaker (study §7.1): the breaker OVERRIDES the controller when it trips; a breaker trip is treated as a forced cut (next eval starts from reduced E, cooldown armed).
    • Persisted state: dir/service/budget_state.json — effective_max_agents, floor, ceiling, last_up_at, last_down_at, trailing window counters, budget, kill_switch, pin, moves_24h. Survives daemon restart (so a restart does not reset hysteresis and immediately re-grow into a still-rate-limiting provider).
  2. Writes the authority field from T1: the controller computes effective_max_agents and writes it to CoordinatorState.runtime_max_agents (which T1 made survive a flagless reload). Reads current dispatch state (alive_count, effective_max_agents) from src/commands/service/coordinator.rs:4815,4850.
  3. Config (src/config.rs): new [budget] section — enabled, usd_per_hour, usd_per_day, and [[budget.limits]] per-model (model, rpm, usd_per_hour). Additive to existing config — no behavior change when absent (controller stays at static max_agents).
  4. Daemon-loop integration (src/commands/service/mod.rs): a parallel control_interval (default 60s) timer that evaluates the controller, reading the T2 rolling window (NOT the instantaneous state). Decoupled from the coordinator spawn tick (spawning is fast/cheap; parallelism decisions must be slow).
  5. CLI (new src/commands/budget_cmd.rs, register in src/commands/mod.rs): wg budget {status, pause, pin <N>, unpin}. status prints JSON + human (the budget_state.json fields from study §10).
  6. Metrics (study §10): wg budget status exposes trailing busy-fraction / sustain-429-rate / spend-usd-per-hour / req-per-min + the move log; wg service status/wg spend already exist.

File scope

  • new src/budget/mod.rs
  • src/config.rs ([budget])
  • src/commands/service/mod.rs (daemon-loop control_interval timer; the controller writes CoordinatorState.runtime_max_agents)
  • new src/commands/budget_cmd.rs, src/commands/mod.rs

Implement directly. The policy + budget + CLI are one cohesive controller; do not split into separate tasks.

Validation

  • Unit test: additive-up requires ALL guards (busy≥0.95 AND 429≤1% AND spend≤0.8x budget AND cooldown elapsed AND E<cap); any one failing blocks the grow.
  • Unit test: subtractive-down priority — a sustained-429 rate ≥10% forces a cut EVEN IF busy-fraction is high (rate guardrail wins ties).
  • Unit test: hysteresis — after a down-move, no up-move for cooldown (90s); no second down-move for cut_lockout (30s). The persisted move log proves no oscillation faster than cooldown.
  • Unit test: sustained-vs-blip — 1-2 isolated 429s (blip) => no change; ≥3 in the window OR ≥2 distinct agents (quorum) => cut.
  • Unit test: budget derivation — projected_spend_per_hour(E) and rpm_cap computed correctly from [budget] + model pricing; a free model binds only on rpm_cap.
  • Unit test: kill-switches — credit-exhausted (402) hard-stop sets effective_max_agents to floor/0 and blocks grow until balance confirmed; daily-wall trips on cumulative spend ≥ usd_per_day.
  • Unit test: the controller writes CoordinatorState.runtime_max_agents AND it survives a flagless reload (this is the T1 fix the controller depends on — assert it end-to-end).
  • Unit test: global-outage breaker overrides the controller; a breaker trip is treated as a forced cut with cooldown armed.
  • Unit test: dir/service/budget_state.json round-trips across a simulated daemon restart (hysteresis not reset).
  • wg budget status prints the trailing window + move log; wg budget pin 3 / unpin round-trips with reason + TTL.
  • cargo build + cargo test pass; cargo fmt --check + cargo clippy clean.
  • No model pin; routes through the active zai profile.

Depends on

Required by

Log