← graph

implement-pi-stalled

Implement Pi stalled-session watchdog and safe resume

Metadata

Statusdone
Assignedagent-897
Agent identity6841f97a7a8e6cbc213051ce0ffb3c85c92c80cb61fd699428ab3fed6cb6ba94
Created2026-07-26T11:07:29.033736506+00:00
Started2026-07-26T18:36:27.897299742+00:00
Completed2026-07-26T19:24:06.713694593+00:00
Tokens23198430 in / 53225 out

Description

Objective

Implement the amended docs/design-pi-session-watchdog.md protocol. Recover a genuinely stalled or autonomously settled/exited Pi task worker only through evidence-driven, safely continuable same-session recovery inside the same task generation, source attempt, route snapshot, and worktree. Five minutes is a soft observation threshold, never a universal interruption or lifecycle deadline.

Prerequisites and ownership boundary

  • Implement only after implement-authoritative-lifecycle, fix-admission-deferrals, design-pi-stalled, and amend-pi-watchdog. Use the lifecycle kernel/event ledger introduced there; do not add a direct task-status writer or parallel coordinator state path.
  • The watchdog owns Pi worker observation, process/continuation epochs, same-session attestation, and related diagnostics only. It does not own canonical Done, generic dead-agent cleanup, chat-session recovery, admission, evaluation/FLIP, provider-breaker policy, or the long-lived graph-health supervisor.
  • Preserve the generic dead-owner/worktree fix as baseline behavior. impl-supervisor-hard-agent is serialized after this task for shared daemon-loop seams and must remain prohibited from reaping/resuming live Pi attempts.
  • Follow the amended design's ratified file-level seam list. If implementation discovers a required unlisted lifecycle/spawn seam, stop and amend/log the design boundary before editing it.
  • Adaptive thresholds, cross-route fallback, fresh-session recovery, and a generic multi-executor watchdog are explicit non-goals.

Completion and lifecycle authority

  • A Pi lifecycle tool is only an intent when LifecycleKernel durably accepts a current (task, generation, attempt, attempt_fence, process_epoch, toolCallId) receipt under first-terminal-wins. agent_end, agent_settled, stdout/EOF, exit status, silence, elapsed time, files, evaluator opinion, and prompt wording cannot substitute.
  • Accepted current-epoch wg_done is only SuccessIntent; it does not make the task Done. Done requires the exact terminal receipt, current-epoch PiQuiescenceReceipt with current worktree manifest, a candidate checkpoint bound to those receipts, required validation/evaluation receipts, and the authoritative merge/finalization event. Keep lifecycle-kernel/finalizer ownership exactly as design §4.3 specifies.
  • Accepted current-epoch wg_fail and correlated wg_wait disarm the watchdog and follow their own failure/park dispositions without the success candidate path. Old-epoch terminal reports are late evidence only.
  • Implement the explicit lifecycle-kernel reconciliation for RuntimeExit / NoCompletionProtocol: only a persisted, policy-valid Pi continuation authorization may keep a child-process-epoch exit pre-terminal. Do not weaken generic executor failure mapping or suppress failure through observer-side special casing.
  • Time may authorize observation or make safe continuation eligible. It never proves success, failure, abandonment, quiescence, or absence of side effects.

Required observation/action projection

Implement Active | WaitingUser | LongTool | Suspect | HardResumeEligible | NeedsFinalization | Fencing | Resuming | StalledOperatorRequired as watchdog classifications, not canonical task-generation statuses. Apply these distinct actions:

ObservationRequired action
accepted current-epoch wg_donedisarm; quiesce exact process; hand terminal + manifest receipts to candidate validation/evaluation/finalization; never label Done early
accepted current-epoch wg_fail / correlated wg_waitdisarm and follow matching lifecycle disposition; no completion inference
agent_settled without terminal receiptenter nonterminal NeedsFinalization; promptly quiesce/reconcile and issue one bounded same-session completion action without waiting 300s
exact process exit/EOF without terminal receiptenter NeedsFinalization; classify effect ambiguity and, when safe, promptly reopen the exact same session/attempt/route/worktree for that action; exit code chooses neither success nor failure
provider/model request in flight without meaningful outputat 300s append Suspect and perform a read-only probe only; keep the current process/provider operation intact until separate phase/route hard eligibility plus hard grace
advancing token/thinking/tool-call/session/worktree progressstay Active, advance the monotonic progress sequence, cancel stale suspicion; total runtime is irrelevant
valid declared long-tool leasestay LongTool; protect through lease/renewal/hard-cap and use effect-specific expiry behavior
accepted explicit user waitdisarm and park; only matching correlation wakes through normal lifecycle
alive with unknown phase/ownership/effectsuspect/probe, then observe/operator-hold; never guess provider stall or auto-kill
silent/no-terminal without safe same-session/route/process/effect proofStalledOperatorRequired; no fresh session, fallback route, duplicate effect, inferred failure, or automatic new attempt

NeedsFinalization means only “Pi stopped autonomously without completing the WG protocol.” It is neither completion evidence nor a canonical Done/Failed state.

Two static clocks and conservative fencing

  • Persist meaningful progress from native Pi events. meaningful_silence_secs = 300 is exactly the static soft-suspect threshold. At crossing, append evidence and run one read-only probe. Probe traffic, wall-clock runtime, ordinary messages/logs/status polling, and bare process/pipe heartbeats never reset progress. The probe cannot authorize a prompt, TERM/KILL, fence, or lifecycle disposition.
  • Add frozen phase/route hard_resume_after_secs plus separate hard_resume_grace_secs. Initial free/low-QoS provider/TTFT and generation/inter-token hard thresholds MUST be >=900 seconds and may be longer by route policy. Hard grace begins only after that threshold. There is no automatic hard threshold for Unknown, advancing progress, accepted wait, or a valid long tool.
  • At/after hard policy plus grace, automatic fencing still requires one unchanged progress/session/process CAS, a fresh read-only probe with no new evidence, exact PID/start/boot/nonce identity and containment, safe effect classification, no terminal reservation, available finite budget, and exact same-session/route/attempt/worktree attestation. Any failed or unknown check holds rather than killing.
  • Settled/exited states are completion-action eligible from explicit quiescence evidence, not timers. Preserve exact process-epoch fence/reap proof for exited/replaced processes.
  • Keep all thresholds static and configuration-validated. Record route/model/QoS, phase, input size/tokens, TTFT, provider duration, inter-token gaps, tool/worktree progress, suspect/probe results, false suspects, resume latency/success, and possible duplicate inference cost. Telemetry cannot rewrite/decrease policy. Future p99 adaptation requires separate ratification and a safety floor.

Same-session proof, action, process safety, and finite recovery

  • Use (task, generation, attempt, fence, continuation_epoch, process_epoch) plus the design's session/branch and exact handler/provider/model/reasoning/endpoint snapshot proof. Persist before launch and re-attest before resumed tools/provider requests. Missing/mismatched proof fails closed; never create a fresh session.
  • Preserve the worktree lease under the same attempt/fence. CAS only the process sublease/continuation epoch after exact old PID + start/boot/nonce and process group are fenced/terminated/reaped or exact prior exit/reap is proven. No two PIDs/session epochs may own/write concurrently, including PID reuse/restart.
  • Implement durable idempotent probe, hard-eligibility, reserve, fence, TERM/bounded KILL, reap, launch, session-attestation, prompt-marker, and permit actions. Replay produces at most one signal, prompt append, replacement, and budget charge.
  • Use the one versioned neutral stock prompt from design §9.3. It states only a bounded observed fact and absence of a terminal receipt; requires inspection of the durable same Pi session, leased worktree, task contract, candidate state, tests, and receipt summaries; forbids repeating side effects; and requires exactly one wg_done, wg_fail, or correlated wg_wait. Its wording is guidance, never proof.
  • Make prompt append/send idempotent by action ID and (attempt, process_epoch, continuation_epoch, prompt_version/digest). Replay first inspects the exact durable session marker and never sends a second copy for that action; uncertainty holds.
  • Auto-continue only when effect classification proves read-only/idempotent or receipt-backed safety. If a non-idempotent effect may have executed without a durable receipt, hold for operator; prompt wording cannot prevent replay.
  • Retain the ratified finite maximum replacement epochs and reserved elapsed budget. Charge before launch, once, never refund/reset on tick/restart. These bound recovery exposure, not original/progressing task runtime. Exhaustion or ambiguous effects require an audited finite manual grant or explicit lifecycle abort/fail/cancel; do not infer failure. Long-tool boundaries remain clamped/effect-aware, never blind kills.
  • Continuations consume no new-task admission, source retry/generation budget, spawn/provider breaker count, or evaluation satellite/job. Record continuation costs/failures/exact-route errors separately.
  • The continuation input is Pi session input, not wg msg; ordinary messages change no lifecycle, owner, progress clock, or eligibility.
  • Freeze the resolved Pi route/session tuple. Profile changes apply only to future attempts. Exact-route failure remains held/backing off on that route; never fall back to another model/provider/endpoint/executor/session.

First-terminal-wins and race behavior

  • Accepted done/fail/wait/cancel/operator-abort before continuation-epoch CAS cancels pending watchdog actions and prevents launch.
  • Once a new process epoch is fenced in, old-epoch terminal/exit/tool receipts are late evidence only. Only the current epoch may request disposition.
  • Duplicate/contradictory lifecycle tools are consumed exactly once. Late exit cannot overwrite done/fail/park, late done cannot overwrite fail/abort, and diagnostics never terminalize/reopen.

Diagnostics and controls

Expose through real CLI/TUI/service diagnostics: task/generation/attempt/fence; session/branch and frozen route/QoS; process/continuation epoch and exact PID identity; classification/phase; monotonic progress and current silence; 300s soft threshold and probe result; phase hard threshold or none, eligibility and hard-grace state; tool/wait/effect receipts; completion action/prompt marker; finite budget; pending outbox action; exact-route error; and next safe operator action. Implement safe manual resume and abort/fail with stable reason codes.

RED-first permanent validation

Build the credential-free native-format Fake-Pi fixture with stable session/branch, virtual clock, deterministic crash barriers, launch/stdin/tool recording, and receipt-keyed side effects. Write failing tests first and retain them:

  1. soft/hard clocks: 299s active; exactly 300s one soft suspect/read-only probe; no fence at 300s + superseded 180s grace; 899s preserves process; >=900s starts phase hard eligibility/grace; fence only after hard grace and all safety proofs; invalid/missing/Unknown hard policy never fences.
  2. progress/runtime: tokens, thinking, tool-call/session/worktree progress advance sequence/reset stale clocks; a progressing 20-minute run is untouched; probes/messages/logs/status/heartbeats/main-tree visibility do not reset progress.
  3. provider lifecycle: slow TTFT, request-in-flight, response body/inter-token gaps, provider retry, compaction retry, and queued follow-up preserve the current operation before hard policy; agent_end(willRetry=true) is not settled.
  4. NeedsFinalization: agent_settled, zero/nonzero/signal exit, and EOF without terminal receipt promptly classify nonterminal and get one safe same-session prompt without waiting 300s; none becomes Done/Failed. Unsafe exit/effect proof holds.
  5. explicit outcomes/finalization: current done/fail/wait receipts disarm; done requires terminal + quiescence/manifest + candidate + validation/evaluation + authoritative finalization; missing/old-epoch receipts never satisfy it.
  6. eligible partial work: after hard threshold/grace, exact guarded fence/reap resumes the same attested session/route/worktree exactly once, observes prior state, performs a receipt-backed effect once, and reaches explicit disposition; every failed/unknown guard holds.
  7. wait/unknown/tool: accepted wait survives time/restart; only correlation wakes normally; unknown alive silence never auto-kills; valid renewable long tool survives soft/hard times; expired/ambiguous effects use reconcile/hold without replay.
  8. restart/idempotency: crash before/after suspect, probe, hard eligibility/grace, reservation/charge, sublease revoke, TERM/KILL/reap, launch, attestation/permit, prompt marker, effect receipt, and terminal receipt; assert one prompt/action/process/charge/disposition.
  9. race/PID: duplicate ticks and both terminal-vs-epoch-CAS orderings for done/fail/park/cancel/abort; PID reuse/start mismatch and stale wrapper reports; first-terminal-wins and late evidence.
  10. proof/route/budget: session/branch/route mismatch, unavailable exact route, missing attestation, epoch/elapsed exhaustion, and prompt-marker uncertainty hold without fresh session/fallback/inferred failure; finite manual grants charge once.
  11. domain isolation: no continuation path changes admission/source retry/breakers/evaluation, creates another owner/attempt/generation/worktree lease, or duplicates accounting.
  12. dangling effects: every design §7.2 crash window repairs only with read-only/postcondition/receipt proof; ambiguous effect has zero replay.

Add one grow-only permanent smoke scenario owned by implement-pi-stalled that runs the installed binary, real daemon/service/wrapper/diagnostic/operator paths, and Fake-Pi through PTY/tmux/expect—not a library/direct-helper substitute. It must visibly:

  • display production soft 300s and low/free-QoS hard >=900s, then use ordered short test values;
  • show soft Suspect + read-only probe, no old-grace fence, pre-hard hold, and only after hard grace/proofs HardResumeEligible -> Fencing -> Resuming -> Active;
  • show a progressing long run, provider retry/in-flight, wait, valid long tool, and unknown alive silence untouched;
  • show settled and safe exits promptly enter NeedsFinalization and receive exactly one same-session prompt without becoming Done/Failed;
  • prove explicit current-epoch done/fail/wait plus the layered Done tuple;
  • restart around prompt, continuation, and budget boundaries; exercise terminal races, manual resume, and abort/fail; and
  • assert no duplicate prompt/effect/owner/PID/session/route/accounting or replenished budget.

Register it in grow-only tests/smoke/manifest.toml. An optional attended low/free-QoS canary may record telemetry but is never required for credential-free CI and cannot alter policy.

Validation

  • All RED-first Fake-Pi cases fail on pre-change behavior and pass after implementation with deterministic virtual time/native events.
  • The exact observation table and soft-vs-hard clocks are implemented: 300s probes only, no old-grace fence, free/low-QoS hard >=900s plus later grace, and no automatic hard threshold for unknown/progress/wait/valid-long-tool.
  • Settled/exit prompt promptly into nonterminal NeedsFinalization; canonical Done is proven only by terminal + quiescence/manifest + candidate + validation/evaluation + finalization receipts.
  • Same-session/route/worktree proof, exact PID fencing, append-once prompt, restart replay, finite budgets, and race tests prove one current owner/process/action and first-terminal-wins.
  • Ambiguous effects/proof mismatch/exhaustion hold without inferred failure, fallback, fresh session, or duplicate work.
  • The installed-binary PTY human flow covers every listed subcase and is permanently registered with this task as owner.
  • cargo fmt --check, cargo clippy, focused tests, cargo build, cargo test, cargo install --path . --locked, installed-binary validation, and owned smoke pass.

Depends on

Required by

✉ Messages 3 messages (3 unread)

  1. #1user2026-07-26T11:47:30.976324837+00:00delivered
    RED fixture must reproduce: isolated writer silent >5m, falsely marked failed while still alive, writes after failure, exits/abandons, superior worktree artifact stranded. Fixed behavior: no terminal failure while PID/epoch active; suspect/probe; same-session finalization prompt; candidate checkpoint survives; explicit done/fail only.
  2. #2design-pi-stalled2026-07-26T12:11:14.726091933+00:00delivered
    Design committed at docs/design-pi-session-watchdog.md (751b43bb). Key implementation contracts: lifecycle-owned PiContinuationAuthorization/Held pre-terminal mapping; dedicated Pi worker RPC epochs; exact session header/leaf/prefix + frozen route/endpoint attestation; 300s silence + explicit 60s grace; 3 epochs/1800s; fsynced toolCallId intent/end receipts; exact PID/start/boot/nonce + group quiescence; first-terminal CAS; Fake-Pi/PTY matrix and file-level seams in §§14-15.
  3. #3implement-pi-stalled2026-07-26T18:36:35.731209984+00:00read
    Acknowledged queued requirements — I will preserve live PID/epoch authority, implement suspect/probe and exact same-session finalization/continuation, retain candidate checkpoints, and require explicit lifecycle receipts.

Log