← graph

implement-isolated-worktree

Implement isolated-worktree progress observer

Metadata

Statusdone
Assignedagent-892
Agent identity6841f97a7a8e6cbc213051ce0ffb3c85c92c80cb61fd699428ab3fed6cb6ba94
Created2026-07-26T14:46:29.528585841+00:00
Started2026-07-26T15:03:51.570934629+00:00
Completed2026-07-26T16:06:46.114724088+00:00
Tokens37757674 in / 76096 out

Description

Objective

Implement a fail-closed, isolated-worktree activity observer that makes real candidate-content changes visible to the Pi watchdog and operators without turning filesystem churn into lifecycle authority. This task refines and must remain consistent with docs/design-pi-session-watchdog.md, docs/design-simplified-task-lifecycle.md, and the candidate inclusion/finalization contract in docs/design-candidate-finalization-transaction.md.

The observer is an evidence producer. It does not detect completion, mark a task done/failed/waiting, create a candidate, merge, renew a worktree lease, or resume/reopen an attempt. LifecycleKernel remains the only task/attempt/lease writer; the Pi watchdog remains the owner of Pi meaningful-progress classification, suspicion, fencing, and continuation.

Normative decisions and rationale

1. Observe one exact leased attempt worktree

Bind each observer instance before the child receives an execution permit to:

(task_id, generation, attempt_id, attempt_fence,
 worktree_id, worktree_lease_epoch, process_epoch,
 canonical_worktree_root, root_device/file_identity, observer_epoch)
  • Resolve the root from the authoritative current attempt/worktree lease, never from CWD, the integration checkout, main, a branch-name guess, or the agent ID alone.
  • Verify the canonical root and directory identity at attach and every reconciliation. Do not follow a replaced root or a symlink outside it. .git administration may point elsewhere but is control state, never candidate activity.
  • Establish and persist the initial candidate manifest before execution is permitted. Reused/retained work already present at lease transfer is the new attempt's baseline, not new activity.
  • If startup/restart finds no trustworthy prior baseline, reconcile and label it baseline_time_unknown; expose the bytes for preservation but do not advance an activity clock from an invented timestamp.
  • One task may have historical worktrees and attempts. Only the exact current tuple can influence its live watchdog projection. Main-tree writes and other worktrees are always inert.

Rationale: path/branch similarity is not ownership. The lease tuple plus canonical directory identity prevents main-tree visibility, a reused path, or an old observer from becoming progress evidence.

2. Reuse one candidate path classifier

Implement a versioned CandidatePathPolicy shared with, or byte-for-byte compatible with, the candidate finalizer's inclusion policy. Snapshot its digest on the attempt. Classification precedence is:

  1. exact internal/control exclusions (.git administration, WG/agent session/control files, observer journal/state, sockets/devices/FIFOs, escaping paths);
  2. explicit task deliverables/generated-path overrides;
  3. tracked candidate entries (content, Git mode, deletion, symlink target, gitlink identity), except paths explicitly classified as generated by the snapshotted policy;
  4. plausible untracked candidate regular files/symlinks: beneath the root, non-ignored, non-volatile, and not a dependency/cache/build tree; and
  5. volatile/excluded paths such as target/, dependency trees, caches, temp files, WG logs/streams/heartbeats, and configured generated output, with a stable exclusion reason code.

Do not globally exclude extensions such as .log when the path is tracked or explicitly declared a deliverable. Do not allow an ambient config edit to reclassify paths mid-attempt; configuration changes affect future attempts. A task-declared deliverable may include a normally generated path, but internal Git/WG control state remains excluded.

Fingerprint candidate semantics, not metadata:

path bytes + kind + Git mode + content/blob BLAKE3 + size

A candidate-manifest advance may contain create, content-changing write, mode change, symlink-target change, delete, or atomic replacement. Rename is represented deterministically as delete+add unless a stronger content identity is available. Mtime/ctime, open/close, chmod with no Git-mode change, repeated identical bytes, and volatile-only churn do not count.

Rationale: the observer and finalizer must not disagree about what could become source. Content fingerprints prevent repeated touches or same-content rewrites from manufacturing activity.

3. Keep two clocks with different authority

Persist and render these clocks separately:

last_observed_worktree_activity
  = content-confirmed candidate-manifest advance in the exact leased worktree
last_proven_progress
  = the Pi watchdog's receipt/session/provider/token/tool progress sequence

last_proven_progress is the existing watchdog meaningful-progress clock; do not create a competing implementation or advance it from a filesystem event. A receipt-aware write/edit can independently advance both clocks: the fs observer confirms candidate bytes, while the Pi/plugin receipt proves the tool/session phase.

The fixed Pi meaningful-silence default remains 300 seconds. Candidate activity receives only a bounded false-suspicion deferral, snapshotted per attempt:

observed_activity_grace_secs = 120       # default
max_observed_only_extension_secs = 600  # hard per-proof-window cap

proof_deadline = last_proven_progress + meaningful_silence_secs
observed_deadline = min(last_observed_worktree_activity + observed_activity_grace_secs,
                        proof_deadline + max_observed_only_extension_secs)
suspect_at = max(proof_deadline, observed_deadline)

Apply observed_deadline only when a candidate sequence advanced after the current proven-progress sequence. Persist the resulting absolute deadline/cap consumption; duplicate ticks, watcher restart, daemon restart, clock rollback, or repeated observations cannot replenish it. A new receipt-proven Pi progress event may start a new proof window. With no candidate change, suspicion remains exactly at the 300-second proof deadline.

Observed-only activity:

  • may postpone entering Suspect only within the bound above;
  • never resets or aliases last_proven_progress;
  • never extends probe grace, long-tool hard caps, process/continuation epoch leases, continuation elapsed/epoch budgets, or operator holds;
  • never cancels a fence after the watchdog's progress/fence CAS unless accompanied by independently valid receipt-proven progress under the watchdog protocol; and
  • never authorizes success, failure, park, candidate promotion, evaluation, merge, retry, or generation creation.

After the observed-only cap, continuous candidate rewriting remains visible but the normal watchdog suspect/probe/fence policy proceeds. This bounds a malicious source rewrite loop or orphaned compiler while allowing gradual real writing beyond five minutes to avoid a premature false suspect.

Rationale: candidate bytes are useful corroborating evidence, but a generic watcher cannot prove which process wrote them, whether Pi/session work advanced, or whether a side effect is replay-safe.

4. Events are hints; reconciliation is truth

Use a low-latency recursive notify watcher only as a wakeup channel. Every accepted activity record comes from a content rescan/fingerprint comparison.

  • Debounce/coalesce bursts, but do not assume one event equals one write.
  • On create/write/remove/rename, rescan affected candidate paths and recompute the aggregate candidate delta manifest.
  • On watcher overflow/error, mark rescan_required, wake the service, and immediately reconcile. If native watching is unavailable, run poll-only with a visible degraded-health reason.
  • Run a bounded periodic full reconciliation (default 15 seconds), before every watchdog suspicion/fence decision, and at daemon startup. This is the lost-event and network-filesystem safety net.
  • Detect .gitignore, index/tracked-set, submodule/gitlink, and snapshotted policy effects during reconciliation; index churn alone is not candidate activity.
  • Hash symlink text without following it. Refuse escaping/case-colliding/unreadable/special entries and expose a classification hold rather than guessing.
  • For a file changing during a scan, record scan_unstable, do not claim a stable manifest advance from a mixed read, and retry. This observer manifest is diagnostic/watchdog evidence, never a quiescence or checkpoint manifest.
  • Atomic rename/replacement converges by rescanning the parent/final path. A create-and-delete that leaves the candidate manifest unchanged between confirmed snapshots does not manufacture activity.

Persist outside the watched source tree, for example:

.wg/attempts/<attempt-id>/worktree-observer/
  policy.json
  baseline.json
  activity.jsonl      # hash-linked/fsync records
  state.json          # atomic derived projection

Each activity record contains the full source/lease/process/observer tuple, monotonic content_seq, prior/new manifest digests, changed path(s) and class/reason, operation kind, before/after digest and byte delta where available, wall timestamp from the injected clock, and event/reconciliation source. Dedupe by tuple plus prior/new manifest digest. Filesystem mtimes are never reconstructed as progress time.

5. Fence, restart, and late-write behavior

At daemon restart, replay observer state, revalidate the current lease/root/policy, and reconcile persisted manifest versus disk before applying a watchdog decision. A real post-baseline digest difference advances one sequence once; unchanged content does not. Persisted deadlines remain unchanged.

When the process epoch or attempt fence is revoked, keep the observer in preservation/quarantine mode until the finalizer seals or retains the tree:

  • A candidate digest change from the stale tuple is LateWorktreeMutationObserved, never current progress.
  • It cannot wake, reopen, resurrect, renew ownership, cancel a terminal reservation, or change task status.
  • It invalidates any earlier manifest/quiescence assumption and requests a typed preservation/quarantine reconciliation through the lifecycle/finalizer seam. It is retained as rescue evidence.
  • A write after a purported exact reap is late_write_after_reap, proving ownership/quiescence ambiguity; sealing/merge/cleanup must hold until reconciled.
  • Old watcher callbacks carry observer_epoch and the source tuple; callbacks received after detach are stale evidence only.

Rationale: stale authority does not make late bytes worthless, but valuable bytes do not restore stale authority.

6. Long thought and continuous writing

The worktree channel is corroborating, never the sole detector:

  • A long provider/thinking phase with no writes is governed exclusively by native Pi provider/token/probe evidence and bounded grace.
  • Gradual candidate writes with advancing digests can use the bounded observed-only deferral.
  • Continuous target/cache/dependency/WG-log churn leaves both clocks unchanged.
  • Continuous candidate rewrites remain visible but hit the observed-only hard cap; absent receipt-proven Pi progress, the watchdog proceeds to suspicion.

Operator and read-model contract

Add human and stable JSON diagnostics to wg show <task> and wg service status, and publish the shared read model consumed by the downstream expose-live-worktree TUI task. This task does not edit TUI files. Show both clocks on separate lines and label filesystem activity observed/unproven:

Worktree activity: observed/unproven seq=42 writing docs/x.md (+18 KiB, 12s ago)
Pi progress: proven seq=117 thinking_delta (8s ago)
Watchdog: proof deadline in 292s; observed-only extension 90s / 600s cap
Observer: event+reconcile healthy; scan 2s ago; ignored target churn=812

Also expose the exact attempt/fence/worktree lease/root, manifest and policy digests, observer epoch/health, last reconciliation, excluded-churn counts by bounded reason code, unstable/overflow state, late-write/quarantine evidence, and the next safe operator action. Bound/redact path lists and never interpolate untrusted file contents.

The read surfaces are diagnostic only. There is no mark-done-from-write, touch-progress, or operator command that converts an observation into completion. Existing explicit lifecycle/watchdog/finalizer commands remain the only control paths.

Implementation seams and ownership

  • Add a focused worktree-observer module for policy classification, fingerprinting, journal/projection, watcher adapter, reconciliation, injected clock, and deadline calculation. Do not broaden the graph-file watcher into a recursive source scanner.
  • Attach/detach it from the spawn/service path using the authoritative attempt and worktree lease. Persist the baseline before the child execution permit; reconcile before suspect/fence decisions.
  • Add typed read-only evidence/projection fields through the lifecycle/attempt diagnostics. Do not write Task.status, last_interaction_at, completion intent, or lease ownership from the observer.
  • Feed only the bounded suspect_at corroboration into the Pi watchdog seam. implement-pi-stalled owns native receipt ingestion and all suspect/fence/continuation CAS behavior.
  • Reuse the candidate finalizer's path/inclusion rules; if that shared implementation is not yet landed, define one versioned module/interface consumed later rather than creating divergent allow/deny lists.
  • Add attempt-scoped config validation for debounce, reconciliation interval, observed grace, and hard extension cap. Invalid/oversized values fail closed; production defaults do not adapt from telemetry.
  • Add concise wg show/service diagnostics plus a stable JSON/read-model projection without granting mutation authority. Leave TUI rendering to downstream expose-live-worktree.
  • Register a grow-only smoke scenario owned by implement-isolated-worktree.

RED-first deterministic validation

Write failing tests before implementation and retain them permanently:

  1. Exact domain: two attempts plus deliberately different main; only the current lease/root advances activity. A replaced root/path, stale observer epoch, old process epoch, or main write is inert.
  2. Gradual source write: a Fake-Pi/virtual-clock fixture appends distinct content to one source file every 30 seconds for at least 7 minutes. Each digest advance is visible and the bounded observed deadline prevents a premature 300-second suspect. Same-content rewrites do not advance.
  3. Classification: tracked create/modify/mode/delete, plausible untracked deliverable, explicit generated deliverable, .gitignore change, symlink, and gitlink cases are deterministic. target, cache, dependency, generated, WG log/stream/heartbeat, and special-file churn records only bounded ignored diagnostics and never renews either clock.
  4. Events/reconcile: atomic replacement, rename delete+add, lost event, watcher overflow, watcher unavailable/poll-only, unstable partial read, and daemon restart converge to one content sequence and manifest. Restart never refreshes a timestamp or cap from mtime.
  5. Clock authority: filesystem activity and receipt-proven Pi progress advance independently and may arrive in either order. Assert the 300s/120s/600s defaults and deadline formula with virtual time. Duplicate events/restarts cannot replenish the bound; a source rewrite loop eventually reaches Suspect.
  6. Long thought: no worktree changes while native provider/thinking/token evidence progresses, followed by one final atomic source write. No decision relies on main or a filesystem heartbeat.
  7. Late fence: write immediately before the fence CAS, during TERM/reap, after fence, and after a purported reap. Current pre-fence evidence is sequenced; stale writes are preserved/quarantined, invalidate sealing where applicable, and cannot wake or terminalize the task.
  8. No completion authority: candidate creation/deletion/rename and even a complete-looking output never submit done/fail/wait, candidate checkpoint, evaluation, merge, retry, or generation events and never update task last_interaction_at.

Live human-flow and permanent smoke

Add a credential-free installed-binary scenario under tests/smoke/scenarios/ and list implement-isolated-worktree in its grow-only manifest owners.

The scenario must run the real daemon/service and wrapper against a real isolated worktree (Fake-Pi only for provider behavior), then drive the operator-facing terminal flow—not a direct Rust helper—to show:

  • main remains deliberately different/inert;
  • gradual source writes display observed/unproven separately from Pi-proven progress and do not false-stall beyond five minutes of virtual time;
  • continuous target/WG-log churn is visibly ignored;
  • overflow/restart reconciliation and atomic rename converge;
  • a long thought is explained by Pi evidence;
  • the observed-only cap makes a continuous source rewrite loop become suspect;
  • a late fenced write is retained/quarantined without waking the task; and
  • no file mutation makes the task Done.

The smoke may shorten intervals only via explicit test configuration, but must display/assert the production defaults. Run cargo install --path . --locked before this installed-runtime check.

Validation

  • RED tests are written first and fail on the pre-change behavior.
  • The observation domain is the exact current attempt/fence/worktree lease/root; main, old attempts, and stale observer/process epochs are inert.
  • One versioned classifier separates candidate content from target/cache/dependency/generated/WG-log churn and aligns with finalization inclusion; only content/mode/identity digest changes advance activity.
  • last_observed_worktree_activity and receipt-proven last_proven_progress are persisted/rendered as separate clocks; observed-only deferral follows the fixed bounded formula and cannot replenish budgets or authorize fencing/continuation/completion.
  • Lost events, overflow, unavailable watcher, restart reconciliation, atomic rename, same-content rewrite, unstable read, late fenced/post-reap write, long thought, gradual writing, and continuous malicious/volatile writing are deterministic and fail closed.
  • CLI/operator/JSON diagnostics clearly label observed versus proven evidence, ignored churn, cap/deadline, exact source tuple, health, and safe next action; no observation mutates task status, interaction time, ownership, candidate, evaluation, or merge state.
  • The real installed-binary daemon/wrapper/operator terminal smoke covers the listed human flow and is permanently registered with this task as owner.
  • Focused tests, cargo fmt --check, cargo clippy, cargo build --locked, cargo test, cargo install --path . --locked, and owned smoke pass with no regressions (pre-existing unrelated failures must be identified, not silently ignored).

Depends on

Required by

✉ Messages 2 messages (2 unread)

  1. #1quality-pass-worktree2026-07-26T14:52:35.343642908+00:00delivered
    Architecture quality pass complete: metadata now specifies exact lease/root binding, shared candidate classifier, two-clock authority, bounded 300/120/600 deadline policy, watcher+reconciliation, restart/fence semantics, UI evidence, and RED/live-smoke acceptance. Follow the updated task description.
  2. #2implement-isolated-worktree2026-07-26T15:03:59.709474946+00:00read
    Acknowledged quality-pass guidance — I will implement against the exact lease/root tuple, separate observed and proven clocks, bounded 300/120/600 deadlines, finalizer-compatible policy, reconciliation, and late-write preservation.

Log