← graph

pin-park-resume-live-owner-race

Pin live-owner park/resume race as a regression

Metadata

Statusdone
Assignedagent-934
Modelpi:openai-codex:gpt-5.6-sol
Created2026-07-29T14:11:25.070567803+00:00
Started2026-07-29T14:11:31.900144885+00:00
Completed2026-07-29T14:54:06.602284197+00:00
Tagslifecycle, regression, park-resume
Tokens18759769 in / 38556 out

Description

An external older-cluster incident reported: a worker parked while external Slurm jobs ran, operator/condition resume overlapped the still-live parked process owning its isolated worktree, repeated ownership refusals charged the per-task spawn breaker, and the ready replacement stayed suppressed for hundreds of ticks even though jobs 5111221 and 5111243 completed successfully. A clean-room probe against main 1a585a2c passed: overlap was breaker-neutral, no competitor launched, and one exact-session/worktree resume occurred after owner exit. Preserve that proof permanently.

Implement a credential-free smoke regression for the exact overlap, preferably by extending resume_waiting_operator.sh or adding a narrowly named scenario. The fake Pi worker must call wg wait, remain alive and retain the isolated worktree until the test explicitly releases it, while an operator immediately runs wg resume. Hold the overlap across at least max_spawn_failures dispatcher ticks. Assert: no competing resumed process launches; task spawn_failures remains zero; no breaker is surfaced/tripped; ownership/preparation deferrals do not corrupt lifecycle authority; dirty WIP and the attested Pi session remain intact. Then release the old process and assert exactly one resumed attempt launches, in the same generation, exact session/session leaf, same worktree, with preserved WIP and no breaker repair/manual graph edit. Make synchronization deterministic rather than relying on a short fixed sleep. Keep all scratch state isolated and preserve failure evidence. Do not change production behavior unless this test exposes a current-main defect; if it does, diagnose and fix narrowly. Add scenario ownership metadata as required.

Incident provenance to mention in test comments: Clean G2 Slurm 5111221 COMPLETED 0:0 in 3:01; Fault G2 Slurm 5111243 COMPLETED 0:0 in 2:02; recovery task complete-v21-current-rc-2n under agent-1688 reconciled preserved artifacts. Do not encode cluster-specific paths or secrets.

Validation

  • Run the new/extended scenario at least three consecutive times to catch race sensitivity.
  • Run tests/smoke/scenarios/resume_waiting_operator.sh.
  • Run tests/smoke/scenarios/spawn_circuit_breaker_self_heal.sh.
  • Run cargo fmt --check.
  • Run focused Rust tests for any touched production module; if production Rust changes, also run cargo clippy.

Depends on

Required by

✉ Messages 7 messages (7 unread)

  1. #1user2026-07-29T14:12:11.711598351+00:00read
    Validation environment note: source HEAD is 1a585a2c, but the currently installed /home/bot/.cargo/bin/wg reports build 6e907183c1c6. Do not treat PATH-global smoke results alone as current-HEAD proof. Build the candidate/current source and put its wg binary first in PATH for every scenario subprocess (including fake Pi calls), or install only when safe per repo guide. Record the tested binary build/commit. The earlier one-off probe used the PATH binary, so your permanent current-source regression is the authoritative proof.
  2. #2user2026-07-29T14:18:14.810999815+00:00read
    I inspected current WIP. Important race in the fixture itself: fake Pi calls `wg wait` immediately. My first one-off probe did exactly that; wait mutated lifecycle revision before wrapper `pi-watchdog bootstrap`, bootstrap failed `stale_revision`, and wrapper killed fake Pi before its hold loop. A fixed 1s sleep made it pass, but acceptance asks deterministic sync. Please gate `wg wait` on observable bootstrap/process-epoch authorization/state (or another exact readiness signal) before parking, and exercise cold runs. Do not mistake this fixture bootstrap race for the incident.
  3. #3pin-park-resume-live-owner-race2026-07-29T14:18:15.871049459+00:00read
    Acknowledged — PATH-global result is not authoritative. I will make the permanent scenario build target/debug/wg from its source tree, prepend that candidate for daemon + fake-Pi child calls, print/record candidate commit/build identity, and rerun all required scenarios through that candidate.
  4. #4user2026-07-29T14:21:34.978480332+00:00read
    Resource/CI note: permanent scenario should not run `cargo build` internally. The smoke suite already has the `WG_SMOKE_CANDIDATE_BIN` convention (see flip_first_required_gate.sh, tui_help_modal_scroll.sh, public_add_visible_publish_tui.sh); otherwise resolve `${CARGO_TARGET_DIR:-$REPO_ROOT/target}/debug/wg` and fail if missing. Build once in your validation command, set WG_SMOKE_CANDIDATE_BIN/PATH, then run three times. Self-building every scenario run makes the smoke gate heavy and timeout-sensitive. Also, disabling `set -e` around `wg wait` does not solve bootstrap authority ordering; please deterministically await bootstrap readiness before wait as noted.
  5. #5pin-park-resume-live-owner-race2026-07-29T14:30:30.439333338+00:00read
    Acknowledged bootstrap-race note — the fixture now waits on both exact watchdog state (PID/session/classification) and the committed pi-continuation-authorized lifecycle event before invoking wg wait; no fixed sleep is used. I am exercising cold/candidate runs.
  6. #6pin-park-resume-live-owner-race2026-07-29T14:30:30.578435939+00:00read
    Acknowledged CI/resource note — I will remove the internal cargo build, adopt WG_SMOKE_CANDIDATE_BIN with the documented target/debug fallback and loud missing/stale-candidate failure, then build once externally and run all scenario subprocesses with candidate-first PATH three times.
  7. #7pin-park-resume-live-owner-race2026-07-29T14:52:19.179715206+00:00read
    Done in f74e5385: fixture awaits exact committed Pi bootstrap authorization without sleeps, uses caller-built candidate convention (no internal cargo build), and passes 3/3 on source f74e5385 / sha256 4f57207b. Current source exposed a separate exact-session handoff defect, fixed narrowly by persisting only watchdog/lifecycle-attested session_id atomically with wg wait; tampered guards fail closed. Companion scenarios, fmt, focused test, clippy, and install pass.

Log