fix-worktree-cow-build-storage — WorksGood live mirror

Metadata

Status	in-progress
Assigned	`agent-17`
Agent identity	`02e879681e52e0a384106169be043416c4d946e850ab26b2269c57681b52a6e7`
Created	2026-08-06T13:00:13.036826565+00:00
Started	2026-08-06T13:11:47.315246568+00:00

Description

Bootstrap order — prevent this task from reproducing the incident

Work only in your assigned isolated worktree. BEFORE running cargo test, cargo clippy, cargo install, or any broad Rust build, make the minimal test-profile/storage change that prevents full DWARF plus incremental state from being reproduced. Confirm with a tiny focused target first. Then consolidate integration harnesses incrementally and measure physical bytes. Do not run the current full 177-target suite in an unchanged fresh target. Do not use or mutate another worktree target, and do not share a writable target with another agent.

The worktree isolation system currently duplicates enormous Cargo target trees per agent and can kill simultaneous workers with ENOSPC. This is a critical infrastructure defect, not a request to serialize builds.

Observed evidence (2026-08-06): the repository has 177 independent Rust integration-test targets. Default test/dev settings emit fully debuginfo-heavy, statically linked 300-385 MiB test executables. One target tree reached 74 GiB: 57 GiB debug/deps plus 15 GiB incremental. Two isolated full builds can therefore consume roughly 140+ GiB. Git worktrees share Git objects only; the host is ext4 and has no reflink CoW.

Implement a single coherent artifact-storage model that preserves divergent-worktree correctness while physically sharing unchanged build artifacts. Do not point divergent writers at one mutable Cargo target directory. Prefer an immutable content-addressed/warmed lower layer with per-attempt CoW upper state (overlayfs or an equally safe ext4-capable design), stable build paths/remapping, and keys covering source baseline, Cargo.lock, rustc/toolchain, target triple, features, profile, and relevant flags. A pure compiler cache that still materializes a 70 GiB target per worker is insufficient.

Also attack the amplification at source: choose appropriate test-profile debuginfo/incremental policy, reduce unnecessary independent full-link test binaries where practical, and ensure worker validation runs focused tests while the integrated full suite runs once. Preserve useful failure diagnostics and CI equivalence.

Unify disk admission/accounting with owned cache lifecycle. Active layers must never be reaped; source/uncommitted work must never be touched. Refuse a build before launch when measured reserve is insufficient. A zero-headroom state must never report Healthy, and the disk sentinel must not be silently disabled for normal service operation. Delete superseded per-worker cache/controller paths rather than layering another mechanism.

Validation

Explain and measure the before/after physical-byte model, including debug/deps, incremental state, immutable shared bytes, and per-attempt deltas.
Two concurrent worktrees at the same baseline share unchanged physical artifacts; the second does not materialize another complete target tree.
Two divergent worktrees can build concurrently without clobbering, stale-artifact reuse, or mutable-cache races.
On the representative full-test build, physical storage is bounded to one shared baseline plus small per-worktree deltas, not N times the full target.
Focused installed-binary tests and the integrated full suite pass.
Fault tests cover ENOSPC admission, active-layer protection, crash cleanup, stale-layer GC, cache-key invalidation, and a host without native reflink support.
Production authority/helper/cache-path counts do not increase; obsolete storage paths are migrated once and removed.

Mandatory first deliverable: eliminate test-binary amplification

Before designing or implementing sophisticated CoW promotion, substantially consolidate the 177 top-level Cargo integration-test crates into a bounded set of domain harness binaries. A representative 311 MiB test executable contains only 3.1 MiB of .text; DWARF sections account for about 300 MiB (.debug_info 129.7 MiB, .debug_str 116.6 MiB, .debug_line 34.6 MiB, plus ranges/locations). Preserve necessary process isolation using nextest or separate harness groups for cwd/env/global-state/port-mutating tests. Apply a routine test profile with bounded debug information and incremental storage while retaining an explicit full-debug rerun path. Measure clean and incremental target size before and after. The task is not complete if it only serializes builds or adds a compiler cache while continuing to materialize the same per-worker target size.

## Bootstrap order — prevent this task from reproducing the incident
Work only in your assigned isolated worktree. BEFORE running cargo test, cargo clippy, cargo install, or any broad Rust build, make the minimal test-profile/storage change that prevents full DWARF plus incremental state from being reproduced. Confirm with a tiny focused target first. Then consolidate integration harnesses incrementally and measure physical bytes. Do not run the current full 177-target suite in an unchanged fresh target. Do not use or mutate another worktree target, and do not share a writable target with another agent.

## Validation
- Explain and measure the before/after physical-byte model, including debug/deps, incremental state, immutable shared bytes, and per-attempt deltas.
- Two concurrent worktrees at the same baseline share unchanged physical artifacts; the second does not materialize another complete target tree.
- Two divergent worktrees can build concurrently without clobbering, stale-artifact reuse, or mutable-cache races.
- On the representative full-test build, physical storage is bounded to one shared baseline plus small per-worktree deltas, not N times the full target.
- Focused installed-binary tests and the integrated full suite pass.
- Fault tests cover ENOSPC admission, active-layer protection, crash cleanup, stale-layer GC, cache-key invalidation, and a host without native reflink support.
- Production authority/helper/cache-path counts do not increase; obsolete storage paths are migrated once and removed.

## Mandatory first deliverable: eliminate test-binary amplification
Before designing or implementing sophisticated CoW promotion, substantially consolidate the 177 top-level Cargo integration-test crates into a bounded set of domain harness binaries. A representative 311 MiB test executable contains only 3.1 MiB of .text; DWARF sections account for about 300 MiB (.debug_info 129.7 MiB, .debug_str 116.6 MiB, .debug_line 34.6 MiB, plus ranges/locations). Preserve necessary process isolation using nextest or separate harness groups for cwd/env/global-state/port-mutating tests. Apply a routine test profile with bounded debug information and incremental storage while retaining an explicit full-debug rerun path. Measure clean and incremental target size before and after. The task is not complete if it only serializes builds or adds a compiler cache while continuing to materialize the same per-worker target size.

Depends on

done .assign-fix-worktree-cow-build-storage

Required by

open fix-recurring-shell-orphan-convergence

✉ Messages 1 message

#1user2026-08-06T13:11:58.999395551+00:00read

Urgent bootstrap constraint: do not run the existing broad cargo test/clippy/install first. In your isolated worktree, first land the minimal test-profile change that eliminates full DWARF and large incremental state; validate with one focused target and measure it. Then consolidate harnesses. Another worker is active, so never use a shared mutable target.

Log

2026-08-06T13:00:09.837714350+00:00 Task paused
2026-08-06T13:00:15.719913366+00:00 Task published
2026-08-06T13:00:27.923453517+00:00 Lightweight assignment: agent=Careful Programmer (02e87968), exec_mode=full, context_scope=graph, reason=The Careful Programmer is the only available and best-fit agent for this correctness-critical, cross-component storage, build, admission, GC, and validation implementation.
2026-08-06T13:04:59.005685880+00:00 Measured a representative 311 MiB integration-test executable: only 3.1 MiB is .text. DWARF dominates: .debug_info 129.7 MiB, .debug_str 116.6 MiB, .debug_line 34.6 MiB, .debug_ranges 22.1 MiB, .debug_aranges 5.4 MiB, .debug_loc 5.0 MiB. The apparent 300-500 MiB/test is overwhelmingly duplicated debug metadata, not executable code. Test-profile line-tables-only or split/deduplicated debug storage should cut the immediate amplification drastically while retaining useful backtraces.
2026-08-06T13:07:06.616439623+00:00 Priority correction: fix amplification before sophisticated CoW. Cargo treats every top-level tests/*.rs file as a separate integration crate and executable; this repo has 177, each relinking the same library DWARF. Reorganize into a bounded set of domain harness binaries (potentially one where safe), using per-test process isolation such as nextest or separate harnesses for tests that mutate cwd/env/globals/ports. Measure a representative consolidated harness plus line-tables-only/no-incremental profile first. If this reduces clean per-worktree targets to a bounded few GiB, retain worktree isolation and make immutable shared/CAS layers a performance optimization rather than an ENOSPC-critical prerequisite.
2026-08-06T13:11:47.844920553+00:00 Spawned by coordinator --executor pi --model openai-codex:gpt-5.6-sol --isolation required-worktree
2026-08-06T13:12:10.368691245+00:00 Starting implementation; message service unavailable on initial check. First applying bounded Cargo test profile before any build, then auditing harness/storage infrastructure.
2026-08-06T13:17:42.959016453+00:00 Mandatory bootstrap complete: set test debuginfo=line-tables-only and incremental=false, consolidated 176 auto integration crates into 8 domain harnesses, then ran one exact lib test successfully. Clean focused target physical/logical size: 2.2G, incremental=4.0K.