fix-coordinator-daemon — WorksGood live mirror

Metadata

Status	done
Assigned	`agent-852`
Created	2026-07-26T06:36:45.976862902+00:00
Started	2026-07-26T07:35:32.639800199+00:00
Completed	2026-07-26T07:51:59.764989489+00:00
Tags	`bug`, `daemon`, `stability`
Tokens	14081236 in / 61262 out

Description

Objective

The service daemon has died silently multiple times this session (and before): the daemon.log just ends mid coordinator tick with no ERROR/panic, the process is gone, and dispatch stops entirely until a manual wg service start. study-long-lived designs a supervisor that would restart it, but this task is to find and fix WHY it crashes (and surface it loudly).

Evidence

Daemon PID gone with no error line; log tail ends at "Coordinator tick #N starting/complete" then nothing.
Recurred at least twice in this session (and reportedly before).
Not OOM (59G RAM free, no dmesg OOM).

Work

Reproduce / attach diagnostics: ensure panics, unwinds, and fatal errors always hit the log (and stderr) before exit; add a crash sentinel the supervisor/launcher can detect.
Identify the crash path (tick handler, fs-watcher, graph reload, IPC, agency pipeline) via logs + the incident timestamps.
Make the daemon auto-restart on unexpected exit (supervisor/systemd/in-process guard) so a crash doesn't halt all dispatch until a human notices.

Validation

Root cause of the silent exit identified and fixed (or, if external signal, documented + mitigated).
Any future crash writes a clear ERROR/panic to daemon.log before exit (no more silent disappearances).
Daemon auto-restarts on unexpected exit; dispatch resumes without manual intervention.

Reference

This session: daemon found "not running" twice with no log error.

## Objective
The service daemon has died silently multiple times this session (and before): the daemon.log just ends mid coordinator tick with no ERROR/panic, the process is gone, and dispatch stops entirely until a manual `wg service start`. `study-long-lived` designs a supervisor that would *restart* it, but this task is to find and fix WHY it crashes (and surface it loudly).

## Evidence
- Daemon PID gone with no error line; log tail ends at "Coordinator tick #N starting/complete" then nothing.
- Recurred at least twice in this session (and reportedly before).
- Not OOM (59G RAM free, no dmesg OOM).

## Work
- Reproduce / attach diagnostics: ensure panics, unwinds, and fatal errors always hit the log (and stderr) before exit; add a crash sentinel the supervisor/launcher can detect.
- Identify the crash path (tick handler, fs-watcher, graph reload, IPC, agency pipeline) via logs + the incident timestamps.
- Make the daemon auto-restart on unexpected exit (supervisor/systemd/in-process guard) so a crash doesn't halt all dispatch until a human notices.

## Validation
- [ ] Root cause of the silent exit identified and fixed (or, if external signal, documented + mitigated).
- [ ] Any future crash writes a clear ERROR/panic to daemon.log before exit (no more silent disappearances).
- [ ] Daemon auto-restarts on unexpected exit; dispatch resumes without manual intervention.

## Reference
This session: daemon found "not running" twice with no log error.

Depends on

done .assign-fix-coordinator-daemon

Required by

✉ Messages 3 messages (3 unread)

#1user2026-07-26T07:32:02.295636550+00:00delivered

Emergency recovery mode: Pi Codex Sol/high, evaluation gates temporarily disabled. Resume preserved WIP and land/install the daemon stability fix. Do not spawn/decompose or alter profile/evaluation settings.

#2fix-coordinator-daemon2026-07-26T07:35:46.592559875+00:00read

Acknowledged emergency recovery mode — I will resume preserved WIP, avoid decomposition/profile changes, validate, commit/push, install, and complete.

#3fix-coordinator-daemon2026-07-26T07:51:55.668124142+00:00sent

Done — landed and pushed d3ccf30a (main implementation 590e0395), installed global binaries, and validated live SIGKILL restart + SIGHUP survival/logging + resumed ticks + clean supervisor stop.

Log

2026-07-26T06:36:45.964657165+00:00 Task paused
2026-07-26T06:37:08.492021672+00:00 Task published
2026-07-26T06:46:08.014766958+00:00 Spawned by coordinator --executor pi --model zai:glm-5.2 --isolation required-worktree
2026-07-26T06:58:33.376403027+00:00 Root cause: daemon has NO signal handlers -> stray SIGTERM/SIGHUP/SIGINT kills it silently (no panic/Err/log). Fix: signal handlers (survive+log) + stronger panic hook + crash sentinel + external auto-restart supervisor.
2026-07-26T07:15:20.671677801+00:00 Task unclaimed: agent 'agent-821' (PID 2802057) process exited
2026-07-26T07:15:22.128924121+00:00 Minted evaluation pipeline evalp-9f3de185024525358669887f for source attempt 2 (coordinator dead-agent retry)
2026-07-26T07:18:14.767649094+00:00 Spawned by coordinator --executor pi --model zai:glm-5.2 --isolation required-worktree
2026-07-26T07:18:39.262186233+00:00 Reconciliation: task recovered from orphaned state (was InProgress, agent: agent-837)
2026-07-26T07:18:39.262197434+00:00 Minted evaluation pipeline evalp-e5d5edbf95fe1f3de0634e5a for source attempt 3 (coordinator orphan reconciliation retry)
2026-07-26T07:27:36.010060842+00:00 Task paused
2026-07-26T07:31:35.588662589+00:00 Task resumed
2026-07-26T07:31:37.994979902+00:00 Spawned by coordinator --executor pi --model openai-codex:gpt-5.6-sol --isolation required-worktree
2026-07-26T07:31:55.519761809+00:00 Resuming prior WIP: reviewing signal diagnostics, crash sentinel, and supervisor implementation before completing tests.
2026-07-26T07:35:20.838494660+00:00 Task unclaimed (was assigned to @agent-849)
2026-07-26T07:35:20.838532901+00:00 Minted evaluation pipeline evalp-ac7cc486ac5401f1e476439b for source attempt 4 (explicit unclaim redispatch)
2026-07-26T07:35:35.047004699+00:00 Spawned by coordinator --executor pi --model openai-codex:gpt-5.6-sol --isolation required-worktree
2026-07-26T07:35:46.646391382+00:00 Resuming preserved WIP; inspecting branch and diagnostics now.
2026-07-26T07:45:12.189742123+00:00 Implemented signal self-pipe diagnostics, deadlock-safe panic logging, clean-shutdown sentinel, default-on external restart supervisor/backoff, lifecycle cleanup, and live crash/restart smoke scenario. Commit 590e0395 pushed. cargo check passes; full binary link was resource-contended/time-bounded.
2026-07-26T07:51:37.277239971+00:00 Fixed test-only IPC call sites discovered by cargo test; supervisor unit tests pass (2/2). Live installed-binary crash supervisor smoke passes: SIGKILL auto-restarted, SIGHUP logged/survived, ticks resumed, clean stop reaped supervisor. cargo install --path . --locked completed.
2026-07-26T07:51:55.596607330+00:00 Validated task criteria: historical incident class documented as unprovable external default-disposition signal/SIGKILL; catchable signals now named and mitigated, panic hook cannot deadlock and writes raw stderr, parent logs fatal child exits, sentinel distinguishes clean stop, default supervisor restarts and resumes ticks. cargo check/build/install, fmt check, targeted unit tests, and owned live smoke pass.
2026-07-26T07:51:59.764994839+00:00 Task marked as done; scheduled evaluator is advisory evidence only (execution is not a quality pass)