fig5-raw-manymany-impg-similarity-2kb-single-node-race

Metadata

Description

Submit a no-array single-node Slurm race for the Fig5 raw many:many 2 kb IMPG similarity scan. Do not run WFMASH, SweepGA/FastGA, minimap2, seqwish, odgi, or any new alignment. Use existing raw unfiltered many:many PAFs and the existing 2 kb full-genome BEDs/manifests from paper_prep/_brainstorming/fig5_raw_manymany_impg_similarity_2kb_sharded/.

Goal: test the user's preferred simpler execution shape: one Slurm job per method x comparison, no array, one node, explicit threads from the allocation. For each of the six method/comparison pairs, call /home/erikg/.cargo/bin/impg similarity once with the full 2 kb target BED and --threads ${SLURM_CPUS_PER_TASK}. Use --cpus-per-task based on available node class; prefer 96 on tux nodes if the partition supports it, otherwise 48 on octopus/workers. Do not hard-code 1 thread.

Outputs should go to a separate scratch directory so they do not overwrite the sharded run: paper_prep/_brainstorming/fig5_raw_manymany_impg_similarity_2kb_single_node_race/. Submit a Slurm dependency finalizer for these six jobs, not a WG polling loop. Preserve all-hit IMPG outputs, and build best-per-window plotting summaries with the same rule as the sharded finalizer: one best interchromosomal hit per 2 kb target window; tie-break by estimated.identity, intersection, dice, cosine, jaccard, then stable lexical target coordinates.

Validation:
- Exactly six non-array Slurm jobs submitted, one per method x comparison, using ${SLURM_CPUS_PER_TASK}.
- No new alignment is performed.
- Finalizer is a Slurm dependency job.
- Report job IDs, node/CPU settings, output paths, and whether this is expected to supersede the current array run.
- Do not cancel current arrays 1706840-1706845 unless explicitly instructed by the user.

Depends on

• done fig5-raw-manymany-impg-similarity-2kb-sharded
• done .assign-fig5-raw-manymany-impg-similarity-2kb-single-node-race

Required by

• (none)

Log

• 2026-06-27T15:58:01.056147364+00:00 Lightweight assignment: agent=random-agent-46f6237a (46f6237a), exec_mode=full, context_scope=task, reason=Pipeline Operator is the closest fit for concrete Slurm orchestration and job-submission work, with a style aligned to executing the requested single-node race jobs directly.
• 2026-06-27T15:59:28.399253176+00:00 Spawned by coordinator --executor codex --model gpt-5.5
• 2026-06-27T15:59:38.394686572+00:00 Starting single-node IMPG race setup; checking messages, prior WIP, and sharded manifests.
• 2026-06-27T16:03:23.812342758+00:00 Added single-node race build/finalize scripts; validating syntax and manifest generation before submission.
• 2026-06-27T16:04:17.895092251+00:00 Pre-submit validation passed: six non-array scripts, no #SBATCH --array, full BED inputs, and literal SLURM_CPUS_PER_TASK threading.
• 2026-06-27T16:05:50.535886412+00:00 Submitted single-node race jobs 1706862-1706867 on tux with 96 CPUs/task; dependency finalizer 1706868 uses afterany on those six jobs.
• 2026-06-27T16:07:51.066328902+00:00 Validated: py_compile passed; six non-array Slurm jobs submitted; finalizer submitted as Slurm dependency job; current arrays 1706840-1706845 left untouched.
• 2026-06-27T16:08:40.805769499+00:00 Committed and pushed 9086b6f; artifacts recorded for report, job manifest, and submission manifest.
• 2026-06-27T16:09:23.983273024+00:00 Task marked as done