test/hil, ci: contain a wedged USB stack instead of stranding the runner

A wedged USB device used to take the whole HIL run with it. Every worker that
touched the poisoned node blocked uninterruptibly, the pool could not be joined,
map_async discarded every board's result, and the job ran to the GitHub ceiling
with no report at all -- while the self-hosted runner's single job slot stayed
occupied and every queued job waited behind it.

Bound the calls a worker makes itself. read_sysfs, bounded_open and run_cmd all
answer within a wall clock; read_sysfs distinguishes "absent" from "unknown",
because a blocked read is not evidence of absence, and caps stranded readers at
four (each costs a thread and an fd for the life of the process) after which the
worker declares itself blind. mtype, the gio unmount, the libmtp session and the
arecord/iperf reaps go through those bounds; the MTP session runs in a disposable
subprocess, since libmtp's ctypes calls block unkillably in D state.

Bound the run. A pool guard (HIL_POOL_TIMEOUT, 60 min) fires before any job
ceiling and still writes a report. When the pool will not shut down, the sweep
kills what the workers spawned -- descendants, not just direct children, since
flashers run in their own session -- confirms each kill actually landed, and
exits early so the runner is freed. Whatever survived is named in the report.

Deliberately shallow past that point. We do not re-scan process groups, prove
pid ownership, or escalate through sudo: a root-owned survivor is reported, not
force-killed, because signalling a pid we cannot prove is ours is the worse
failure, and the job ceiling backstops whatever this misses. A D-state holder
was never killable anyway.

Recover instead of reporting a wedge. A HUNG usbtest case reflashes its own DUT
through its roster flasher, but only where the flasher can reach its probe past
a poisoned node -- openocd pinned to a validated vid_pid, or esptool. Where it
cannot, the run says so rather than reserving budget for a path that cannot fire.

Raise the CI ceilings above the pool guard so the guard fires first and still
writes its report, and pin --retry 1 on every HIL leg: the guard is a flat
constant and does not scale with max_retry, so argparse's default of 3 would
triple the serialized usbtest tail against an unchanged guard.

Split the module: execution in hil_test/hil_flash/usbtest, infrastructure in
helper/ (locking, health, selection, shared bounded IO), and the two matrix
generators into .github/scripts/ -- ci_set_matrix.py sat in workflows/, where
GitHub treats every file as a workflow definition. 193 tests cover the bounded
paths, the kill ladder, the guard and the selector against synthetic /proc trees
and PATH-injected fakes; a real wedge cannot be manufactured on demand.

1 files changed, 11 insertions, 11 deletions

diff --git a/docs/superpowers/specs/2026-07-29-hil-pr-scoped-selection-design.md b/docs/superpowers/specs/2026-07-29-hil-pr-scoped-selection-design.md
index 8158758bc..898b3c8ab 100644
--- a/docs/superpowers/specs/2026-07-29-hil-pr-scoped-selection-design.md
+++ b/docs/superpowers/specs/2026-07-29-hil-pr-scoped-selection-design.md
@@ -1,4 +1,4 @@
-# PR-scoped HIL selection: hil_select.py
+# PR-scoped HIL selection: helper/hil_select.py
 
 **Date:** 2026-07-29
 **Branch:** `claude/hil-select` (based on `claude/hil-pool-check`, which carries the
@@ -26,17 +26,17 @@ confident; every uncertainty widens to the full matrix.
 - Scoping push/master/scheduled runs (always full).
 - Changing hil_test.py behavior (the selector only *composes* existing `-b`/`-bt` args).
 
-## Component: `test/hil/hil_select.py`
+## Component: `test/hil/helper/hil_select.py`
 
 Stdlib-only, importable and CLI. Lives beside the harness so `hil_ci.sh` copies are unaffected
 (it runs on the GitHub runner / dev PC, not on the rig). It must NOT import `hil_test.py`
 (which drags pyserial/pymtp onto the bare GitHub runner): the three test lists
-(`device_tests`, `dual_tests`, `host_test`) move verbatim into a tiny stdlib-only
-`test/hil/hil_examples.py` that both `hil_test.py` and `hil_select.py` import (behavior
-preserving; `hil_ci.sh` scp list gains the new file).
+(`device_tests`, `dual_tests`, `host_test`) move verbatim into the stdlib-only
+`test/hil/helper/hil_util.py` that both `hil_test.py` and `hil_select.py` import (behavior
+preserving; `hil_ci.sh` copies the whole `helper/` directory).
 
 ```
-python3 test/hil/hil_select.py --base <ref> [--diff-file <path>] CONFIG.json [CONFIG.json...]
+python3 test/hil/helper/hil_select.py --base <ref> [--diff-file <path>] CONFIG.json [CONFIG.json...]
 ```
 
 - `--base REF`: changed files = `git diff --name-only $(git merge-base HEAD REF)..HEAD`
@@ -118,7 +118,7 @@ is skipped, not widened (running unrelated boards would test nothing relevant).
 ## CI wiring (`.github/workflows/build.yml`)
 
 - `set-matrix` (PR events only): after generating today's matrices, run
-  `hil_select.py --base origin/${{ github.base_ref }} test/hil/tinyusb.json test/hil/hfp.json`
+  `helper/hil_select.py --base origin/${{ github.base_ref }} test/hil/tinyusb.json test/hil/hfp.json`
   (checkout with enough history to reach the merge base: `fetch-depth: 0` on this one job, or
   an explicit `git fetch origin $BASE_REF`). New job outputs: `hil_select_full`,
   `hil_args_tinyusb`, `hil_args_hfp`, plus the selected-board list consumed by the matrix
@@ -137,15 +137,15 @@ is skipped, not widened (running unrelated boards would test nothing relevant).
 ## Local use
 
 - pre-pr's "Map changes to boards" step delegates to
-  `python3 test/hil/hil_select.py --base $BASE test/hil/tinyusb.json` and derives its
+  `python3 test/hil/helper/hil_select.py --base $BASE test/hil/tinyusb.json` and derives its
   one-board-per-family sample from the selector's board set (its capping/sampling policy is
   unchanged — the selector provides the affected set, pre-pr samples it).
-- Manual: `python3 test/hil/hil_test.py -B examples $(python3 test/hil/hil_select.py --base master test/hil/tinyusb.json | jq -r '.args["tinyusb.json"]') test/hil/tinyusb.json`
+- Manual: `python3 test/hil/hil_test.py -B examples $(python3 test/hil/helper/hil_select.py --base master test/hil/tinyusb.json | jq -r '.args["tinyusb.json"]') test/hil/tinyusb.json`
   — documented in the hil skill.
 
 ## Testing
 
-`test/hil/test_hil_select.py` — stdlib `unittest`, no hardware, injected diffs via
+`test/hil/test/test_hil_select.py` — stdlib `unittest`, no hardware, injected diffs via
 `--diff-file`/API. Cases (the acceptance examples):
 1. `src/portable/raspberrypi/rp2040/dcd_rp2040.c` → only rp2040-family roster boards, device
    tests only, host-only boards absent, `full` false.
@@ -161,7 +161,7 @@ is skipped, not widened (running unrelated boards would test nothing relevant).
 7. `hw/bsp/rp2040/family.cmake` → rp2040-family boards, all their tests.
 8. Mixed device+host diff → no pruning (both roles present).
 The suite runs in `set-matrix` before the selector is used, and locally via
-`python3 test/hil/test_hil_select.py`.
+`python3 test/hil/test/test_hil_select.py`.
 
 ## Safety properties