Merge pull request #3803 from hathach/claude/hil-wedge-containment

hil, ci: contain a wedged USB stack instead of stranding the runner

7 files changed, 1162 insertions, 11 deletions

diff --git a/docs/superpowers/followup/pr3803-flasher-recover.md b/docs/superpowers/followup/pr3803-flasher-recover.md
new file mode 100644
index 000000000..e9fff7480
--- /dev/null
+++ b/docs/superpowers/followup/pr3803-flasher-recover.md
@@ -0,0 +1,280 @@
+# `flasher_recover` Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Give the 15 HIL boards whose flasher cannot reach its probe past a poisoned usbfs
+node a second, convoy-safe flasher used only for recovery.
+
+**Architecture:** An optional roster key `flasher_recover` beside `flasher`.
+`hil_flash.recover_flasher(board)` picks it when present; `hil_test` substitutes it into the
+`--recover-board` JSON so `usbtest.py` never learns a second entry exists. Delivery over
+openocd's jlink driver is convoy-safe by construction, but the flash command form must
+differ from the one `flash_openocd` uses, so the recovery gets its own flasher name.
+
+**Tech Stack:** Python 3.13 stdlib, openocd 0.12.0+dev (build 0ce743125 on ci.lan),
+libjaylink, J-Link probes.
+
+## Global Constraints
+
+- Roster JSON: `test/hil/tinyusb.json`. `flasher_recover` is OPTIONAL; absent means today's
+  behaviour (`recover_flasher` returns the primary).
+- Never change the shape of `board['flasher']` — it is read as a dict in `hil_flash`,
+  `hil_test`, `usbtest`, `hil_pool_check`, `hil_select` and the roster lint, and is shipped
+  as JSON to a subprocess.
+- Flasher dispatch is by name: `getattr(hil_flash, f'flash_{name}')` / `reset_{name}`.
+- `RECOVER_FLASH_TIMEOUT = 90`, `RECOVER_RESET_TIMEOUT = 30` (`usbtest.py`). Any board whose
+  flash cannot finish inside 90 s is not a candidate.
+- Tests run offline: `cd test/hil && python3 test/test_hil_select.py`.
+
+## What is already established
+
+**Landed on PR #3803 and inert without roster entries:** `hil_flash.recover_flasher()`,
+`convoy_safe()` accepting openocd-over-jlink, `hil_test` substituting the recovery flasher
+into `--recover-board`, and `test_hil_select.FlasherRecoverEntry` (4 tests).
+
+**Verified in source:**
+- openocd's jlink driver ignores `adapter usb vid_pid` — `jlink.c` never reads
+  `adapter_usb_get_vids/pids`; selection is `adapter serial` / USB address / usb location.
+  Do NOT lint a jlink recovery entry for `vid_pid`.
+- It is convoy-safe anyway: libjaylink `discovery_usb.c` returns early unless
+  `idVendor == 0x1366` and the PID is in its table, and only THEN calls `libusb_open`. A
+  wedged `cafe:4010` DUT is never opened.
+- CMSIS-DAP stays pin-gated: `cmsis_dap_usb_bulk.c:107` skips before `libusb_open`, and
+  `id_filter` is only `vids[0] || pids[0]`.
+
+**Measured on ci.lan 2026-08-17**, base args
+`-f interface/jlink.cfg -c "transport select swd" -c "adapter speed 4000" -f target/<cfg>`:
+
+| Board | target cfg | flash | reset |
+|--------------------------|--------------|-------|-------|
+| stm32f407disco           | stm32f4x     | OK    | OK    |
+| stm32f072disco           | stm32f0x     | OK    | OK    |
+| stm32f723disco           | stm32f7x     | OK    | OK    |
+| stm32l476disco           | stm32l4x     | OK    | OK    |
+| feather_nrf52840_express | nrf52        | OK    | OK    |
+| metro_m4_express         | atsame5x     | OK    | OK    |
+| frdm_k64f                | k60          | OK    | OK    |
+
+`frdm_k64f` is host-only (`tests.device == false`) — verify its reset over UART
+(`/dev/serial/by-id/usb-SEGGER_J-Link_000621000000-if00`), never by USB disconnect.
+
+**Excluded, with reasons:** `lpcxpresso11u37` — 118 s for 24 KB at 1 MHz with a verify
+mismatch, versus 0.277 s via JLinkExe; cannot fit `RECOVER_FLASH_TIMEOUT`.
+`mimxrt1064_evk`, `ra4m1_ek`, `nrf54lm20dk` — no target config exists in this openocd
+build, so they cannot be covered at all. **The board that wedges most (mimxrt1064_evk) is
+therefore still uncovered by this work.**
+
+**The blocker this plan solves:** `flash_openocd` issues `program <fw> verify reset exit`,
+which fails over the jlink transport on BOTH families tried (`stm32f4x`, `stm32f0x`) with
+`Examination failed` → `auto_probe failed`, with or without a preceding `init; reset halt`.
+Every successful flash above used the explicit sequence in Task 1.
+
+**Why this is a separate PR:** it adds a roster capability and a new flasher backend, which
+is a different scope from containing a wedge; and it needs bench time on seven boards.
+
+## File Structure
+
+- `test/hil/hil_flash.py` — add `flash_openocd_seq` / `reset_openocd_seq`; extend
+  `convoy_safe` to accept the new name. This is the only file that learns the command form.
+- `test/hil/tinyusb.json` — seven `flasher_recover` entries.
+- `test/hil/test/test_hil_select.py` — extend `FlasherRecoverEntry`; add a roster lint.
+
+---
+
+### Task 1: `openocd_seq` flasher backend
+
+**Files:**
+- Modify: `test/hil/hil_flash.py` (beside `flash_openocd`, ~line 100)
+- Test: `test/hil/test/test_hil_select.py`
+
+**Interfaces:**
+- Consumes: `_openocd_cmd_base(flasher)`, `hil_util.run_cmd`.
+- Produces: `flash_openocd_seq(board, firmware, timeout=None)`,
+  `reset_openocd_seq(board, timeout=None)`, both returning
+  `subprocess.CompletedProcess`; `convoy_safe()` returns True for
+  `{'name': 'openocd_seq', 'args': '...interface/jlink.cfg...'}`.
+
+- [ ] **Step 1: Write the failing test**
+
+```python
+    def test_openocd_seq_is_convoy_safe_over_jlink(self):
+        self.assertTrue(hil_flash.convoy_safe(
+            {'name': 'openocd_seq', 'args': '-f interface/jlink.cfg -f target/stm32f4x.cfg'}))
+
+    def test_openocd_seq_uses_explicit_flash_commands_not_program(self):
+        """`program` fails over the jlink transport: Examination failed -> auto_probe
+        failed, measured on stm32f4x and stm32f0x."""
+        seen = {}
+        real = hil_util.run_cmd
+        hil_util.run_cmd = lambda cmd, **k: seen.setdefault('cmd', cmd) or real('true')
+        try:
+            hil_flash.flash_openocd_seq(
+                {'flasher': {'name': 'openocd_seq', 'uid': 'X', 'args': '-f interface/jlink.cfg'}},
+                '/tmp/fw.elf', timeout=5)
+        finally:
+            hil_util.run_cmd = real
+        self.assertIn('flash write_image erase /tmp/fw.elf', seen['cmd'])
+        self.assertIn('verify_image /tmp/fw.elf', seen['cmd'])
+        self.assertNotIn('program ', seen['cmd'])
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `cd test/hil && python3 test/test_hil_select.py FlasherRecoverEntry -v`
+Expected: FAIL — `module 'hil_flash' has no attribute 'flash_openocd_seq'`
+
+- [ ] **Step 3: Write minimal implementation**
+
+```python
+def flash_openocd_seq(board, firmware, timeout=None):
+    # Explicit commands, NOT `program`: over the jlink transport `program` fails at the
+    # flash bank probe ("Examination failed" -> "auto_probe failed"), measured on
+    # stm32f4x and stm32f0x, with or without a preceding reset halt. This sequence
+    # succeeded on all seven candidate boards.
+    flasher = board['flasher']
+    verify = f' -c "verify_image {firmware}"' if flasher.get('verify', True) else ''
+    return hil_util.run_cmd(
+        f'{_openocd_cmd_base(flasher)} -c "init" -c "reset halt" '
+        f'-c "flash write_image erase {firmware}"{verify} -c "reset run" -c "shutdown"',
+        timeout=timeout)
+
+
+def reset_openocd_seq(board, timeout=None):
+    flasher = board['flasher']
+    return hil_util.run_cmd(
+        f'{_openocd_cmd_base(flasher)} -c "init" -c "reset run" -c "shutdown"',
+        timeout=timeout)
+```
+
+In `convoy_safe`, replace `if name != 'openocd':` with:
+
+```python
+    if name not in ('openocd', 'openocd_seq'):
+        return False
+```
+
+- [ ] **Step 4: Run test to verify it passes**
+
+Run: `cd test/hil && python3 test/test_hil_select.py FlasherRecoverEntry -v`
+Expected: PASS
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add test/hil/hil_flash.py test/hil/test/test_hil_select.py
+git commit -m "hil: add openocd_seq flasher for convoy-safe recovery delivery"
+```
+
+---
+
+### Task 2: Roster entries for the seven validated boards
+
+**Files:**
+- Modify: `test/hil/tinyusb.json`
+- Test: `test/hil/test/test_hil_select.py`
+
+**Interfaces:**
+- Consumes: `flash_openocd_seq` / `reset_openocd_seq` from Task 1.
+- Produces: seven boards for which `hil_flash.convoy_safe(hil_flash.recover_flasher(b))`
+  is True.
+
+- [ ] **Step 1: Write the failing test**
+
+```python
+    def test_roster_recover_entries_are_convoy_safe_and_named_openocd_seq(self):
+        import json, pathlib
+        roster = json.loads((pathlib.Path(__file__).parent.parent / 'tinyusb.json').read_text())
+        recover = [b for b in roster['boards'] if 'flasher_recover' in b]
+        self.assertGreaterEqual(len(recover), 7)
+        for b in recover:
+            f = b['flasher_recover']
+            self.assertEqual(f['name'], 'openocd_seq', b['name'])
+            self.assertIn('interface/jlink.cfg', f['args'], b['name'])
+            self.assertIn('adapter speed', f['args'], b['name'])   # required; see below
+            self.assertTrue(hil_flash.convoy_safe(f), b['name'])
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `cd test/hil && python3 test/test_hil_select.py FlasherRecoverEntry -v`
+Expected: FAIL — `0 >= 7`
+
+- [ ] **Step 3: Add the entries**
+
+`adapter speed` is REQUIRED: without it examination fails outright on the jlink driver.
+Add to each board below, using the SAME `uid` as its primary jlink entry:
+
+```json
+"flasher_recover": {
+  "name": "openocd_seq",
+  "uid": "<same probe serial as flasher.uid>",
+  "args": "-f interface/jlink.cfg -c \"transport select swd\" -c \"adapter speed 4000\" -f target/<cfg>.cfg"
+}
+```
+
+| Board | `uid` | `<cfg>` |
+|--------------------------|----------------|-----------|
+| stm32f407disco           | 000773661813   | stm32f4x  |
+| stm32f072disco           | 779541626      | stm32f0x  |
+| stm32f723disco           | 000776606156   | stm32f7x  |
+| stm32l476disco           | 777632258      | stm32l4x  |
+| feather_nrf52840_express | 681295394      | nrf52     |
+| metro_m4_express         | 123456         | atsame5x  |
+| frdm_k64f                | 000621000000   | k60       |
+
+- [ ] **Step 4: Run test to verify it passes**
+
+Run: `cd test/hil && python3 test/test_hil_select.py -v`
+Expected: PASS, and no other selector test regresses.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add test/hil/tinyusb.json test/hil/test/test_hil_select.py
+git commit -m "hil: give seven J-Link boards a convoy-safe recovery flasher"
+```
+
+---
+
+### Task 3: Bench validation on the rig
+
+**Files:** none — this task produces evidence, not code.
+
+- [ ] **Step 1: Confirm the rig is idle and take the locks**
+
+```bash
+ssh [email protected] 'if pgrep -f "[h]il_test.py" >/dev/null; then echo BUSY; exit 1; fi'
+ssh [email protected] 'cd ~/actions-runner/_work/tinyusb/tinyusb && \
+  nohup timeout 900 python3 test/hil/helper/hil_lock.py hold <boards...> --reason "flasher_recover validation" &'
+```
+
+Guard with `if`, never `cmd && echo || echo` — that form only gates the echo and will take
+locks during a live CI run.
+
+- [ ] **Step 2: For each board, flash then reset through the recovery entry**
+
+```bash
+python3 test/hil/hil_test.py -b <board> test/hil/tinyusb.json   # normal path still works
+```
+
+Then force the recovery path by running usbtest with the recovery flags and a firmware that
+hangs a case, or drive `hil_flash.flash_openocd_seq` / `reset_openocd_seq` directly.
+
+- [ ] **Step 3: Verify**
+
+Device boards: `sudo dmesg` shows `USB disconnect` then a fresh enumeration.
+`frdm_k64f`: UART shows the boot banner (see above).
+Every flash must finish well inside `RECOVER_FLASH_TIMEOUT` (90 s).
+
+- [ ] **Step 4: Release locks and record the results in the PR body**
+
+---
+
+## Out of scope, and why
+
+- **`mimxrt1064_evk`** needs an i.MX RT target config that this openocd build does not
+  have. Sourcing or writing one is its own investigation; until then the board with the
+  most wedges has no automated recovery.
+- **Changing `flash_openocd`** to the explicit form would cover these boards without a new
+  name, but `program` is what nine pinned CMSIS-DAP boards use in CI daily and no CMSIS-DAP
+  image could be built in the originating worktree (no pico-sdk) to re-validate it.
diff --git a/docs/superpowers/followup/pr3803-hil-blindness-reporting.md b/docs/superpowers/followup/pr3803-hil-blindness-reporting.md
new file mode 100644
index 000000000..69ff939b0
--- /dev/null
+++ b/docs/superpowers/followup/pr3803-hil-blindness-reporting.md
@@ -0,0 +1,185 @@
+# Blindness Reporting Gaps Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Make a HIL worker's sysfs blindness reach the report in the two cases where it
+currently does not — an untested producer, and a board that raises.
+
+**Architecture:** A worker returns `hil_util.sysfs_blind()` as the last field of its result
+tuple; `_blind_note()` turns that into a report banner. Two holes: nothing tests the
+producer, and a board that raises returns no tuple at all, so its blindness is lost.
+
+**Tech Stack:** Python 3.13 stdlib, multiprocessing Pool with `maxtasksperchild=1`.
+
+## Global Constraints
+
+- A blind worker answers `SYSFS_UNKNOWN` for every attribute, so its "device not found"
+  means "could not tell". The report must say so or a red cell reads as a broken board.
+- `maxtasksperchild=1`: one worker per board, so the flag is per-board and must not be
+  smeared across boards.
+- Tests: `cd test/hil && python3 test/test_hil_bounded.py`.
+
+## What is already established
+
+- `hil_test.test_board` returns `(..., hil_util.sysfs_blind(), stray)`; `_blind_note(mret)`
+  renders the banner; wired into all three report paths.
+- **The producer is provably untested**: replacing `hil_util.sysfs_blind()` with `False` in
+  the return leaves all tests green. Nothing drives `test_board` — it needs a board dict, a
+  real flock, a flasher and `test_example` per test.
+- Blindness fired for real on ci.lan: four workers went blind in one run, and cells failed
+  *because* of it (`Printer device not found ... (this worker is blind)`).
+
+**Why this is a separate PR:** closing it means making `test_board` testable, which is a
+refactor of the harness's orchestration layer — a different scope from the containment
+work, and the reason the gap was accepted rather than papered over.
+
+## File Structure
+
+- `test/hil/hil_test.py` — extract the result-tuple assembly from `test_board` so it can be
+  built and asserted without running a board; carry blindness out of the raise path.
+- `test/hil/test/test_hil_bounded.py` — tests for both.
+
+---
+
+### Task 1: Make the result tuple assembly testable
+
+**Files:**
+- Modify: `test/hil/hil_test.py` (`test_board`, the `return (name, err_count, ...)` at the
+  end of the try block)
+- Test: `test/hil/test/test_hil_bounded.py`
+
+**Interfaces:**
+- Produces: `_board_result(name, err_count, failed_tests, rows, t_total, board_wide_fail)`
+  returning the 7-tuple `(name, err_count, failed, rows, t_total, blind, stray)`, reading
+  `hil_util.sysfs_blind()` and `hil_health.kill_own_children()` itself.
+
+- [ ] **Step 1: Write the failing test**
+
+```python
+class BoardResultCarriesBlindness(unittest.TestCase):
+    def test_a_blind_worker_reports_it(self):
+        from helper import hil_util, hil_health
+        self.addCleanup(setattr, hil_util, 'sysfs_blind', hil_util.sysfs_blind)
+        self.addCleanup(setattr, hil_health, 'kill_own_children', hil_health.kill_own_children)
+        hil_util.sysfs_blind = lambda: True
+        hil_health.kill_own_children = lambda: 0
+        row = hil_test._board_result('b', 0, [], [], 1.0, False)
+        self.assertTrue(row[5], 'blindness did not reach the result tuple')
+        self.assertIn('b', hil_test._blind_note([row]))
+
+    def test_a_sighted_worker_does_not(self):
+        from helper import hil_util, hil_health
+        self.addCleanup(setattr, hil_util, 'sysfs_blind', hil_util.sysfs_blind)
+        self.addCleanup(setattr, hil_health, 'kill_own_children', hil_health.kill_own_children)
+        hil_util.sysfs_blind = lambda: False
+        hil_health.kill_own_children = lambda: 0
+        row = hil_test._board_result('b', 0, [], [], 1.0, False)
+        self.assertFalse(row[5])
+        self.assertEqual(hil_test._blind_note([row]), '')
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `cd test/hil && python3 test/test_hil_bounded.py BoardResultCarriesBlindness -v`
+Expected: FAIL — `module 'hil_test' has no attribute '_board_result'`
+
+- [ ] **Step 3: Write minimal implementation**
+
+```python
+def _board_result(name, err_count, failed_tests, rows, t_total, board_wide_fail):
+    """Assemble a worker's result tuple. Separate from test_board so the two fields only
+    the WORKER can answer -- its process-global blindness latch and what it could not kill
+    -- are testable without running a board."""
+    stray = hil_health.kill_own_children()
+    return (name, err_count, [] if board_wide_fail else sorted(set(failed_tests)),
+            rows, t_total, hil_util.sysfs_blind(), stray)
+```
+
+Replace the tail of `test_board` with:
+
+```python
+        return _board_result(name, err_count, failed_tests, rows, t_total, board_wide_fail)
+```
+
+- [ ] **Step 4: Run test to verify it passes**
+
+Run: `cd test/hil && python3 test/test_hil_bounded.py -v`
+Expected: PASS, and the existing `BlindWorkerReachesTheReport` tests still pass.
+
+- [ ] **Step 5: Verify the mutation is now caught**
+
+Replace `hil_util.sysfs_blind()` with `False` inside `_board_result` and re-run; the suite
+MUST fail. Restore it.
+
+- [ ] **Step 6: Commit**
+
+```bash
+git add test/hil/hil_test.py test/hil/test/test_hil_bounded.py
+git commit -m "test/hil: make the worker result tuple testable, covering blindness"
+```
+
+---
+
+### Task 2: Carry blindness out of the worker-raise path
+
+**Files:**
+- Modify: `test/hil/hil_test.py` (`test_board`'s except/finally, and `main`'s worker-raise
+  handler that builds synthetic rows)
+- Test: `test/hil/test/test_hil_bounded.py`
+
+**Interfaces:**
+- Consumes: `_board_result` from Task 1.
+- Produces: a board that raises still contributes a row whose blindness field is accurate.
+
+- [ ] **Step 1: Write the failing test**
+
+```python
+    def test_a_board_that_raises_still_reports_blindness(self):
+        """The result tuple is returned inside a try whose finally only releases the lock,
+        so a board that dies by exception contributed nothing -- and its blindness, the
+        thing that most explains its failure, was lost with it."""
+        from helper import hil_util
+        self.addCleanup(setattr, hil_util, 'sysfs_blind', hil_util.sysfs_blind)
+        hil_util.sysfs_blind = lambda: True
+        row = hil_test._board_result_on_error('b', RuntimeError('boom'))
+        self.assertTrue(row[5])
+        self.assertIn('b', hil_test._blind_note([row]))
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `cd test/hil && python3 test/test_hil_bounded.py BoardResultCarriesBlindness -v`
+Expected: FAIL — no `_board_result_on_error`
+
+- [ ] **Step 3: Write minimal implementation**
+
+```python
+def _board_result_on_error(name, exc):
+    """A row for a board that died by exception. err_count 1, no per-test detail, but the
+    blindness and stray fields are still accurate -- they explain the failure more often
+    than the exception text does."""
+    rows = [(name, {BOUNDARY_CELL: f'{REPORT_CELL["fail"]} {type(exc).__name__}'}, None)]
+    return _board_result(name, 1, [], rows, 0.0, True)
+```
+
+Wrap the body of `test_board` so the exception path returns it instead of propagating.
+
+- [ ] **Step 4: Run test to verify it passes**
+
+Run: `cd test/hil && python3 test/test_hil_bounded.py -v`
+Expected: PASS
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add test/hil/hil_test.py test/hil/test/test_hil_bounded.py
+git commit -m "test/hil: keep a raising board's blindness in the report"
+```
+
+---
+
+## Caution
+
+`test_board`'s `finally` releases the board flock. Any restructuring MUST keep that
+release on every path, including the new error path — a leaked flock locks the board until
+the host reboots.
diff --git a/docs/superpowers/followup/pr3803-hil-iar-rerun-spec.md b/docs/superpowers/followup/pr3803-hil-iar-rerun-spec.md
new file mode 100644
index 000000000..fe377f741
--- /dev/null
+++ b/docs/superpowers/followup/pr3803-hil-iar-rerun-spec.md
@@ -0,0 +1,118 @@
+# IAR HIL Leg Re-run Spec Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Let the `hil-hfp-iar` CI leg re-run only its failed boards, as the other two HIL
+legs already do.
+
+**Architecture:** `hil_test.py` writes a `<config>.failed` spec into `HIL_REPORT_DIR`; a
+workflow step reads it on the next attempt and passes the boards back as arguments. The IAR
+leg passes `--retry 1` like the others but sets no `HIL_REPORT_DIR` and has no read-back
+step, so its spec is written into the workspace and never read.
+
+**Tech Stack:** GitHub Actions YAML, self-hosted runner.
+
+## Global Constraints
+
+- `.github/workflows/build.yml`. The two working legs are `hil-tinyusb` (matrix) — see its
+  `Set HIL report dir (per run+job; persists across run attempts)` and `Get re-run spec from
+  previous attempt` steps — and they are the pattern to copy.
+- The report dir must be keyed by run id AND job so a matrix leg does not collide with
+  another, and must survive across run attempts (that is the whole point).
+- The IAR leg is the only HIL job that BUILDS inline; its `Build` step is bounded at
+  `timeout-minutes: 30` under a 120-minute job ceiling. Do not disturb that.
+
+## What is already established
+
+- Verified by reading the workflow: `hil-hfp-iar` has neither `HIL_REPORT_DIR` nor a
+  `Get re-run spec` step, while passing `--retry 1`.
+- Consequence: a GitHub re-run of that job re-tests its whole matrix. **This is not a
+  regression** — that leg never had the mechanism — and the unread spec costs only a file.
+- The report artifact upload for that leg is named `hil-report-hfp-iar`.
+
+**Why this is a separate PR:** it is CI plumbing with no code change, it needs a real
+re-run on the self-hosted runner to prove, and it duplicates ~15 lines of workflow that
+would be better factored — a decision worth making on its own.
+
+## File Structure
+
+- `.github/workflows/build.yml` — the `hil-hfp-iar` job only.
+
+---
+
+### Task 1: Give the IAR leg a persistent report dir and a re-run spec
+
+**Files:**
+- Modify: `.github/workflows/build.yml` (job `hil-hfp-iar`)
+
+**Interfaces:**
+- Consumes: `hil_test.py`'s existing `--report-dir` / `.failed` behaviour — no code change.
+- Produces: `env.HIL_REPORT_DIR` for the job, and `$RERUN_ARGS` for the test step.
+
+- [ ] **Step 1: Copy the two steps from `hil-tinyusb`, before the Build step**
+
+```yaml
+      - name: Set HIL report dir (per run+job; persists across run attempts)
+        run: |
+          BASE=$HOME/hil-reports
+          echo "HIL_REPORT_DIR=$BASE/${GITHUB_RUN_ID}-hfp-iar" >> "$GITHUB_ENV"
+
+      - name: Get re-run spec from previous attempt
+        run: |
+          SPEC="$HIL_REPORT_DIR/hfp.json.failed"
+          if [ -f "$SPEC" ]; then
+            echo "RERUN_ARGS=$(cat "$SPEC")" >> "$GITHUB_ENV"
+            echo "re-running only: $(cat "$SPEC")"
+          fi
+```
+
+Match the exact spec filename `hil_test.py` writes for this leg's config — read
+`_write_failed_spec` and the `failed_fname` construction rather than assuming.
+
+- [ ] **Step 2: Pass the spec to the test step**
+
+```yaml
+          python3 test/hil/hil_test.py --retry 1 $SEL_ARGS hfp.json $RERUN_ARGS
+```
+
+`--retry 1` stays FIRST so argparse's last-wins keeps any explicit override working.
+
+- [ ] **Step 3: Point the artifact upload at the report dir**
+
+```yaml
+          path: ${{ env.HIL_REPORT_DIR }}/hil_report.md
+```
+
+- [ ] **Step 4: Validate the YAML**
+
+Run: `python3 -c "import yaml,sys; d=yaml.safe_load(open('.github/workflows/build.yml')); j=d['jobs']['hil-hfp-iar']; print(j['timeout-minutes'], [s.get('name') for s in j['steps']])"`
+Expected: the ceiling is still 120, the Build step still carries `timeout-minutes: 30`, and
+the two new steps appear before Build.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add .github/workflows/build.yml
+git commit -m "ci: let the IAR HIL leg re-run only its failed boards"
+```
+
+---
+
+### Task 2: Prove it on a real re-run
+
+**Files:** none — evidence only.
+
+- [ ] **Step 1:** Push and let `hil-hfp-iar` run to a failure (or force one).
+- [ ] **Step 2:** Confirm `$HIL_REPORT_DIR/hfp.json.failed` exists on the runner after the
+      job.
+- [ ] **Step 3:** Use GitHub's "Re-run failed jobs" and confirm the log line
+      `re-running only: ...` and that only those boards are tested.
+- [ ] **Step 4:** Record the run URL in the PR body.
+
+---
+
+## Consider first
+
+Three jobs would then carry the same ~15 lines. Factoring them into a composite action, or
+computing the report dir inside `hil_test.py` from `GITHUB_RUN_ID`, may be the better
+change — decide that before copying the block a third time.
diff --git a/docs/superpowers/followup/pr3803-pci-rebind-stranding.md b/docs/superpowers/followup/pr3803-pci-rebind-stranding.md
new file mode 100644
index 000000000..de1f7163b
--- /dev/null
+++ b/docs/superpowers/followup/pr3803-pci-rebind-stranding.md
@@ -0,0 +1,157 @@
+# `pci-rebind` Stranding Investigation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Settle when a PCI unbind/rebind of an xHCI controller strands it driverless, so
+the `usb-kernel-recover` skill can state a rule instead of a hypothesis.
+
+**Architecture:** No product code. This is a controlled reproduction against the rig's
+kernel, ending in a documentation change and — if the boundary turns out to be
+detectable — a guard in `usb_recover.sh`.
+
+**Tech Stack:** Linux 6.12.96 (ci.lan), Renesas uPD720201 xHCI, `usb_recover.sh`.
+
+## Global Constraints
+
+- ci.lan is a live CI rig. Take every affected board's lock first
+  (`hil_lock.py hold --all --reason ...`) and confirm no `hil_test.py` is running, with an
+  `if`, not an `&&` chain.
+- A stranded controller takes every fixture on it offline; recovery is
+  `usb_recover.sh pci-bind <addr>` or, failing that, a PVE **host** power cycle — an
+  operator action. Do not start this without being able to reach the host.
+- The rig has two Renesas controllers plus an AMD one; pick the controller with the fewest
+  fixtures for the experiment.
+
+## What is already established
+
+**The skill claimed, unconditionally, that `pci-rebind`'s re-bind hangs on the D-state URB
+and leaves the controller with no driver.** That claim was generalised from ONE observation
+and was used to delete `pci-rebind` and `pci-bind` from `usb_recover.sh` entirely.
+
+**It was refuted in the field on 2026-08-17.** After `hub-cycle 17-2.7` failed to clear a
+wedge, `pci-rebind 0000:05:00.0` recovered the controller in about one second:
+
+```
+02:34:41  remove, state 4 / USB bus 18 deregistered
+02:34:41  remove, state 1 / USB bus 17 deregistered
+02:34:42  xHCI Host Controller / new USB bus registered, assigned bus number 1
+02:34:42  new USB bus registered, assigned bus number 2
+```
+
+Both actions were restored, with the guidance scoped to failure mode: **dead controller →
+use it; device-lock convoy → do not**. Buses renumbered 17/18 → 1/2, which is why rig-wide
+operations need every board's lock.
+
+**What is NOT known:** why the earlier attempt stranded and this one did not. The leading
+hypothesis is that it turns on whether a live D-state URB exists **on that controller** at
+the moment of the re-bind — but in the 02:34 incident the wedged board (17-2.7) was on that
+very controller, which weakens it. An alternative is that `hub-cycle` had already cleared
+the holder, leaving only a dead controller.
+
+**Why this is a separate PR:** it is an experiment that risks taking the rig offline, and
+its output is a documentation change plus possibly a guard — a different scope from any
+code change.
+
+## File Structure
+
+- `.claude/skills/usb-kernel-recover/SKILL.md` — replace the hypothesis in section 3b and
+  the Common-mistakes entry with whatever the experiment establishes.
+- `.claude/skills/usb-kernel-recover/scripts/usb_recover.sh` — only if the boundary is
+  detectable from userspace.
+
+---
+
+### Task 1: Reproduce a controller-scoped D-state wedge
+
+**Files:** none.
+
+- [ ] **Step 1: Establish the safety net**
+
+```bash
+ssh [email protected] 'if pgrep -f "[h]il_test.py" >/dev/null; then echo BUSY; exit 1; fi'
+# hold ALL boards on the target controller
+```
+
+Confirm host access to pve.lan before continuing.
+
+- [ ] **Step 2: Create a wedge deliberately**
+
+Run `usbtest.py` against a board known to hang (`mimxrt1064_evk` has wedged eight times,
+TEST 9/10/24/27), or drive `testusb` directly until a case does not return.
+
+- [ ] **Step 3: Confirm the holder and its controller**
+
+```bash
+ps -eo pid,stat,etimes,wchan:22,args | awk '$2 ~ /D/'
+sudo cat /proc/<pid>/stack        # usbdev_ioctl + [usbtest] = the owner
+readlink -f /sys/bus/usb/devices/usb<N>   # bus -> PCI addr
+```
+
+Record whether the holder is on the SAME controller you will rebind.
+
+---
+
+### Task 2: Rebind and record the outcome
+
+**Files:** none.
+
+- [ ] **Step 1: Rebind, with a bounded observer**
+
+```bash
+timeout 120 sudo usb_recover.sh pci-rebind <addr>; echo "rc=$?"
+```
+
+- [ ] **Step 2: Record which of the three outcomes occurred**
+
+1. Re-bind completes, controller recovers (as on 2026-08-17).
+2. Re-bind hangs; `/sys/bus/pci/devices/<addr>/driver` is gone → **stranded**.
+3. Re-bind completes but the wedge persists.
+
+Capture `sudo journalctl -k --since ...` around the attempt either way.
+
+- [ ] **Step 3: If stranded, recover**
+
+```bash
+sudo usb_recover.sh pci-bind <addr>
+```
+
+If that hangs too, the only remaining step is a PVE host power cycle — an operator action.
+
+- [ ] **Step 4: Repeat at least three times**
+
+One observation is what produced the wrong rule in the first place. Vary whether a D-state
+holder is live on that controller at rebind time; that is the hypothesis under test.
+
+---
+
+### Task 3: Write down what was learned
+
+**Files:**
+- Modify: `.claude/skills/usb-kernel-recover/SKILL.md`
+
+- [ ] **Step 1: Replace section 3b's scoping with the measured rule**
+
+State the condition under which stranding occurs, with the journal lines. If the experiment
+does NOT reproduce stranding, say that too, with the attempt count — "not reproduced in N
+attempts" is a better record than an unexplained warning.
+
+- [ ] **Step 2: If the boundary is detectable, guard the script**
+
+For example, refuse `pci-rebind` when a D-state holder exists on that controller, since the
+holder is enumerable from `/proc` and the controller from `readlink`. Only add this if the
+experiment shows it predicts the outcome.
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add .claude/skills/usb-kernel-recover/
+git commit -m "skills: replace the pci-rebind stranding hypothesis with measurement"
+```
+
+---
+
+## Abort criteria
+
+Stop and hand back to the operator if: a rebind strands the controller and `pci-bind` does
+not recover it; `uhubctl` starts hanging (the convoy has spread to the hub locks); or a CI
+run starts while the rig is in a broken state.
diff --git a/docs/superpowers/followup/pr3803-usbtest-recovery-reserve.md b/docs/superpowers/followup/pr3803-usbtest-recovery-reserve.md
new file mode 100644
index 000000000..eb8959520
--- /dev/null
+++ b/docs/superpowers/followup/pr3803-usbtest-recovery-reserve.md
@@ -0,0 +1,175 @@
+# usbtest Recovery Reserve Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Make the post-hang recovery reserve a derived, asserted property instead of an
+accident of four independently-set constants.
+
+**Architecture:** `hil_test` passes `--budget` and `--outer-timeout` to `usbtest.py`, which
+decides at runtime whether a recovery still fits. Today the reserve survives only because
+the four numbers happen to line up; nothing ties them together or fails when they stop.
+
+**Tech Stack:** Python 3.13 stdlib.
+
+## Global Constraints
+
+- `usbtest.py`: `RECOVER_FLASH_TIMEOUT = 90`, `RECOVER_RESET_TIMEOUT = 30`.
+- `hil_test.py`: `USBTEST_BATTERY_BUDGET = 260`, `USBTEST_RECOVERY_BUDGET = 250`,
+  `USBTEST_OVERSHOOT = 120`; `outer = BATTERY_BUDGET + (RECOVERY_BUDGET if recovery else
+  OVERSHOOT)`, used for both the child's `--outer-timeout` and the parent's `run_cmd` bound.
+- All five are env-overridable via `hil_util.pos_int_env`, so a rig can change them.
+- Tests: `cd test/hil && python3 test/test_hil_health.py` and `test_hil_bounded.py`.
+
+## What is already established
+
+The reserve holds at the shipped values, checked by hand:
+
+- The battery checks its budget BEFORE dispatching a case, so it can overshoot by one
+  case — worst case `260 + 60 + 5 = 325 s`.
+- Recovery is gated on `_time_left() >= RECOVER_RESET_TIMEOUT`, where
+  `_time_left() = outer_timeout - elapsed - 35`; with `outer = 510` that allows recovery
+  until `elapsed = 445 s`, and the reflash until `385 s`.
+- So ~60 s of margin survives, and recovery does fire.
+
+**The defect is structural, not arithmetic:** lower `--outer-timeout`, raise `--timeout`, or
+raise `USBTEST_BATTERY_BUDGET` via the env and the reserve silently disappears. The failure
+mode is a skipped reflash that leaves the D-state holder for the next job — the exact thing
+the containment exists to prevent — with no error anywhere.
+
+**Why this is a separate PR:** it changes the timing contract between `hil_test` and
+`usbtest.py`, which affects every board's run duration, so it wants its own review and a
+full rig run.
+
+## File Structure
+
+- `test/hil/usbtest.py` — a `reserve_ok()` predicate plus a startup assertion.
+- `test/hil/hil_test.py` — derive the battery budget from the outer bound rather than
+  setting both independently.
+- `test/hil/test/test_hil_health.py` — tests.
+
+---
+
+### Task 1: Assert the reserve at startup
+
+**Files:**
+- Modify: `test/hil/usbtest.py` (constants block, and `main()` after argparse)
+- Test: `test/hil/test/test_hil_health.py`
+
+**Interfaces:**
+- Produces: `usbtest.reserve_ok(budget, outer, case_timeout)` returning bool.
+
+- [ ] **Step 1: Write the failing test**
+
+```python
+class RecoveryReserveIsChecked(unittest.TestCase):
+    """The battery may overshoot its budget by ONE already-started case, so the outer bound
+    must leave room for that overshoot AND a bounded recovery afterwards."""
+
+    def setUp(self):
+        import usbtest
+        self.u = usbtest
+
+    def test_the_shipped_numbers_leave_room(self):
+        self.assertTrue(self.u.reserve_ok(budget=260, outer=510, case_timeout=60))
+
+    def test_a_tighter_outer_bound_is_rejected(self):
+        self.assertFalse(self.u.reserve_ok(budget=260, outer=380, case_timeout=60))
+
+    def test_a_longer_case_timeout_is_rejected(self):
+        self.assertFalse(self.u.reserve_ok(budget=260, outer=510, case_timeout=200))
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Run: `cd test/hil && python3 test/test_hil_health.py RecoveryReserveIsChecked -v`
+Expected: FAIL — `module 'usbtest' has no attribute 'reserve_ok'`
+
+- [ ] **Step 3: Write minimal implementation**
+
+```python
+def reserve_ok(budget: int, outer: int, case_timeout: int) -> bool:
+    """Does `outer` leave room for the battery's worst case AND a bounded recovery?
+
+    The budget is checked BEFORE dispatch, so the battery can run to
+    `budget + case_timeout + 5` (the +5 is run_case's reap). _time_left() subtracts a
+    further 35 s of fixed tail. A reflash needs RECOVER_FLASH_TIMEOUT beyond that.
+    """
+    worst_case_end = budget + case_timeout + 5
+    return outer - worst_case_end - 35 >= RECOVER_FLASH_TIMEOUT
+```
+
+In `main()`, after parsing args:
+
+```python
+    if args.budget and args.outer_timeout and not reserve_ok(
+            args.budget, args.outer_timeout, args.timeout):
+        print(f'warning: --outer-timeout {args.outer_timeout} leaves no room for a bounded '
+              f'recovery after a --budget {args.budget} battery with --timeout '
+              f'{args.timeout} cases; a HUNG board will be left wedged', file=sys.stderr)
+```
+
+Warn, do not exit: a caller that deliberately runs without recovery is legitimate.
+
+- [ ] **Step 4: Run test to verify it passes**
+
+Run: `cd test/hil && python3 test/test_hil_health.py RecoveryReserveIsChecked -v`
+Expected: PASS
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add test/hil/usbtest.py test/hil/test/test_hil_health.py
+git commit -m "usbtest: check the recovery reserve instead of assuming it"
+```
+
+---
+
+### Task 2: Derive the outer bound from one place
+
+**Files:**
+- Modify: `test/hil/hil_test.py` (constants block ~line 227, and `test_device_usbtest`)
+- Test: `test/hil/test/test_hil_bounded.py`
+
+**Interfaces:**
+- Consumes: `usbtest.reserve_ok` semantics (duplicate the arithmetic, do not import
+  usbtest — `hil_test` must not import it).
+- Produces: an assertion at module import that the shipped constants satisfy the reserve.
+
+- [ ] **Step 1: Write the failing test**
+
+```python
+    def test_the_shipped_constants_satisfy_the_reserve(self):
+        """Whatever the env overrides, the pair hil_test computes must leave recovery room:
+        outer - (budget + case_timeout + 5) - 35 >= 90."""
+        outer = hil_test.USBTEST_BATTERY_BUDGET + hil_test.USBTEST_RECOVERY_BUDGET
+        self.assertGreaterEqual(outer - (hil_test.USBTEST_BATTERY_BUDGET + 60 + 5) - 35, 90)
+```
+
+- [ ] **Step 2: Run test to verify it fails**
+
+Temporarily set `HIL_USBTEST_RECOVERY_BUDGET=100` and run; expect FAIL. Unset.
+
+- [ ] **Step 3: Add the guard**
+
+```python
+# The recovery reserve is a PROPERTY of these two, not a coincidence: the battery may
+# overshoot its budget by one already-started case (checked before dispatch), and a bounded
+# reflash needs 90 s after a 35 s fixed tail. Env overrides make this checkable at import
+# rather than discoverable when a wedge is left unrecovered.
+if USBTEST_RECOVERY_BUDGET - 60 - 5 - 35 < 90:
+    print(f'warning: HIL_USBTEST_RECOVERY_BUDGET={USBTEST_RECOVERY_BUDGET} leaves no room '
+          f'for a bounded reflash after a one-case overshoot; HUNG boards will stay wedged',
+          file=sys.stderr)
+```
+
+- [ ] **Step 4: Run tests to verify they pass**
+
+Run: `cd test/hil && python3 test/test_hil_bounded.py -v`
+Expected: PASS
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add test/hil/hil_test.py test/hil/test/test_hil_bounded.py
+git commit -m "hil: warn when the timeout constants leave no recovery reserve"
+```
diff --git a/docs/superpowers/specs/2026-07-29-hil-pr-scoped-selection-design.md b/docs/superpowers/specs/2026-07-29-hil-pr-scoped-selection-design.md
index 8158758bc..898b3c8ab 100644
--- a/docs/superpowers/specs/2026-07-29-hil-pr-scoped-selection-design.md
+++ b/docs/superpowers/specs/2026-07-29-hil-pr-scoped-selection-design.md
@@ -1,4 +1,4 @@
-# PR-scoped HIL selection: hil_select.py
+# PR-scoped HIL selection: helper/hil_select.py
 
 **Date:** 2026-07-29
 **Branch:** `claude/hil-select` (based on `claude/hil-pool-check`, which carries the
@@ -26,17 +26,17 @@ confident; every uncertainty widens to the full matrix.
 - Scoping push/master/scheduled runs (always full).
 - Changing hil_test.py behavior (the selector only *composes* existing `-b`/`-bt` args).
 
-## Component: `test/hil/hil_select.py`
+## Component: `test/hil/helper/hil_select.py`
 
 Stdlib-only, importable and CLI. Lives beside the harness so `hil_ci.sh` copies are unaffected
 (it runs on the GitHub runner / dev PC, not on the rig). It must NOT import `hil_test.py`
 (which drags pyserial/pymtp onto the bare GitHub runner): the three test lists
-(`device_tests`, `dual_tests`, `host_test`) move verbatim into a tiny stdlib-only
-`test/hil/hil_examples.py` that both `hil_test.py` and `hil_select.py` import (behavior
-preserving; `hil_ci.sh` scp list gains the new file).
+(`device_tests`, `dual_tests`, `host_test`) move verbatim into the stdlib-only
+`test/hil/helper/hil_util.py` that both `hil_test.py` and `hil_select.py` import (behavior
+preserving; `hil_ci.sh` copies the whole `helper/` directory).
 
 ```
-python3 test/hil/hil_select.py --base <ref> [--diff-file <path>] CONFIG.json [CONFIG.json...]
+python3 test/hil/helper/hil_select.py --base <ref> [--diff-file <path>] CONFIG.json [CONFIG.json...]
 ```
 
 - `--base REF`: changed files = `git diff --name-only $(git merge-base HEAD REF)..HEAD`
@@ -118,7 +118,7 @@ is skipped, not widened (running unrelated boards would test nothing relevant).
 ## CI wiring (`.github/workflows/build.yml`)
 
 - `set-matrix` (PR events only): after generating today's matrices, run
-  `hil_select.py --base origin/${{ github.base_ref }} test/hil/tinyusb.json test/hil/hfp.json`
+  `helper/hil_select.py --base origin/${{ github.base_ref }} test/hil/tinyusb.json test/hil/hfp.json`
   (checkout with enough history to reach the merge base: `fetch-depth: 0` on this one job, or
   an explicit `git fetch origin $BASE_REF`). New job outputs: `hil_select_full`,
   `hil_args_tinyusb`, `hil_args_hfp`, plus the selected-board list consumed by the matrix
@@ -137,15 +137,15 @@ is skipped, not widened (running unrelated boards would test nothing relevant).
 ## Local use
 
 - pre-pr's "Map changes to boards" step delegates to
-  `python3 test/hil/hil_select.py --base $BASE test/hil/tinyusb.json` and derives its
+  `python3 test/hil/helper/hil_select.py --base $BASE test/hil/tinyusb.json` and derives its
   one-board-per-family sample from the selector's board set (its capping/sampling policy is
   unchanged — the selector provides the affected set, pre-pr samples it).
-- Manual: `python3 test/hil/hil_test.py -B examples $(python3 test/hil/hil_select.py --base master test/hil/tinyusb.json | jq -r '.args["tinyusb.json"]') test/hil/tinyusb.json`
+- Manual: `python3 test/hil/hil_test.py -B examples $(python3 test/hil/helper/hil_select.py --base master test/hil/tinyusb.json | jq -r '.args["tinyusb.json"]') test/hil/tinyusb.json`
   — documented in the hil skill.
 
 ## Testing
 
-`test/hil/test_hil_select.py` — stdlib `unittest`, no hardware, injected diffs via
+`test/hil/test/test_hil_select.py` — stdlib `unittest`, no hardware, injected diffs via
 `--diff-file`/API. Cases (the acceptance examples):
 1. `src/portable/raspberrypi/rp2040/dcd_rp2040.c` → only rp2040-family roster boards, device
    tests only, host-only boards absent, `full` false.
@@ -161,7 +161,7 @@ is skipped, not widened (running unrelated boards would test nothing relevant).
 7. `hw/bsp/rp2040/family.cmake` → rp2040-family boards, all their tests.
 8. Mixed device+host diff → no pruning (both roles present).
 The suite runs in `set-matrix` before the selector is used, and locally via
-`python3 test/hil/test_hil_select.py`.
+`python3 test/hil/test/test_hil_select.py`.
 
 ## Safety properties
 
diff --git a/docs/superpowers/specs/2026-07-30-hil-usbtest-fleet-wedge-design.md b/docs/superpowers/specs/2026-07-30-hil-usbtest-fleet-wedge-design.md
new file mode 100644
index 000000000..3ed0c1519
--- /dev/null
+++ b/docs/superpowers/specs/2026-07-30-hil-usbtest-fleet-wedge-design.md
@@ -0,0 +1,236 @@
+# HIL fleet-wedge containment
+
+Date: 2026-07-30
+Status: implemented, then superseded in part — addendum last checked 2026-08-12
+against the shipped code; where they disagree the CODE and the usb-kernel-recover
+skill win, never this document.
+
+- **Pool guard.** A single constant, not the flat 4200s below and not a derivation:
+  `POOL_TIMEOUT = pos_int_env('HIL_POOL_TIMEOUT', 3600)`. A per-controller model briefly
+  lived here and was removed -- it under-modelled the flash phase and could INVERT
+  (adding a usbtest board lowered the guard, because the derived value fell below the
+  baseline it was meant to raise). The guard's only job is to stop a wedged pool short
+  of the job ceiling so the report still gets written; predicting a healthy run's
+  duration is a different problem. `pos_int_env` warns only on a non-integer or a value
+  <= 0: there is NO upper clamp and no warning above any threshold, so a pin larger than
+  a job ceiling silently restores the inversion this work removed.
+- **Job ceilings.** 90/90/120 min (build.yml), not 60/60/90 and not the 85/115 below.
+  They must clear the 3600s guard plus the pre-pool checkout/artifact merge and the
+  post-guard sweep and report upload. No job pins `HIL_POOL_TIMEOUT`.
+- **Battery budgets.** `USBTEST_BATTERY_BUDGET` 260s, `USBTEST_RECOVERY_BUDGET` 250s.
+  The 200s-with-a-197s-floor derivation recorded here was never shipped; the floor
+  assertion was removed with it.
+- **HUNG recovery.** Reflash of the DUT through its roster flasher
+  (`usbtest.py --recover-board/--recover-fw`), not the root-cycle-first recovery in
+  section 1d — replaced after the 2026-08-11 ppps measurement (uhubctl never cuts
+  VBUS; root-cycle is probe-only). Since 2026-08-12 the reflash is SKIPPED
+  when `hil_flash.convoy_safe(board['flasher'])` is false (usbtest.py:675): the flasher
+  would enumerate by opening usbfs nodes, block on the same convoy, and become a second
+  stray rather than clear the first. A holder that owns the device lock inside a driver
+  ioctl is terminal either way -- a reflash only produces a disconnect, and
+  `usb_disconnect()` needs that same lock -- and that state needs a reboot.
+
+Step 0 done — the host was rebooted 2026-07-30 14:11 and the rig
+came back clean. The device that triggered this incident was removed from the rig, so
+only the containment work remains relevant.
+Rig: `ci.lan` (Proxmox guest on `pve.lan`)
+
+## Problem
+
+On 2026-07-29/30 every board in the `ci.lan` usbtest fleet failed, `openocd` processes
+landed in uninterruptible sleep, and no subsequent HIL run could start. Two GitHub
+Actions runs were stranded: `30484641269` sat `in_progress` for over eight hours
+(past GitHub's own 360-minute default), and `30485082274` sat `queued` behind it from
+2026-07-29 19:35 UTC onward. Both report directories were written empty.
+
+A reboot of the `ci` guest at 10:48 did not clear the condition: the same kernel state
+re-formed at 10:52:23.
+
+## Root cause
+
+Five layers, each independently observable.
+
+### 1. A permanently wedged hub worker holds a root-hub device lock
+
+A device that repeatedly re-asserts connect while failing to enumerate keeps
+`hub_event()` busy, and `hub_event()` holds `usb_lock_device(hdev)` on its hub for its
+whole run (hub.c:5896/5989). The `usb_hub_wq` worker sits in `hub_port_reset`, so that
+hub's `device_lock` is effectively never released:
+
+```
+kworker/14:6+usb_hub_wq  (state D, 400+ s)
+  msleep+0x2b
+  hub_port_reset+0x1a4 [usbcore]
+  hub_event+0x727 [usbcore]
+```
+
+`usb usbN-portM: Cannot enable. Maybe the USB cable is bad?` is logged every four seconds
+for as long as it lasts.
+
+Verified against hub.c v6.12.96 rather than inferred: the kernel does **not** retry
+without bound, and root and downstream ports are bounded identically —
+`hub_port_reset()` tries `PORT_RESET_TRIES` then logs that message (hub.c:3149),
+`hub_port_connect()` wraps it in `PORT_INIT_TRIES` = 4 and disables the port on give-up
+(hub.c:5455/5619). A count in the thousands is therefore that many separate connect
+events, not one runaway loop, and it indicts the device rather than the port.
+
+### 2. A parked board storms the second controller
+
+`ra6m5_ek` (`test/hil/tinyusb.json`, uid `8419032D32363657364EF4622D294B4E`, at
+`13-3.3`) runs dfu firmware (`cafe:400b`) and re-enumerates every 1-2 seconds
+continuously, wrapping the entire bus-13 devnum space (`...120 -> 127 -> 4 -> 6 -> 10`).
+This is standing `hub_event` and Address-Device pressure on controller `03:00.0`,
+concurrent with parallel usbtest batteries on the same silicon.
+
+The board is already listed in `boards-skip`, which is precisely why it storms:
+`boards-skip` stops testing a board but never parks it, so it keeps running whatever
+firmware it last received. Park-flash only runs as teardown of a board that actually
+executed tests.
+
+### 3. The kernel `usbtest` control-queue case waits without a timeout
+
+`test_ctrl_queue` blocks on an untimed `wait_for_completion()` while `usbdev_ioctl`
+holds the DUT's `device_lock`:
+
+```
+wait_for_completion+0x8a          <- no _timeout variant
+test_ctrl_queue+0x4ab [usbtest]
+usbtest_do_ioctl+0x501 [usbtest]
+usbdev_ioctl+0x6b8 [usbcore]
+```
+
+`--timeout 60` in `test/hil/usbtest.py` is a subprocess timeout only. `SIGKILL` is not
+delivered to a task in uninterruptible sleep. `usbtest.py` already recognises this and
+reports `HUNG`, then calls `usb_recover.sh root-cycle`.
+
+### 4. openocd inherits the convoy and the whole fleet dies
+
+Once a device lock is stuck, `port_event()` takes a child device's lock to warm-reset
+it and blocks while still holding its hub's lock. Any later
+`open("/dev/bus/usb/BBB/DDD")` against such a device blocks uninterruptibly:
+
+```
+usbdev_open+0xdc [usbcore]  ->  __mutex_lock
+chrdev_open  ->  do_sys_openat2  ->  __x64_sys_openat
+```
+
+That is the state of the three `openocd` processes at 04:16:51 (pids 207921, 207987,
+208034) — the flasher, unkillable. Because one controller carries two buses, a single
+convoy takes out every board on both, which is why the failure presents as the entire
+fleet.
+
+The existing `HUNG` recovery cannot help here. A root-port VBUS cycle frees a
+*device-lock* holder; it cannot free a lock held by a stuck *hub worker*, and on this
+rig the cycle lands on the controller that is already wedged.
+
+### 5. Nothing bounds the damage, so one bad run becomes a CI outage
+
+- `hil-tinyusb` and `hil-tinyusb-esp` in `.github/workflows/build.yml` carry no
+  `timeout-minutes`. Only `hil-hfp-iar` does.
+- `ci.lan` runs a single runner service, so there is one job slot.
+- `test/hil/hil_test.py` bounds the pool with `POOL_TIMEOUT` (4200 s), and that guard
+  fires correctly — but the recovery path does not survive a D-state worker:
+
+```python
+with Pool(processes=os.cpu_count() or 1, initializer=init_worker, initargs=initargs) as pool:
+    async_ret = pool.map_async(test_board, config_boards)
+    try:
+        mret = async_ret.get(timeout=POOL_TIMEOUT)
+    except MpTimeoutError:
+        pool.terminate()
+        pool.join()      # blocks forever: a D-state worker never reaps
+        raise RuntimeError(f'HIL worker pool timed out after {POOL_TIMEOUT}s')
+```
+
+`multiprocessing` joins workers unbounded, so both `pool.terminate()` and
+`pool.join()` hang, as does the `with Pool(...)` exit on the success path. Normal
+`hil-tinyusb (tinyusb.json)` runs take 10-20 minutes; one recent run took 71.3
+minutes, which is the 70-minute guard firing and succeeding. The eight-hour run is the
+pathological case.
+
+## Design
+
+### Step 0 — recovery (manual prerequisite)
+
+Power-cycle the PVE **host**, not the `ci` guest. A guest reboot is not sufficient;
+hubs latch up across the PCIe reset, which the 10:48 reboot demonstrated. Nothing
+below can be verified until the rig is clean.
+
+### Section 1 — CI containment
+
+**1a. Two layered timers.** An inner guard inside `hil_test.py` (`POOL_TIMEOUT`, 70 min)
+that fails gracefully -- it writes a report naming the timeout and the dispatched boards,
+shuts the pool down and exits -- and an outer `timeout-minutes` per rig job (85 for the
+hil-tinyusb jobs; 115 for hil-hfp-iar, which also builds four boards with IAR in the same
+job) as the backstop for when even exiting cannot free the runner. The ceiling must stay
+ABOVE the inner guard, or GitHub kills the job before the report is written.
+
+> **Corrected after measurement.** An earlier revision cut the guard to 30 min on the
+> reading that real runs take 9-17 min and everything longer was the old guard firing.
+> That was wrong. `hil_lock.py` records 22.2/14.3/12.5/10.8 min at usbtest width 1/2/3/4,
+> and raising the per-battery budget to 380s made hung boards cost more again. The 30 min
+> guard then fired on 5 of the last 8 HIL job executions across both rigs, and because
+> `map_async` is all-or-nothing each of those runs published a banner instead of any
+> per-board result. Restored to 4200s, the value whose original rationale -- usbtest
+> batteries are serialized fleet-wide, lengthening the tail -- was correct.
+
+**1b. Bound the pool shutdown.** Add a helper to `test/hil/hil_test.py`:
+
+```python
+def _shutdown_pool(pool, grace=30):
+    """terminate() a Pool without ever blocking forever: multiprocessing joins its
+    workers unbounded, and a worker in uninterruptible sleep (wedged usbfs) never
+    reaps -- which would hold the runner's only job slot indefinitely."""
+    t = threading.Thread(target=pool.terminate, daemon=True)
+    t.start()
+    t.join(grace)
+    return not t.is_alive()
+```
+
+On the `MpTimeoutError` path: write the report first, recording the boards that never
+reported so the run stops producing an empty report directory; then `_shutdown_pool`;
+then `os._exit(1)` if it did not return. The hard exit is the point — it is the only
+way past a kernel-side unkillable child. Use the same helper for the `with Pool(...)`
+exit path.
+
+**1c. Pre-flight rig health check.** `check_rig_health()` runs before the build and
+**never aborts**. It probes `/proc` unprivileged (dmesg is restricted on the rig) for a
+wedged `usb_hub_wq` worker, and reports a `/proc` too restricted to trust as its own
+distinct cause rather than as a diagnosed fault.
+
+It is deliberately non-fatal: the rig is unattended and every remedy for a real wedge is
+manual, so aborting would not fix anything -- it would discard the per-board results the
+run can still collect and leave CI red until a human noticed. It emits a GitHub
+`::error::` annotation and continues. The automatic containment is 1a and 1b, which bound
+a stuck run and explain it without anyone touching the rig.
+
+**1d. Order the recovery correctly.** In `test/hil/usbtest.py`, attempt
+`usb_recover.sh root-cycle` FIRST on a `HUNG` case, and only check for a wedged hub worker
+*afterwards*.
+
+> **Corrected during implementation.** This section originally said to check for a wedged
+> worker *before* the cycle and skip it on a hit. That is backwards. Our own stuck
+> `testusb` holds the DUT's device lock, so any port event drives a hub worker into
+> `usb_lock_device()` on it -- uninterruptible, so it reads `D` in ~100% of samples and the
+> confirmation window makes the wrong verdict *more* confident, not less. Cutting VBUS is
+> precisely what completes the in-flight URB, returns the ioctl and frees that worker, so
+> gating on that signature would suppress the recovery in the exact ordering it exists for.
+> A worker still wedged after the cycle is the genuinely unrecoverable case, and that is
+> what the code now reports.
+
+## Verification
+
+- Unit-test `shutdown_pool` and the `hil_health` detectors against a synthetic `/proc`.
+  A real wedge cannot be manufactured on demand, so they are tested against fabricated
+  inputs rather than live hardware.
+- Confirm the detectors flag a genuinely wedged rig, and return clean on a healthy one.
+- One clean full-fleet `hil_test.py` run to prove `check_rig_health` does not
+  false-abort.
+
+## Out of scope
+
+- **`ra6m5_ek` park and its dfu reset loop.** Dropped by decision. Consequence: the
+  layer-2 devnum storm remains as standing pressure on controller `03:00.0`. Unplugging
+  the board or flashing `board_test` by hand resolves it without any code change.
+- **An unattended PVE watchdog** that detects the wedge and power-cycles the host.
+  Declined: more moving parts, and it can cut a running CI job.