diff options
Diffstat (limited to 'docs')
4 files changed, 120 insertions, 365 deletions
diff --git a/docs/superpowers/followup/pr3803-hil-blindness-reporting.md b/docs/superpowers/followup/pr3803-hil-blindness-reporting.md deleted file mode 100644 index 374ee62c7..000000000 --- a/docs/superpowers/followup/pr3803-hil-blindness-reporting.md +++ /dev/null @@ -1,186 +0,0 @@ -# Blindness Reporting Gaps Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make a HIL worker's sysfs blindness reach the report in the two cases where it -currently does not — an untested producer, and a board that raises. - -**Architecture:** A worker returns `hil_util.sysfs_blind()` as the last field of its result -tuple; `_blind_note()` turns that into a report banner. Two holes: nothing tests the -producer, and a board that raises returns no tuple at all, so its blindness is lost. - -**Tech Stack:** Python 3.13 stdlib, multiprocessing Pool with `maxtasksperchild=1`. - -## Global Constraints - -- A blind worker answers `SYSFS_UNKNOWN` for every attribute, so its "device not found" - means "could not tell". The report must say so or a red cell reads as a broken board. -- `maxtasksperchild=1`: one worker per board, so the flag is per-board and must not be - smeared across boards. -- Tests: `cd test/hil && python3 test/test_hil_bounded.py`. - -## What is already established - -- `hil_test.test_board` returns `(..., hil_util.sysfs_blind(), stray)`; `_blind_note(mret)` - renders the banner; wired into all three report paths. -- **The producer is provably untested**: replacing `hil_util.sysfs_blind()` with `False` in - the return leaves all tests green. Nothing drives `test_board` — it needs a board dict, a - real flock, a flasher and `test_example` per test. -- Blindness fired for real on ci.lan: four workers went blind in one run, and cells failed - *because* of it (`Printer device not found ... (this worker is blind)`). - -**Why this is a separate PR:** closing it means making `test_board` testable, which is a -refactor of the harness's orchestration layer — a different scope from the containment -work, and the reason the gap was accepted rather than papered over. - -## File Structure - -- `test/hil/hil_test.py` — extract the result-tuple assembly from `test_board` so it can be - built and asserted without running a board; carry blindness out of the raise path. -- `test/hil/test/test_hil_bounded.py` — tests for both. - ---- - -### Task 1: Make the result tuple assembly testable - -**Files:** -- Modify: `test/hil/hil_test.py` (`test_board`, the `return (name, err_count, ...)` at the - end of the try block) -- Test: `test/hil/test/test_hil_bounded.py` - -**Interfaces:** -- Produces: `_board_result(name, err_count, failed_tests, rows, t_total, board_wide_fail)` - returning the 7-tuple `(name, err_count, failed, rows, t_total, blind, stray)`, reading - `hil_util.sysfs_blind()` and `hil_health.kill_own_children()` itself. - -- [ ] **Step 1: Write the failing test** - -```python -class BoardResultCarriesBlindness(unittest.TestCase): - def test_a_blind_worker_reports_it(self): - from helper import hil_util, hil_health - self.addCleanup(setattr, hil_util, 'sysfs_blind', hil_util.sysfs_blind) - self.addCleanup(setattr, hil_health, 'kill_own_children', hil_health.kill_own_children) - hil_util.sysfs_blind = lambda: True - hil_health.kill_own_children = lambda: 0 - row = hil_test._board_result('b', 0, [], [], 1.0, False) - self.assertTrue(row[5], 'blindness did not reach the result tuple') - self.assertIn('b', hil_test._blind_note([row])) - - def test_a_sighted_worker_does_not(self): - from helper import hil_util, hil_health - self.addCleanup(setattr, hil_util, 'sysfs_blind', hil_util.sysfs_blind) - self.addCleanup(setattr, hil_health, 'kill_own_children', hil_health.kill_own_children) - hil_util.sysfs_blind = lambda: False - hil_health.kill_own_children = lambda: 0 - row = hil_test._board_result('b', 0, [], [], 1.0, False) - self.assertFalse(row[5]) - self.assertEqual(hil_test._blind_note([row]), '') -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd test/hil && python3 test/test_hil_bounded.py BoardResultCarriesBlindness -v` -Expected: FAIL — `module 'hil_test' has no attribute '_board_result'` - -- [ ] **Step 3: Write minimal implementation** - -```python -def _board_result(name, err_count, failed_tests, rows, t_total, board_wide_fail): - """Assemble a worker's result tuple. Separate from test_board so the two fields only - the WORKER can answer -- its process-global blindness latch and what it could not kill - -- are testable without running a board.""" - stray = hil_health.kill_own_children() - return (name, err_count, [] if board_wide_fail else sorted(set(failed_tests)), - rows, t_total, hil_util.sysfs_blind(), stray) -``` - -Replace the tail of `test_board` with: - -```python - return _board_result(name, err_count, failed_tests, rows, t_total, board_wide_fail) -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd test/hil && python3 test/test_hil_bounded.py -v` -Expected: PASS, and the existing `BlindWorkerReachesTheReport` tests still pass. - -- [ ] **Step 5: Verify the mutation is now caught** - -Replace `hil_util.sysfs_blind()` with `False` inside `_board_result` and re-run; the suite -MUST fail. Restore it. - -- [ ] **Step 6: Commit** - -```bash -git add test/hil/hil_test.py test/hil/test/test_hil_bounded.py -git commit -m "test/hil: make the worker result tuple testable, covering blindness" -``` - ---- - -### Task 2: Carry blindness out of the worker-raise path - -**Files:** -- Modify: `test/hil/hil_test.py` (`test_board`'s except/finally, and `main`'s worker-raise - handler that builds synthetic rows) -- Test: `test/hil/test/test_hil_bounded.py` - -**Interfaces:** -- Consumes: `_board_result` from Task 1. -- Produces: a board that raises still contributes a row whose blindness field is accurate. - -- [ ] **Step 1: Write the failing test** - -```python - def test_a_board_that_raises_still_reports_blindness(self): - """The result tuple is returned inside a try whose finally only releases the lock, - so a board that dies by exception contributed nothing -- and its blindness, the - thing that most explains its failure, was lost with it.""" - from helper import hil_util - self.addCleanup(setattr, hil_util, 'sysfs_blind', hil_util.sysfs_blind) - hil_util.sysfs_blind = lambda: True - row = hil_test._board_result_on_error('b', RuntimeError('boom')) - self.assertTrue(row[5]) - self.assertIn('b', hil_test._blind_note([row])) -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd test/hil && python3 test/test_hil_bounded.py BoardResultCarriesBlindness -v` -Expected: FAIL — no `_board_result_on_error` - -- [ ] **Step 3: Write minimal implementation** - -```python -def _board_result_on_error(name, exc): - """A row for a board that died by exception. err_count 1, no per-test detail, but the - blindness and stray fields are still accurate -- they explain the failure more often - than the exception text does.""" - rows = [(name, {hil_report.BOUNDARY_CELL: - f'{hil_report.REPORT_CELL["fail"]} {type(exc).__name__}'}, None)] - return _board_result(name, 1, [], rows, 0.0, True) -``` - -Wrap the body of `test_board` so the exception path returns it instead of propagating. - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd test/hil && python3 test/test_hil_bounded.py -v` -Expected: PASS - -- [ ] **Step 5: Commit** - -```bash -git add test/hil/hil_test.py test/hil/test/test_hil_bounded.py -git commit -m "test/hil: keep a raising board's blindness in the report" -``` - ---- - -## Caution - -`test_board`'s `finally` releases the board flock. Any restructuring MUST keep that -release on every path, including the new error path — a leaked flock locks the board until -the host reboots. diff --git a/docs/superpowers/followup/pr3803-usbtest-recovery-reserve.md b/docs/superpowers/followup/pr3803-usbtest-recovery-reserve.md deleted file mode 100644 index eb8959520..000000000 --- a/docs/superpowers/followup/pr3803-usbtest-recovery-reserve.md +++ /dev/null @@ -1,175 +0,0 @@ -# usbtest Recovery Reserve Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make the post-hang recovery reserve a derived, asserted property instead of an -accident of four independently-set constants. - -**Architecture:** `hil_test` passes `--budget` and `--outer-timeout` to `usbtest.py`, which -decides at runtime whether a recovery still fits. Today the reserve survives only because -the four numbers happen to line up; nothing ties them together or fails when they stop. - -**Tech Stack:** Python 3.13 stdlib. - -## Global Constraints - -- `usbtest.py`: `RECOVER_FLASH_TIMEOUT = 90`, `RECOVER_RESET_TIMEOUT = 30`. -- `hil_test.py`: `USBTEST_BATTERY_BUDGET = 260`, `USBTEST_RECOVERY_BUDGET = 250`, - `USBTEST_OVERSHOOT = 120`; `outer = BATTERY_BUDGET + (RECOVERY_BUDGET if recovery else - OVERSHOOT)`, used for both the child's `--outer-timeout` and the parent's `run_cmd` bound. -- All five are env-overridable via `hil_util.pos_int_env`, so a rig can change them. -- Tests: `cd test/hil && python3 test/test_hil_health.py` and `test_hil_bounded.py`. - -## What is already established - -The reserve holds at the shipped values, checked by hand: - -- The battery checks its budget BEFORE dispatching a case, so it can overshoot by one - case — worst case `260 + 60 + 5 = 325 s`. -- Recovery is gated on `_time_left() >= RECOVER_RESET_TIMEOUT`, where - `_time_left() = outer_timeout - elapsed - 35`; with `outer = 510` that allows recovery - until `elapsed = 445 s`, and the reflash until `385 s`. -- So ~60 s of margin survives, and recovery does fire. - -**The defect is structural, not arithmetic:** lower `--outer-timeout`, raise `--timeout`, or -raise `USBTEST_BATTERY_BUDGET` via the env and the reserve silently disappears. The failure -mode is a skipped reflash that leaves the D-state holder for the next job — the exact thing -the containment exists to prevent — with no error anywhere. - -**Why this is a separate PR:** it changes the timing contract between `hil_test` and -`usbtest.py`, which affects every board's run duration, so it wants its own review and a -full rig run. - -## File Structure - -- `test/hil/usbtest.py` — a `reserve_ok()` predicate plus a startup assertion. -- `test/hil/hil_test.py` — derive the battery budget from the outer bound rather than - setting both independently. -- `test/hil/test/test_hil_health.py` — tests. - ---- - -### Task 1: Assert the reserve at startup - -**Files:** -- Modify: `test/hil/usbtest.py` (constants block, and `main()` after argparse) -- Test: `test/hil/test/test_hil_health.py` - -**Interfaces:** -- Produces: `usbtest.reserve_ok(budget, outer, case_timeout)` returning bool. - -- [ ] **Step 1: Write the failing test** - -```python -class RecoveryReserveIsChecked(unittest.TestCase): - """The battery may overshoot its budget by ONE already-started case, so the outer bound - must leave room for that overshoot AND a bounded recovery afterwards.""" - - def setUp(self): - import usbtest - self.u = usbtest - - def test_the_shipped_numbers_leave_room(self): - self.assertTrue(self.u.reserve_ok(budget=260, outer=510, case_timeout=60)) - - def test_a_tighter_outer_bound_is_rejected(self): - self.assertFalse(self.u.reserve_ok(budget=260, outer=380, case_timeout=60)) - - def test_a_longer_case_timeout_is_rejected(self): - self.assertFalse(self.u.reserve_ok(budget=260, outer=510, case_timeout=200)) -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd test/hil && python3 test/test_hil_health.py RecoveryReserveIsChecked -v` -Expected: FAIL — `module 'usbtest' has no attribute 'reserve_ok'` - -- [ ] **Step 3: Write minimal implementation** - -```python -def reserve_ok(budget: int, outer: int, case_timeout: int) -> bool: - """Does `outer` leave room for the battery's worst case AND a bounded recovery? - - The budget is checked BEFORE dispatch, so the battery can run to - `budget + case_timeout + 5` (the +5 is run_case's reap). _time_left() subtracts a - further 35 s of fixed tail. A reflash needs RECOVER_FLASH_TIMEOUT beyond that. - """ - worst_case_end = budget + case_timeout + 5 - return outer - worst_case_end - 35 >= RECOVER_FLASH_TIMEOUT -``` - -In `main()`, after parsing args: - -```python - if args.budget and args.outer_timeout and not reserve_ok( - args.budget, args.outer_timeout, args.timeout): - print(f'warning: --outer-timeout {args.outer_timeout} leaves no room for a bounded ' - f'recovery after a --budget {args.budget} battery with --timeout ' - f'{args.timeout} cases; a HUNG board will be left wedged', file=sys.stderr) -``` - -Warn, do not exit: a caller that deliberately runs without recovery is legitimate. - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd test/hil && python3 test/test_hil_health.py RecoveryReserveIsChecked -v` -Expected: PASS - -- [ ] **Step 5: Commit** - -```bash -git add test/hil/usbtest.py test/hil/test/test_hil_health.py -git commit -m "usbtest: check the recovery reserve instead of assuming it" -``` - ---- - -### Task 2: Derive the outer bound from one place - -**Files:** -- Modify: `test/hil/hil_test.py` (constants block ~line 227, and `test_device_usbtest`) -- Test: `test/hil/test/test_hil_bounded.py` - -**Interfaces:** -- Consumes: `usbtest.reserve_ok` semantics (duplicate the arithmetic, do not import - usbtest — `hil_test` must not import it). -- Produces: an assertion at module import that the shipped constants satisfy the reserve. - -- [ ] **Step 1: Write the failing test** - -```python - def test_the_shipped_constants_satisfy_the_reserve(self): - """Whatever the env overrides, the pair hil_test computes must leave recovery room: - outer - (budget + case_timeout + 5) - 35 >= 90.""" - outer = hil_test.USBTEST_BATTERY_BUDGET + hil_test.USBTEST_RECOVERY_BUDGET - self.assertGreaterEqual(outer - (hil_test.USBTEST_BATTERY_BUDGET + 60 + 5) - 35, 90) -``` - -- [ ] **Step 2: Run test to verify it fails** - -Temporarily set `HIL_USBTEST_RECOVERY_BUDGET=100` and run; expect FAIL. Unset. - -- [ ] **Step 3: Add the guard** - -```python -# The recovery reserve is a PROPERTY of these two, not a coincidence: the battery may -# overshoot its budget by one already-started case (checked before dispatch), and a bounded -# reflash needs 90 s after a 35 s fixed tail. Env overrides make this checkable at import -# rather than discoverable when a wedge is left unrecovered. -if USBTEST_RECOVERY_BUDGET - 60 - 5 - 35 < 90: - print(f'warning: HIL_USBTEST_RECOVERY_BUDGET={USBTEST_RECOVERY_BUDGET} leaves no room ' - f'for a bounded reflash after a one-case overshoot; HUNG boards will stay wedged', - file=sys.stderr) -``` - -- [ ] **Step 4: Run tests to verify they pass** - -Run: `cd test/hil && python3 test/test_hil_bounded.py -v` -Expected: PASS - -- [ ] **Step 5: Commit** - -```bash -git add test/hil/hil_test.py test/hil/test/test_hil_bounded.py -git commit -m "hil: warn when the timeout constants leave no recovery reserve" -``` diff --git a/docs/superpowers/followup/pr3840-mret-board-result.md b/docs/superpowers/followup/pr3840-mret-board-result.md index 7b8da7b9c..77b76b605 100644 --- a/docs/superpowers/followup/pr3840-mret-board-result.md +++ b/docs/superpowers/followup/pr3840-mret-board-result.md @@ -3,6 +3,15 @@ **Origin:** split out of PR #3840 (making `hil_report.md` a rendering of `hil_report.json`). Delete this file when its own PR lands. +> **SUPERSEDED IN PART (2026-08-26).** Written against a 7-field tuple whose index 5 was +> `blind`. The sysfs blindness subsystem is gone: `test_board` now returns **6** fields with +> `stray` at index 5, and its board-locked early return is 5 wide. The problem described +> below is unchanged and still worth fixing — three producers, three widths, and +> `len(r) > 5 and r[5]` reads a WRONG SLOT rather than raising. But drop the `blind` field +> from the proposed NamedTuple and re-derive every index from `hil_test.test_board` before +> executing, or `_stray_note` starts reading a duration as a stray count. +> `StrayNoteSurvivesTheTupleWidth` pins the current shape. + ## What is established `test_board()` returns a bare tuple that three producers build and fourteen call sites read diff --git a/docs/superpowers/specs/2026-07-30-hil-usbtest-fleet-wedge-design.md b/docs/superpowers/specs/2026-07-30-hil-usbtest-fleet-wedge-design.md index 3ed0c1519..a34848f06 100644 --- a/docs/superpowers/specs/2026-07-30-hil-usbtest-fleet-wedge-design.md +++ b/docs/superpowers/specs/2026-07-30-hil-usbtest-fleet-wedge-design.md @@ -1,9 +1,9 @@ # HIL fleet-wedge containment Date: 2026-07-30 -Status: implemented, then superseded in part — addendum last checked 2026-08-12 -against the shipped code; where they disagree the CODE and the usb-kernel-recover -skill win, never this document. +Status: implemented, then superseded in part, then TRIMMED (2026-08-25 — see the +addendum at the end). Last checked against the shipped code 2026-08-25; where they +disagree the CODE and the usb-kernel-recover skill win, never this document. - **Pool guard.** A single constant, not the flat 4200s below and not a derivation: `POOL_TIMEOUT = pos_int_env('HIL_POOL_TIMEOUT', 3600)`. A per-controller model briefly @@ -17,7 +17,9 @@ skill win, never this document. - **Job ceilings.** 90/90/120 min (build.yml), not 60/60/90 and not the 85/115 below. They must clear the 3600s guard plus the pre-pool checkout/artifact merge and the post-guard sweep and report upload. No job pins `HIL_POOL_TIMEOUT`. -- **Battery budgets.** `USBTEST_BATTERY_BUDGET` 260s, `USBTEST_RECOVERY_BUDGET` 250s. +- **Battery budgets.** `USBTEST_BATTERY_BUDGET` 260s. The recovery reserve is no longer a + constant: `usbtest.recovery_reserve(flasher)` derives it per flasher (RP-target openocd 390s, + other openocd/jlink/stlink 190s, esptool/lm4flash 150s) — see the trim addendum. The 200s-with-a-197s-floor derivation recorded here was never shipped; the floor assertion was removed with it. - **HUNG recovery.** Reflash of the DUT through its roster flasher @@ -234,3 +236,108 @@ a stuck run and explain it without anyone touching the rig. the board or flashing `board_test` by hand resolves it without any code change. - **An unattended PVE watchdog** that detects the wedge and power-cycles the host. Declined: more moving parts, and it can cut a running CI job. + +--- + +## Trim addendum — 2026-08-25 + +The containment above grew past what one maintainer could hold. This records what was +removed and, more importantly, the rule that decided it, so the next reader does not +re-derive the deleted layers from the incident above. + +### The dividing principle + +**The CI job ceiling bounds how long a run can burn. It does nothing about state that +outlives the run.** Cut what the ceiling contains; keep what it does not. + +- Contained by the ceiling: a worker blocked on a wedged device. `drain_pool` keeps the + boards that finished, `_write_failed_spec` names the one in flight, `_abandon_exit` + writes and uploads the report, and the job dies at `timeout-minutes` regardless. The + cost is one pool slot. +- **Not** contained: a D-state holder left on a usbfs node, or an unswept stray still + holding a probe. The job dies and those survive it, on a self-hosted runner, into the + next run. That is the original incident. + +### Removed + +- **The sysfs blindness subsystem.** `SYSFS_UNKNOWN`, the `_SysfsUnknown` sentinel, the + path→inode strand memo with its `_STRAND_MISS` miss-sentinel, the four-credit blindness + cap, `sysfs_blind()`/`sysfs_blind_note()`, `note_sysfs_strand()`, `bounded_open()`, + `usb_scan`'s `(list, bool)` return, usbtest's `inconclusive` abort, and `_blind_note`'s + report banner. `read_sysfs` is an ordinary `open().read()` returning `str | None`. + + It was a three-valued contract five files had to reason about, and misreading unknown as + absence was silent — a healthy board reported as a firmware regression. It existed for + exactly one attribute that can block. Verified against v6.12.96 `sysfs.c`: only + `usb_string_attr` (`product`/`manufacturer`/`serial`, sysfs.c:141-143) takes + `usb_lock_device_interruptible`; `idVendor`, `idProduct`, `bcdDevice`, `busnum`, + `devnum` and `speed` are lock-free `sysfs_emit` from cached fields. Two of the five + `read_sysfs` call sites read attributes that cannot block at all. + + **The bound stayed, and it is not opt-in.** An early cut of this trim made `read_sysfs` + unbounded on the theory that a blocked worker costs one pool slot. That is false: + `usb_scan` reads `serial` on every device matching the VID to find the one it wants, and + `hil_lock.controller_of` does exactly that from `controller_permit`, on essentially every + board — so one wedged DUT would stall *every* worker and the pool guard would take the + whole run. `read_sysfs` and `usb_scan` are bounded by `SYSFS_READ_GRACE` by default; + three call sites forgot an opt-in version within a single sitting, and a unit test now + pins the default. + + What is gone is the *contract*, not the bound: no third value, no process-wide blindness + latch, no `(list, bool)` return, no report banner. A give-up reads as None like any + unreadable attribute, and the cost is confined to the device that is actually wedged. + + **`hil_pool_check` is why the memo has to be exact.** It is a standalone + ThreadPoolExecutor tool with no guard behind it, run precisely when a device is suspected + wedged, and it polls (`wait_device` re-scans every 0.5 s). The bounded read gives up and + remembers + the path so a poll loop cannot leak a thread and an fd per pass. That memo is keyed by + **kernfs inode, not by path**: a busport does not change when a board returns to the same + physical port, so a path-only blacklist would outlive the wedge and make the tool's own + recovery flow (reset/reflash → `wait_device` polls for the new inode) never see the board + again. A changed inode is the all-clear; `os.stat` is safe on a wedged device because it + does not invoke `->show()`. A give-up reads as None + — the same as unreadable — and `sysfs_stranded()` lets the footer warn that a "missing" + row may be the tool losing sight of healthy hardware. One local bound with a warning + line, not the five-file three-valued contract that was removed. + +- **The recovery budget arithmetic.** `recovery_steps()`, `_time_left()` and its three + per-step gates. The reserve was an independent 250s — one number for the whole fleet — + that could not contain the ladder it + reserved for (reset 30 + reflash 90 + Rescue-DP POR 90 + retry 90 + settles), which is + why the child re-decided before every step — with a bare `- 35` for downstream costs + that nobody could re-derive. Between them they produced a recovery that skipped its own + steps for most real hangs. The reserve now counts `hil_util.REAP_GRACE` **per bounded + step** — `run_cmd` spends that reaping a child it had to SIGKILL, on top of the step's own + timeout — which is what the `- 35` was standing in for. Undersizing it is worse than not + recovering at all: the outer killpg lands mid-reflash and orphans the flasher on the + probe. A unit test asserts the reserve covers the ladder. `USBTEST_RECOVERY_BUDGET` is now derived from + `usbtest.RECOVER_*` **per flasher and per target**: the Rescue-DP legs are openocd-only + (`rescue_openocd` refuses anything else) and a stub reset is screened out, so an esptool + board no longer reserves 200s it can never spend. The child runs the ladder straight + through, and `--outer-timeout` — parsed but unused once the gates went — is deleted. + +### Deliberately kept + +- The pool guard, `drain_pool`, the re-run spec, `_abandon_exit`, the CI ceilings. +- `hil_health`'s sweep **including** `_kill_and_confirm`. SIGKILL is queued, not delivered, + for a task in uninterruptible sleep, and a healthy in-flight testusb sits in exactly that + state — so `os.kill` returning success proves nothing, and the recheck is the only honest + answer to "is the rig dirty for the next job?". +- usbtest's reset→check→reflash ladder and the `convoy_safe` gate. This is the only thing + that unpoisons the rig mid-run, and PR #3832 extends it from 11 to 18 of 27 boards. +- `mtp_test.py` as a separate process — one job, a clean boundary, and runnable by hand + against a board while debugging. + +### Structural changes with no behaviour change + +- Blocking device IO now runs in a child process everywhere, not just where it was noticed + first. The printer WRITE half joined the read half (`usblp_open` ignores `O_NONBLOCK` and + stalls in `usb_autopm_get_interface()` holding the driver-global `usblp_mutex`), and the + HID echo followed (`hid.enumerate()` reads `manufacturer`/`product` for every HID device + it lists, both under the device lock). `test_device_midi_test` is NOT in that set: ALSA + rawmidi honours `O_NONBLOCK` on open (v6.12.96 rawmidi.c:489), unlike usblp. +- `main()`'s two abort paths were near-identical 40-line blocks; `_abort_report` holds that + shape once. The controller-hint cache and pool construction moved to their own helpers. +- The unit suite stopped sleeping 54 of its 78 seconds — mostly one named-and-zeroable + post-flash settle paid by ten tests against a fake rig. |