summaryrefslogtreecommitdiff
path: root/docs
diff options
context:
space:
mode:
authorHa Thach <[email protected]>2026-08-28 16:09:22 +0700
committerGitHub <[email protected]>2026-08-28 16:09:22 +0700
commit64d952c027df194ea8b107b3d5d0b6c4cb9a994b (patch)
tree1407f23cfc4b667a8bee2f95068440aa5f41d881 /docs
parenteca6caf673452c8ec940e2acf5e46d0631fb72bf (diff)
parent20bb94fcf9ad7fca7fb685e53307f4d03b1340fd (diff)
Merge pull request #3860 from hathach/claude/hil-blindness
hil: drop the sysfs blindness subsystem and derive the recovery reserve
Diffstat (limited to 'docs')
-rw-r--r--docs/superpowers/followup/pr3803-hil-blindness-reporting.md186
-rw-r--r--docs/superpowers/followup/pr3803-usbtest-recovery-reserve.md175
-rw-r--r--docs/superpowers/followup/pr3840-mret-board-result.md9
-rw-r--r--docs/superpowers/specs/2026-07-30-hil-usbtest-fleet-wedge-design.md115
4 files changed, 120 insertions, 365 deletions
diff --git a/docs/superpowers/followup/pr3803-hil-blindness-reporting.md b/docs/superpowers/followup/pr3803-hil-blindness-reporting.md
deleted file mode 100644
index 374ee62c7..000000000
--- a/docs/superpowers/followup/pr3803-hil-blindness-reporting.md
+++ /dev/null
@@ -1,186 +0,0 @@
-# Blindness Reporting Gaps Implementation Plan
-
-> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
-
-**Goal:** Make a HIL worker's sysfs blindness reach the report in the two cases where it
-currently does not — an untested producer, and a board that raises.
-
-**Architecture:** A worker returns `hil_util.sysfs_blind()` as the last field of its result
-tuple; `_blind_note()` turns that into a report banner. Two holes: nothing tests the
-producer, and a board that raises returns no tuple at all, so its blindness is lost.
-
-**Tech Stack:** Python 3.13 stdlib, multiprocessing Pool with `maxtasksperchild=1`.
-
-## Global Constraints
-
-- A blind worker answers `SYSFS_UNKNOWN` for every attribute, so its "device not found"
- means "could not tell". The report must say so or a red cell reads as a broken board.
-- `maxtasksperchild=1`: one worker per board, so the flag is per-board and must not be
- smeared across boards.
-- Tests: `cd test/hil && python3 test/test_hil_bounded.py`.
-
-## What is already established
-
-- `hil_test.test_board` returns `(..., hil_util.sysfs_blind(), stray)`; `_blind_note(mret)`
- renders the banner; wired into all three report paths.
-- **The producer is provably untested**: replacing `hil_util.sysfs_blind()` with `False` in
- the return leaves all tests green. Nothing drives `test_board` — it needs a board dict, a
- real flock, a flasher and `test_example` per test.
-- Blindness fired for real on ci.lan: four workers went blind in one run, and cells failed
- *because* of it (`Printer device not found ... (this worker is blind)`).
-
-**Why this is a separate PR:** closing it means making `test_board` testable, which is a
-refactor of the harness's orchestration layer — a different scope from the containment
-work, and the reason the gap was accepted rather than papered over.
-
-## File Structure
-
-- `test/hil/hil_test.py` — extract the result-tuple assembly from `test_board` so it can be
- built and asserted without running a board; carry blindness out of the raise path.
-- `test/hil/test/test_hil_bounded.py` — tests for both.
-
----
-
-### Task 1: Make the result tuple assembly testable
-
-**Files:**
-- Modify: `test/hil/hil_test.py` (`test_board`, the `return (name, err_count, ...)` at the
- end of the try block)
-- Test: `test/hil/test/test_hil_bounded.py`
-
-**Interfaces:**
-- Produces: `_board_result(name, err_count, failed_tests, rows, t_total, board_wide_fail)`
- returning the 7-tuple `(name, err_count, failed, rows, t_total, blind, stray)`, reading
- `hil_util.sysfs_blind()` and `hil_health.kill_own_children()` itself.
-
-- [ ] **Step 1: Write the failing test**
-
-```python
-class BoardResultCarriesBlindness(unittest.TestCase):
- def test_a_blind_worker_reports_it(self):
- from helper import hil_util, hil_health
- self.addCleanup(setattr, hil_util, 'sysfs_blind', hil_util.sysfs_blind)
- self.addCleanup(setattr, hil_health, 'kill_own_children', hil_health.kill_own_children)
- hil_util.sysfs_blind = lambda: True
- hil_health.kill_own_children = lambda: 0
- row = hil_test._board_result('b', 0, [], [], 1.0, False)
- self.assertTrue(row[5], 'blindness did not reach the result tuple')
- self.assertIn('b', hil_test._blind_note([row]))
-
- def test_a_sighted_worker_does_not(self):
- from helper import hil_util, hil_health
- self.addCleanup(setattr, hil_util, 'sysfs_blind', hil_util.sysfs_blind)
- self.addCleanup(setattr, hil_health, 'kill_own_children', hil_health.kill_own_children)
- hil_util.sysfs_blind = lambda: False
- hil_health.kill_own_children = lambda: 0
- row = hil_test._board_result('b', 0, [], [], 1.0, False)
- self.assertFalse(row[5])
- self.assertEqual(hil_test._blind_note([row]), '')
-```
-
-- [ ] **Step 2: Run test to verify it fails**
-
-Run: `cd test/hil && python3 test/test_hil_bounded.py BoardResultCarriesBlindness -v`
-Expected: FAIL — `module 'hil_test' has no attribute '_board_result'`
-
-- [ ] **Step 3: Write minimal implementation**
-
-```python
-def _board_result(name, err_count, failed_tests, rows, t_total, board_wide_fail):
- """Assemble a worker's result tuple. Separate from test_board so the two fields only
- the WORKER can answer -- its process-global blindness latch and what it could not kill
- -- are testable without running a board."""
- stray = hil_health.kill_own_children()
- return (name, err_count, [] if board_wide_fail else sorted(set(failed_tests)),
- rows, t_total, hil_util.sysfs_blind(), stray)
-```
-
-Replace the tail of `test_board` with:
-
-```python
- return _board_result(name, err_count, failed_tests, rows, t_total, board_wide_fail)
-```
-
-- [ ] **Step 4: Run test to verify it passes**
-
-Run: `cd test/hil && python3 test/test_hil_bounded.py -v`
-Expected: PASS, and the existing `BlindWorkerReachesTheReport` tests still pass.
-
-- [ ] **Step 5: Verify the mutation is now caught**
-
-Replace `hil_util.sysfs_blind()` with `False` inside `_board_result` and re-run; the suite
-MUST fail. Restore it.
-
-- [ ] **Step 6: Commit**
-
-```bash
-git add test/hil/hil_test.py test/hil/test/test_hil_bounded.py
-git commit -m "test/hil: make the worker result tuple testable, covering blindness"
-```
-
----
-
-### Task 2: Carry blindness out of the worker-raise path
-
-**Files:**
-- Modify: `test/hil/hil_test.py` (`test_board`'s except/finally, and `main`'s worker-raise
- handler that builds synthetic rows)
-- Test: `test/hil/test/test_hil_bounded.py`
-
-**Interfaces:**
-- Consumes: `_board_result` from Task 1.
-- Produces: a board that raises still contributes a row whose blindness field is accurate.
-
-- [ ] **Step 1: Write the failing test**
-
-```python
- def test_a_board_that_raises_still_reports_blindness(self):
- """The result tuple is returned inside a try whose finally only releases the lock,
- so a board that dies by exception contributed nothing -- and its blindness, the
- thing that most explains its failure, was lost with it."""
- from helper import hil_util
- self.addCleanup(setattr, hil_util, 'sysfs_blind', hil_util.sysfs_blind)
- hil_util.sysfs_blind = lambda: True
- row = hil_test._board_result_on_error('b', RuntimeError('boom'))
- self.assertTrue(row[5])
- self.assertIn('b', hil_test._blind_note([row]))
-```
-
-- [ ] **Step 2: Run test to verify it fails**
-
-Run: `cd test/hil && python3 test/test_hil_bounded.py BoardResultCarriesBlindness -v`
-Expected: FAIL — no `_board_result_on_error`
-
-- [ ] **Step 3: Write minimal implementation**
-
-```python
-def _board_result_on_error(name, exc):
- """A row for a board that died by exception. err_count 1, no per-test detail, but the
- blindness and stray fields are still accurate -- they explain the failure more often
- than the exception text does."""
- rows = [(name, {hil_report.BOUNDARY_CELL:
- f'{hil_report.REPORT_CELL["fail"]} {type(exc).__name__}'}, None)]
- return _board_result(name, 1, [], rows, 0.0, True)
-```
-
-Wrap the body of `test_board` so the exception path returns it instead of propagating.
-
-- [ ] **Step 4: Run test to verify it passes**
-
-Run: `cd test/hil && python3 test/test_hil_bounded.py -v`
-Expected: PASS
-
-- [ ] **Step 5: Commit**
-
-```bash
-git add test/hil/hil_test.py test/hil/test/test_hil_bounded.py
-git commit -m "test/hil: keep a raising board's blindness in the report"
-```
-
----
-
-## Caution
-
-`test_board`'s `finally` releases the board flock. Any restructuring MUST keep that
-release on every path, including the new error path — a leaked flock locks the board until
-the host reboots.
diff --git a/docs/superpowers/followup/pr3803-usbtest-recovery-reserve.md b/docs/superpowers/followup/pr3803-usbtest-recovery-reserve.md
deleted file mode 100644
index eb8959520..000000000
--- a/docs/superpowers/followup/pr3803-usbtest-recovery-reserve.md
+++ /dev/null
@@ -1,175 +0,0 @@
-# usbtest Recovery Reserve Implementation Plan
-
-> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
-
-**Goal:** Make the post-hang recovery reserve a derived, asserted property instead of an
-accident of four independently-set constants.
-
-**Architecture:** `hil_test` passes `--budget` and `--outer-timeout` to `usbtest.py`, which
-decides at runtime whether a recovery still fits. Today the reserve survives only because
-the four numbers happen to line up; nothing ties them together or fails when they stop.
-
-**Tech Stack:** Python 3.13 stdlib.
-
-## Global Constraints
-
-- `usbtest.py`: `RECOVER_FLASH_TIMEOUT = 90`, `RECOVER_RESET_TIMEOUT = 30`.
-- `hil_test.py`: `USBTEST_BATTERY_BUDGET = 260`, `USBTEST_RECOVERY_BUDGET = 250`,
- `USBTEST_OVERSHOOT = 120`; `outer = BATTERY_BUDGET + (RECOVERY_BUDGET if recovery else
- OVERSHOOT)`, used for both the child's `--outer-timeout` and the parent's `run_cmd` bound.
-- All five are env-overridable via `hil_util.pos_int_env`, so a rig can change them.
-- Tests: `cd test/hil && python3 test/test_hil_health.py` and `test_hil_bounded.py`.
-
-## What is already established
-
-The reserve holds at the shipped values, checked by hand:
-
-- The battery checks its budget BEFORE dispatching a case, so it can overshoot by one
- case — worst case `260 + 60 + 5 = 325 s`.
-- Recovery is gated on `_time_left() >= RECOVER_RESET_TIMEOUT`, where
- `_time_left() = outer_timeout - elapsed - 35`; with `outer = 510` that allows recovery
- until `elapsed = 445 s`, and the reflash until `385 s`.
-- So ~60 s of margin survives, and recovery does fire.
-
-**The defect is structural, not arithmetic:** lower `--outer-timeout`, raise `--timeout`, or
-raise `USBTEST_BATTERY_BUDGET` via the env and the reserve silently disappears. The failure
-mode is a skipped reflash that leaves the D-state holder for the next job — the exact thing
-the containment exists to prevent — with no error anywhere.
-
-**Why this is a separate PR:** it changes the timing contract between `hil_test` and
-`usbtest.py`, which affects every board's run duration, so it wants its own review and a
-full rig run.
-
-## File Structure
-
-- `test/hil/usbtest.py` — a `reserve_ok()` predicate plus a startup assertion.
-- `test/hil/hil_test.py` — derive the battery budget from the outer bound rather than
- setting both independently.
-- `test/hil/test/test_hil_health.py` — tests.
-
----
-
-### Task 1: Assert the reserve at startup
-
-**Files:**
-- Modify: `test/hil/usbtest.py` (constants block, and `main()` after argparse)
-- Test: `test/hil/test/test_hil_health.py`
-
-**Interfaces:**
-- Produces: `usbtest.reserve_ok(budget, outer, case_timeout)` returning bool.
-
-- [ ] **Step 1: Write the failing test**
-
-```python
-class RecoveryReserveIsChecked(unittest.TestCase):
- """The battery may overshoot its budget by ONE already-started case, so the outer bound
- must leave room for that overshoot AND a bounded recovery afterwards."""
-
- def setUp(self):
- import usbtest
- self.u = usbtest
-
- def test_the_shipped_numbers_leave_room(self):
- self.assertTrue(self.u.reserve_ok(budget=260, outer=510, case_timeout=60))
-
- def test_a_tighter_outer_bound_is_rejected(self):
- self.assertFalse(self.u.reserve_ok(budget=260, outer=380, case_timeout=60))
-
- def test_a_longer_case_timeout_is_rejected(self):
- self.assertFalse(self.u.reserve_ok(budget=260, outer=510, case_timeout=200))
-```
-
-- [ ] **Step 2: Run test to verify it fails**
-
-Run: `cd test/hil && python3 test/test_hil_health.py RecoveryReserveIsChecked -v`
-Expected: FAIL — `module 'usbtest' has no attribute 'reserve_ok'`
-
-- [ ] **Step 3: Write minimal implementation**
-
-```python
-def reserve_ok(budget: int, outer: int, case_timeout: int) -> bool:
- """Does `outer` leave room for the battery's worst case AND a bounded recovery?
-
- The budget is checked BEFORE dispatch, so the battery can run to
- `budget + case_timeout + 5` (the +5 is run_case's reap). _time_left() subtracts a
- further 35 s of fixed tail. A reflash needs RECOVER_FLASH_TIMEOUT beyond that.
- """
- worst_case_end = budget + case_timeout + 5
- return outer - worst_case_end - 35 >= RECOVER_FLASH_TIMEOUT
-```
-
-In `main()`, after parsing args:
-
-```python
- if args.budget and args.outer_timeout and not reserve_ok(
- args.budget, args.outer_timeout, args.timeout):
- print(f'warning: --outer-timeout {args.outer_timeout} leaves no room for a bounded '
- f'recovery after a --budget {args.budget} battery with --timeout '
- f'{args.timeout} cases; a HUNG board will be left wedged', file=sys.stderr)
-```
-
-Warn, do not exit: a caller that deliberately runs without recovery is legitimate.
-
-- [ ] **Step 4: Run test to verify it passes**
-
-Run: `cd test/hil && python3 test/test_hil_health.py RecoveryReserveIsChecked -v`
-Expected: PASS
-
-- [ ] **Step 5: Commit**
-
-```bash
-git add test/hil/usbtest.py test/hil/test/test_hil_health.py
-git commit -m "usbtest: check the recovery reserve instead of assuming it"
-```
-
----
-
-### Task 2: Derive the outer bound from one place
-
-**Files:**
-- Modify: `test/hil/hil_test.py` (constants block ~line 227, and `test_device_usbtest`)
-- Test: `test/hil/test/test_hil_bounded.py`
-
-**Interfaces:**
-- Consumes: `usbtest.reserve_ok` semantics (duplicate the arithmetic, do not import
- usbtest — `hil_test` must not import it).
-- Produces: an assertion at module import that the shipped constants satisfy the reserve.
-
-- [ ] **Step 1: Write the failing test**
-
-```python
- def test_the_shipped_constants_satisfy_the_reserve(self):
- """Whatever the env overrides, the pair hil_test computes must leave recovery room:
- outer - (budget + case_timeout + 5) - 35 >= 90."""
- outer = hil_test.USBTEST_BATTERY_BUDGET + hil_test.USBTEST_RECOVERY_BUDGET
- self.assertGreaterEqual(outer - (hil_test.USBTEST_BATTERY_BUDGET + 60 + 5) - 35, 90)
-```
-
-- [ ] **Step 2: Run test to verify it fails**
-
-Temporarily set `HIL_USBTEST_RECOVERY_BUDGET=100` and run; expect FAIL. Unset.
-
-- [ ] **Step 3: Add the guard**
-
-```python
-# The recovery reserve is a PROPERTY of these two, not a coincidence: the battery may
-# overshoot its budget by one already-started case (checked before dispatch), and a bounded
-# reflash needs 90 s after a 35 s fixed tail. Env overrides make this checkable at import
-# rather than discoverable when a wedge is left unrecovered.
-if USBTEST_RECOVERY_BUDGET - 60 - 5 - 35 < 90:
- print(f'warning: HIL_USBTEST_RECOVERY_BUDGET={USBTEST_RECOVERY_BUDGET} leaves no room '
- f'for a bounded reflash after a one-case overshoot; HUNG boards will stay wedged',
- file=sys.stderr)
-```
-
-- [ ] **Step 4: Run tests to verify they pass**
-
-Run: `cd test/hil && python3 test/test_hil_bounded.py -v`
-Expected: PASS
-
-- [ ] **Step 5: Commit**
-
-```bash
-git add test/hil/hil_test.py test/hil/test/test_hil_bounded.py
-git commit -m "hil: warn when the timeout constants leave no recovery reserve"
-```
diff --git a/docs/superpowers/followup/pr3840-mret-board-result.md b/docs/superpowers/followup/pr3840-mret-board-result.md
index 7b8da7b9c..77b76b605 100644
--- a/docs/superpowers/followup/pr3840-mret-board-result.md
+++ b/docs/superpowers/followup/pr3840-mret-board-result.md
@@ -3,6 +3,15 @@
**Origin:** split out of PR #3840 (making `hil_report.md` a rendering of `hil_report.json`).
Delete this file when its own PR lands.
+> **SUPERSEDED IN PART (2026-08-26).** Written against a 7-field tuple whose index 5 was
+> `blind`. The sysfs blindness subsystem is gone: `test_board` now returns **6** fields with
+> `stray` at index 5, and its board-locked early return is 5 wide. The problem described
+> below is unchanged and still worth fixing — three producers, three widths, and
+> `len(r) > 5 and r[5]` reads a WRONG SLOT rather than raising. But drop the `blind` field
+> from the proposed NamedTuple and re-derive every index from `hil_test.test_board` before
+> executing, or `_stray_note` starts reading a duration as a stray count.
+> `StrayNoteSurvivesTheTupleWidth` pins the current shape.
+
## What is established
`test_board()` returns a bare tuple that three producers build and fourteen call sites read
diff --git a/docs/superpowers/specs/2026-07-30-hil-usbtest-fleet-wedge-design.md b/docs/superpowers/specs/2026-07-30-hil-usbtest-fleet-wedge-design.md
index 3ed0c1519..a34848f06 100644
--- a/docs/superpowers/specs/2026-07-30-hil-usbtest-fleet-wedge-design.md
+++ b/docs/superpowers/specs/2026-07-30-hil-usbtest-fleet-wedge-design.md
@@ -1,9 +1,9 @@
# HIL fleet-wedge containment
Date: 2026-07-30
-Status: implemented, then superseded in part — addendum last checked 2026-08-12
-against the shipped code; where they disagree the CODE and the usb-kernel-recover
-skill win, never this document.
+Status: implemented, then superseded in part, then TRIMMED (2026-08-25 — see the
+addendum at the end). Last checked against the shipped code 2026-08-25; where they
+disagree the CODE and the usb-kernel-recover skill win, never this document.
- **Pool guard.** A single constant, not the flat 4200s below and not a derivation:
`POOL_TIMEOUT = pos_int_env('HIL_POOL_TIMEOUT', 3600)`. A per-controller model briefly
@@ -17,7 +17,9 @@ skill win, never this document.
- **Job ceilings.** 90/90/120 min (build.yml), not 60/60/90 and not the 85/115 below.
They must clear the 3600s guard plus the pre-pool checkout/artifact merge and the
post-guard sweep and report upload. No job pins `HIL_POOL_TIMEOUT`.
-- **Battery budgets.** `USBTEST_BATTERY_BUDGET` 260s, `USBTEST_RECOVERY_BUDGET` 250s.
+- **Battery budgets.** `USBTEST_BATTERY_BUDGET` 260s. The recovery reserve is no longer a
+ constant: `usbtest.recovery_reserve(flasher)` derives it per flasher (RP-target openocd 390s,
+ other openocd/jlink/stlink 190s, esptool/lm4flash 150s) — see the trim addendum.
The 200s-with-a-197s-floor derivation recorded here was never shipped; the floor
assertion was removed with it.
- **HUNG recovery.** Reflash of the DUT through its roster flasher
@@ -234,3 +236,108 @@ a stuck run and explain it without anyone touching the rig.
the board or flashing `board_test` by hand resolves it without any code change.
- **An unattended PVE watchdog** that detects the wedge and power-cycles the host.
Declined: more moving parts, and it can cut a running CI job.
+
+---
+
+## Trim addendum — 2026-08-25
+
+The containment above grew past what one maintainer could hold. This records what was
+removed and, more importantly, the rule that decided it, so the next reader does not
+re-derive the deleted layers from the incident above.
+
+### The dividing principle
+
+**The CI job ceiling bounds how long a run can burn. It does nothing about state that
+outlives the run.** Cut what the ceiling contains; keep what it does not.
+
+- Contained by the ceiling: a worker blocked on a wedged device. `drain_pool` keeps the
+ boards that finished, `_write_failed_spec` names the one in flight, `_abandon_exit`
+ writes and uploads the report, and the job dies at `timeout-minutes` regardless. The
+ cost is one pool slot.
+- **Not** contained: a D-state holder left on a usbfs node, or an unswept stray still
+ holding a probe. The job dies and those survive it, on a self-hosted runner, into the
+ next run. That is the original incident.
+
+### Removed
+
+- **The sysfs blindness subsystem.** `SYSFS_UNKNOWN`, the `_SysfsUnknown` sentinel, the
+ path→inode strand memo with its `_STRAND_MISS` miss-sentinel, the four-credit blindness
+ cap, `sysfs_blind()`/`sysfs_blind_note()`, `note_sysfs_strand()`, `bounded_open()`,
+ `usb_scan`'s `(list, bool)` return, usbtest's `inconclusive` abort, and `_blind_note`'s
+ report banner. `read_sysfs` is an ordinary `open().read()` returning `str | None`.
+
+ It was a three-valued contract five files had to reason about, and misreading unknown as
+ absence was silent — a healthy board reported as a firmware regression. It existed for
+ exactly one attribute that can block. Verified against v6.12.96 `sysfs.c`: only
+ `usb_string_attr` (`product`/`manufacturer`/`serial`, sysfs.c:141-143) takes
+ `usb_lock_device_interruptible`; `idVendor`, `idProduct`, `bcdDevice`, `busnum`,
+ `devnum` and `speed` are lock-free `sysfs_emit` from cached fields. Two of the five
+ `read_sysfs` call sites read attributes that cannot block at all.
+
+ **The bound stayed, and it is not opt-in.** An early cut of this trim made `read_sysfs`
+ unbounded on the theory that a blocked worker costs one pool slot. That is false:
+ `usb_scan` reads `serial` on every device matching the VID to find the one it wants, and
+ `hil_lock.controller_of` does exactly that from `controller_permit`, on essentially every
+ board — so one wedged DUT would stall *every* worker and the pool guard would take the
+ whole run. `read_sysfs` and `usb_scan` are bounded by `SYSFS_READ_GRACE` by default;
+ three call sites forgot an opt-in version within a single sitting, and a unit test now
+ pins the default.
+
+ What is gone is the *contract*, not the bound: no third value, no process-wide blindness
+ latch, no `(list, bool)` return, no report banner. A give-up reads as None like any
+ unreadable attribute, and the cost is confined to the device that is actually wedged.
+
+ **`hil_pool_check` is why the memo has to be exact.** It is a standalone
+ ThreadPoolExecutor tool with no guard behind it, run precisely when a device is suspected
+ wedged, and it polls (`wait_device` re-scans every 0.5 s). The bounded read gives up and
+ remembers
+ the path so a poll loop cannot leak a thread and an fd per pass. That memo is keyed by
+ **kernfs inode, not by path**: a busport does not change when a board returns to the same
+ physical port, so a path-only blacklist would outlive the wedge and make the tool's own
+ recovery flow (reset/reflash → `wait_device` polls for the new inode) never see the board
+ again. A changed inode is the all-clear; `os.stat` is safe on a wedged device because it
+ does not invoke `->show()`. A give-up reads as None
+ — the same as unreadable — and `sysfs_stranded()` lets the footer warn that a "missing"
+ row may be the tool losing sight of healthy hardware. One local bound with a warning
+ line, not the five-file three-valued contract that was removed.
+
+- **The recovery budget arithmetic.** `recovery_steps()`, `_time_left()` and its three
+ per-step gates. The reserve was an independent 250s — one number for the whole fleet —
+ that could not contain the ladder it
+ reserved for (reset 30 + reflash 90 + Rescue-DP POR 90 + retry 90 + settles), which is
+ why the child re-decided before every step — with a bare `- 35` for downstream costs
+ that nobody could re-derive. Between them they produced a recovery that skipped its own
+ steps for most real hangs. The reserve now counts `hil_util.REAP_GRACE` **per bounded
+ step** — `run_cmd` spends that reaping a child it had to SIGKILL, on top of the step's own
+ timeout — which is what the `- 35` was standing in for. Undersizing it is worse than not
+ recovering at all: the outer killpg lands mid-reflash and orphans the flasher on the
+ probe. A unit test asserts the reserve covers the ladder. `USBTEST_RECOVERY_BUDGET` is now derived from
+ `usbtest.RECOVER_*` **per flasher and per target**: the Rescue-DP legs are openocd-only
+ (`rescue_openocd` refuses anything else) and a stub reset is screened out, so an esptool
+ board no longer reserves 200s it can never spend. The child runs the ladder straight
+ through, and `--outer-timeout` — parsed but unused once the gates went — is deleted.
+
+### Deliberately kept
+
+- The pool guard, `drain_pool`, the re-run spec, `_abandon_exit`, the CI ceilings.
+- `hil_health`'s sweep **including** `_kill_and_confirm`. SIGKILL is queued, not delivered,
+ for a task in uninterruptible sleep, and a healthy in-flight testusb sits in exactly that
+ state — so `os.kill` returning success proves nothing, and the recheck is the only honest
+ answer to "is the rig dirty for the next job?".
+- usbtest's reset→check→reflash ladder and the `convoy_safe` gate. This is the only thing
+ that unpoisons the rig mid-run, and PR #3832 extends it from 11 to 18 of 27 boards.
+- `mtp_test.py` as a separate process — one job, a clean boundary, and runnable by hand
+ against a board while debugging.
+
+### Structural changes with no behaviour change
+
+- Blocking device IO now runs in a child process everywhere, not just where it was noticed
+ first. The printer WRITE half joined the read half (`usblp_open` ignores `O_NONBLOCK` and
+ stalls in `usb_autopm_get_interface()` holding the driver-global `usblp_mutex`), and the
+ HID echo followed (`hid.enumerate()` reads `manufacturer`/`product` for every HID device
+ it lists, both under the device lock). `test_device_midi_test` is NOT in that set: ALSA
+ rawmidi honours `O_NONBLOCK` on open (v6.12.96 rawmidi.c:489), unlike usblp.
+- `main()`'s two abort paths were near-identical 40-line blocks; `_abort_report` holds that
+ shape once. The controller-hint cache and pool construction moved to their own helpers.
+- The unit suite stopped sleeping 54 of its 78 seconds — mostly one named-and-zeroable
+ post-flash settle paid by ten tests against a fake rig.