summaryrefslogtreecommitdiff
path: root/test/hil/hil_test.py
diff options
context:
space:
mode:
authorhathach <[email protected]>2026-08-27 15:26:43 +0700
committerhathach <[email protected]>2026-08-28 14:18:41 +0700
commitb190840e1d85f806575f58635827502d714febcf (patch)
tree76c6abf45f3670313d5812f3c453d849a98371f9 /test/hil/hil_test.py
parenta5938f71fbc704c85979121bb9e37d5d636ef20d (diff)
test/hil: drop the sysfs blindness subsystem and derive the recovery reserve
Two layers whose cost was a contract to reason about rather than an outcome. SYSFS_UNKNOWN was a three-valued return five files had to keep apart, and misreading unknown as absence was silent: a healthy board reported as a firmware regression. What it guarded is real -- `serial` is served by usb_string_attr, which takes usb_lock_device_interruptible (v6.12.96 sysfs.c:141-143), the same lock a wedged usbfs ioctl holds -- so the BOUND stays, on every caller by default. usb_scan reads `serial` on every device matching the VID, and hil_lock's controller_of does that on essentially every board, so one wedged DUT would otherwise stall every worker, not one. What goes is the third value. read_sysfs now returns str or None, and the question the third value existed to answer is asked directly instead, by two predicates that say which question they answer: sysfs_stranded() is process-wide and sticky, for hil_pool_check's footer ("could anything here be the tool losing sight of healthy hardware?"), and path_stranded(path) is per-device, which is what usbtest needs to tell a DUT whose `serial` is held under device_lock from one that genuinely left the bus -- that difference decides whether it performs driver-registry writes that take the uninterruptible device_lock. Gone: _SysfsUnknown, SYSFS_UNKNOWN, sysfs_blind, sysfs_blind_note, note_sysfs_strand, the cross-process blindness publishing and its report banner, usb_scan's (list, bool) return, usbtest's inconclusive abort, _blind_note's slot in the result tuple, and bounded_open, whose last caller went in the previous commit. The strand memo is rewritten around the one invariant that makes it safe to reuse: it is keyed by the path's kernfs inode, captured BEFORE the read. A busport does not change when a board returns to the same physical port, so a path-only blacklist outlives the wedge and hil_pool_check's own recovery flow -- reset, reflash, wait_device polling that busport -- would never look at the board again. A re-enumeration destroys the kernfs node and makes a new one, so a changed inode is the all-clear. Two ceilings bound different things: per path (_PATH_STRAND_MAX) for a board that flaps while still wedged, and per process (_STRAND_MAX) as a backstop against RLIMIT_NOFILE, counted per PATH rather than per reader because hil_pool_check runs four poll threads over one bus. A board the pool guard never reached is now reported as run-aborted rather than pool-timed-out, and outranks a stale board-locked cell for the same reason the pool-timeout cell does. Both predicates answer conservatively where they are consulted before something irreversible. path_stranded() covers the paths read_sysfs answered None for WITHOUT reading -- past _STRAND_MAX it declines to start another reader, and vouching for a path nobody looked at hands usbtest's fail-CLOSED guard a fabricated all-clear, running remove_id/unbind against a wedged device. usbtest's startup lookup carries the same caveat hil_test's absent arm already did, because its stderr is relayed verbatim into the report cell. strand_note() survives the removal for the same reason master had it: every caller that can say "not found" needs the same sentence, and the one site left to re-invent it got missed -- a wedged-but-enumerated printer was reported as an enumeration failure, sending a maintainer after firmware. The two predicates are not interchangeable, and usbtest needs both. Its per-case verdict is per-DUT -- a peer that stranded at case 2 must not make our board report wedged at case 29 -- but the finally block's cleanup is process-wide: remove_id plus an unbind of EVERY interface under the driver, including that peer's, each taking the uninterruptible device_lock. So the verdict uses path_stranded() and the global cleanup stays gated on sysfs_stranded(). USBTEST_RECOVERY_BUDGET was an independent 250s that could not actually contain the ladder it reserved for, which is why usbtest.py carried a _time_left() gate re-deciding before every step -- with a bare '- 35' for costs paid downstream that nobody could re-derive. Between them the two produced a recovery that skipped its own steps for most real hangs. The reserve is now derived from the bounds usbtest itself declares, per flasher and per target: a probe reset, a reflash, and the Rescue-DP POR plus retry a wedged RP DAP needs, plus the settles and hil_util.REAP_GRACE for each bounded step. The Rescue-DP legs are openocd-only and gated on the RP target cfg, and a stub reset is screened out, so the reserve tracks each board's real ladder instead of one fleet number: 390s for the two RP boards -- whose ladder the old 250 could not contain, which is exactly why the gates skipped their steps -- 190s for the other seventeen probe-reset boards, and 150s for esptool and lm4flash, whose reset is a no-op. Changing a bound in usbtest moves the reserve with it, and a unit test asserts it covers the ladder. With the room actually reserved, the child runs the ladder straight through: recovery_steps, _time_left, the three per-step gates and the parsed-but-unused --outer-timeout are gone. What stays is what decides outcomes -- the convoy_safe gate, reset-before-reflash, the no_op screen so a stub that resets nothing is not claimed, and wedged_pids() as the arbiter, because a clean flash only proves the probe wrote the MCU. hil_util.py 616 -> 514 lines.
Diffstat (limited to 'test/hil/hil_test.py')
-rwxr-xr-xtest/hil/hil_test.py219
1 files changed, 116 insertions, 103 deletions
diff --git a/test/hil/hil_test.py b/test/hil/hil_test.py
index 51c4667e6..ac89bdd95 100755
--- a/test/hil/hil_test.py
+++ b/test/hil/hil_test.py
@@ -63,6 +63,7 @@ from multiprocessing import TimeoutError as MpTimeoutError
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # PYTHONSAFEPATH drops it
import hil_flash
+import usbtest # for the recovery bounds only; hil_test runs it as a subprocess
from helper import hil_health, hil_lock, hil_report, hil_util
from helper.hil_util import device_tests, dual_tests, host_test
@@ -220,13 +221,17 @@ class HilConfig(TypedDict):
POOL_TIMEOUT = hil_util.pos_int_env('HIL_POOL_TIMEOUT', 3600)
-# Headroom on top of a battery's own budget so ONE HUNG recovery (case timeout, SIGKILL
-# wait, bounded reflash, settle) can finish. Only spent when cases actually time out.
-USBTEST_RECOVERY_BUDGET = hil_util.pos_int_env('HIL_USBTEST_RECOVERY_BUDGET', 250)
+# The post-hang recovery reserve is PER BOARD and lives in usbtest.recovery_reserve(),
+# derived from the ladder that file itself declares. Reserved whole, which is what lets the
+# child run the ladder straight through instead of asking "does the next step still fit?"
+# before each step. It only ELAPSES when cases actually time out; a healthy battery returns
+# in ~200s and never touches it.
+
# How long usbtest.py may keep starting new cases (--budget). The outer run_cmd timeout is
-# always this PLUS the recovery headroom, never a separate literal, or lowering one eats
-# the reserve the recovery needs. 0 is refused (usbtest.py reads it as "no limit"); the
-# margin over a healthy battery (~200s) keeps contention from becoming BUDGET entries.
+# always this PLUS the overshoot PLUS the recovery reserve when one can run, never a
+# separate literal, or lowering one eats the room the other needs. 0 is refused (usbtest.py
+# reads it as "no limit"); the margin over a healthy battery (~200s) keeps contention from
+# becoming BUDGET entries.
USBTEST_BATTERY_BUDGET = hil_util.pos_int_env('HIL_USBTEST_BATTERY_BUDGET', 260)
# The battery checks its budget BEFORE dispatching a case, so it can overshoot by one
@@ -484,12 +489,8 @@ def get_printer_dev(id: str, vendor_str, product_str, ifnum: int):
product_str = product_str.replace(' ', '_') if product_str else ''
for lp in glob.glob('/sys/class/usbmisc/lp*'):
try:
- # bounded: same device_lock() exposure as the sibling reads (see read_sysfs)
sn = hil_util.read_sysfs(f'{lp}/device/../serial')
- # UNKNOWN is not None: the sentinel has no __eq__, so an unanswered read
- # would fall through both tests and read as 'not this board' -- the exact
- # absence/unknown conflation read_sysfs exists to prevent.
- if sn is None or sn is hil_util.SYSFS_UNKNOWN:
+ if sn is None:
continue
if sn == id:
return f'/dev/usb/{os.path.basename(lp)}'
@@ -506,7 +507,7 @@ def open_printer_dev(id: str, vendor_str, product_str, ifnum: int) -> str:
lp_dev = wait_until(try_find)
assert lp_dev, (f'Printer device not found for {id} if{ifnum:02d}'
- + hil_util.sysfs_blind_note())
+ + hil_util.strand_note())
return lp_dev
@@ -905,9 +906,9 @@ def test_device_cdc_msc_freertos(board):
def link_is_fs(speed) -> bool:
- """Payload scaling from a `speed` attribute. Anything not positively read as high speed
- counts as FS -- including None and SYSFS_UNKNOWN: the FS payload merely tests an HS
- board less, while the HS payload hard-fails a healthy FS board."""
+ """Payload scaling from a `speed` attribute. Anything not positively read as high
+ speed counts as FS, None included: the FS payload merely tests an HS board less, while
+ the HS payload hard-fails a healthy FS board."""
return speed not in ('480', '5000', '10000')
@@ -946,16 +947,15 @@ def test_device_cdc_msc_throughput(board):
# Detect speed (12 Mbps FS / 480 Mbps HS) for payload scaling; a device we never find
# keeps the FS payload (see link_is_fs)
- # usb_scan, not a private glob: it skips root hubs and remembers paths that already
- # stranded, so one wedged peer cannot spend this worker's blindness budget four reads
- # at a time.
+ # usb_scan, not a private glob: it skips root hubs and filters on the lock-free
+ # descriptor pair before touching `serial`.
is_fs = True
speed_known = False
- devs, _ = hil_util.usb_scan(vid='cafe', serial=uid)
+ devs = hil_util.usb_scan(vid='cafe', serial=uid)
if devs:
speed = hil_util.read_sysfs(os.path.join(devs[0]['dir'], 'speed'))
is_fs = link_is_fs(speed)
- speed_known = speed not in (None, hil_util.SYSFS_UNKNOWN)
+ speed_known = speed is not None
# Put tty in raw mode so dd sees pure binary throughput.
rs = hil_util.run_cmd(f'timeout 30 stty -F {tty} raw -echo')
@@ -1138,16 +1138,27 @@ def test_device_printer_to_cdc(board):
ready = Path(tempfile.gettempdir()) / f'hil-lp-ready-{os.getpid()}-{size}'
payload.write_bytes(test_data)
ready.unlink(missing_ok=True)
+ # +5 like write_cdc's sibling wait below: the bound is on the OPEN, and the child
+ # must first fork, exec and boot CPython, which on a loaded rig routinely exceeds
+ # LP_OPEN_TIMEOUT on its own. A tighter wait here reports a slow interpreter start
+ # as a wedged node.
+ open_deadline = time.monotonic() + LP_OPEN_TIMEOUT + 5
+ saw_ready = False
def read_cdc():
# WAIT for the writer to have the node open, as Test 2's write_cdc does: the
# child has to fork, exec and boot CPython, and reading before it starts just
# burns the serial timeout.
- deadline = time.monotonic() + LP_OPEN_TIMEOUT + 5
+ # ONE deadline, shared with the child's bound below. Two different ones let
+ # the writer open after the parent gave up: it writes the whole payload with
+ # nobody reading, exits 0, and the byte-compare reports FIRMWARE DATA
+ # CORRUPTION for a board whose only problem was a slow open.
+ nonlocal saw_ready
while not ready.exists():
- if time.monotonic() > deadline:
- return # writer never opened; the rc/compare below reports it
+ if time.monotonic() > open_deadline:
+ return # never opened; the assert below reports THAT, not data
time.sleep(0.02)
+ saw_ready = True
# fullspeed devices may need extra time; ser.read is bounded by
# SERIAL_READ_TIMEOUT, so an empty return means the stream went quiet
while len(rd) < size:
@@ -1163,9 +1174,32 @@ def test_device_printer_to_cdc(board):
finally:
ready.unlink(missing_ok=True)
payload.unlink(missing_ok=True)
- assert r.returncode == 0, (f'Printer->CDC writer failed ({size} bytes, rc '
- f'{r.returncode}): '
- f'{hil_util.cmd_stdout_text(r.stderr)[:200]}')
+ # rc 124 is run_alongside's kill, i.e. the open blocked -- and stderr is EMPTY
+ # there, so without the fallback the cell reads 'failed (32 bytes, rc 124):' and
+ # nothing, for the one failure this conversion exists to contain. An OSError is a
+ # FACT about the node (EBUSY from usblp's single-opener rule, ENOENT from a
+ # re-enumeration race) and must not send the operator to usb-kernel-recover.
+ # The bound covers the open AND the whole write, so rc 124 alone does not mean a
+ # wedged node. `ready` is written on the line after os.open() returns, so its
+ # ABSENCE is what says the open never completed -- the case that sends an operator
+ # to usb-kernel-recover. Anything else killed on the bound was a slow drain.
+ detail = hil_util.cmd_stdout_text(r.stderr).strip()[:200]
+ # FIRST: a child that exited on its OWN carries the concrete errno, and only one
+ # we KILLED (rc 124) can be diagnosed as an open that never completed. Asserting
+ # the marker before this reported EBUSY/ENOENT as a wedged node -- the conflation
+ # the comment above exists to prevent. rc is in the message because a child killed
+ # by a signal leaves `detail` empty.
+ assert r.returncode in (0, 124), (
+ f'Printer->CDC writer failed ({size} bytes, rc {r.returncode}): {detail}')
+ # saw_ready, not ready.exists(): a marker that appeared AFTER read_cdc gave up
+ # means the child wrote with nobody reading, and the byte-compare below would call
+ # that firmware data corruption. Report the slow open instead.
+ assert saw_ready, (f'printer: {lp_dev} was not opened for write within '
+ f'{LP_OPEN_TIMEOUT + 5}s (device wedged, or the writer never '
+ f'started); rc {r.returncode}')
+ assert r.returncode == 0, (
+ f'Printer->CDC writer killed on its bound after opening {lp_dev} '
+ f'(rc {r.returncode}): the firmware stopped draining the OUT endpoint')
assert bytes(rd) == test_data, (f'Printer->CDC wrong data ({size} bytes):\n'
f' expected: {test_data[:64]}\n'
f' received: {bytes(rd)[:64]}')
@@ -1462,31 +1496,26 @@ def test_device_usbtest(board):
uid = board['uid']
def usbtest_enumerated():
- """True, False, or None when a bounded read did not answer -- absence unproven."""
# vid_pid FIRST: right after flashing, the previous example's enumeration (same
# serial, different PID) can linger and would fail usbtest.py's lookup -- and
# filtering on the two lock-free descriptor fields rules out every other device
- # on the bus before the one read that can block. usb_scan memoises paths that
- # already stranded, so one wedged peer cannot spend the blindness budget here.
- devs, unknown = hil_util.usb_scan(vid_pid=('cafe', '4010'), serial=uid)
- if devs:
- return True
- return None if unknown else False
+ # on the bus before the one read that can block.
+ return bool(hil_util.usb_scan(vid_pid=('cafe', '4010'), serial=uid))
end = time.monotonic() + enum_timeout()
seen = usbtest_enumerated()
- while time.monotonic() < end and seen is not True:
+ while time.monotonic() < end and not seen:
time.sleep(0.2)
seen = usbtest_enumerated()
# fail before usbtest_permit: an absent device would otherwise queue on the battery
# mutex for minutes behind real batteries just to have usbtest.py report "no device"
- if seen is not True:
+ if not seen:
# 0/30 rather than a bare cell: the battery never ran (30 = standard case count)
- raise TestFail(
- f'no cafe:4010 device with serial {uid}' if seen is False else
- f'cannot tell whether cafe:4010 {uid} is present: the bounded sysfs reads did '
- f'not answer{hil_util.sysfs_blind_note()}',
- metric=f'{hil_report.REPORT_CELL["fail"]} 0/30')
+ # maxtasksperchild=1, so this worker only ever handled THIS board: a give-up here
+ # is about this device. Without the caveat a wedged-but-present DUT reads as a
+ # positive absence claim -- the conflation this whole path exists to avoid.
+ raise TestFail(f'no cafe:4010 device with serial {uid}{hil_util.strand_note()}',
+ metric=f'{hil_report.REPORT_CELL["fail"]} 0/30')
# settle: right after flashing the enumeration can bounce once (and on dual-port parts
# the other port's stale node — same serial and PID — lingers), and testusb run into
# that gap sees the device drop mid-case
@@ -1506,8 +1535,9 @@ def test_device_usbtest(board):
# Post-hang recovery reflashes the DUT through its own probe, NEVER a root-port cycle
# (one board reached instead of every fixture under the port; see usb-kernel-recover).
# _current_fw is the artifact test_example flashed for THIS test: re-deriving it from
- # board['name'] reflashes the wrong build on variant-only boards. --outer-timeout lets
- # usbtest skip a reflash it cannot finish before our run_cmd kill, which would orphan
+ # board['name'] reflashes the wrong build on variant-only boards. Our run_cmd bound
+ # below RESERVES the whole ladder (usbtest.recovery_reserve), which is what lets the
+ # child run it straight through without an outer kill landing mid-flash and orphaning
# the flasher (own session) on the probe. Never under --skip-flash -- and say so: a
# HUNG case then holds the DUT's usbfs lock for the rest of the run, and a probe reset
# is no substitute (the DWC2 pullup survives a core halt).
@@ -1519,14 +1549,12 @@ def test_device_usbtest(board):
# same probe convoy-safely without changing how the board is normally flashed.
_rec_flasher = hil_flash.recover_flasher(board)
recovery = bool(_current_fw and not skip_flash and hil_flash.convoy_safe(_rec_flasher))
- # ONE bound, computed here and used for BOTH the child's --outer-timeout and our own
- # run_cmd kill below. Three separate expressions disagreed: --skip-flash appended no
- # --outer-timeout at all (usbtest reads 0 as "no limit"), and the no-recovery branch
- # narrowed only the CHILD's view while run_cmd still waited the full reserve -- so a
- # board that cannot recover held a pool worker AND its battery permit idle for
- # USBTEST_RECOVERY_BUDGET it had no way to spend, under a usbtest width of 2.
- outer = USBTEST_BATTERY_BUDGET + (USBTEST_RECOVERY_BUDGET if recovery
- else USBTEST_OVERSHOOT)
+ # ONE bound: run_cmd's kill below. It carries the recovery reserve only when a
+ # recovery can actually run, and only what THIS flasher's ladder can spend -- a board
+ # that cannot recover used to hold a pool worker AND its battery permit idle for a
+ # reserve it had no way to spend, under a usbtest width of 2.
+ outer = USBTEST_BATTERY_BUDGET + USBTEST_OVERSHOOT + (
+ usbtest.recovery_reserve(_rec_flasher) if recovery else 0)
if _current_fw and skip_flash:
print('note: --skip-flash disables usbtest hang recovery; a HUNG case will leave '
'the device wedged until it is reflashed', flush=True)
@@ -1540,7 +1568,6 @@ def test_device_usbtest(board):
# child side unaware that a second roster entry exists
rb = json.dumps({'name': board['name'], 'flasher': _rec_flasher})
cmd += f' --recover-board {shlex.quote(rb)} --recover-fw {shlex.quote(_current_fw)}'
- cmd += f' --outer-timeout {outer}'
# The reserve above USBTEST_BATTERY_BUDGET exists because the battery can overrun by
# one already-started case, and a hang there needs room for the recovery (whose reflash
# is bounded by usbtest.RECOVER_FLASH_TIMEOUT, not HIL_CMD_TIMEOUT). Without it run_cmd
@@ -1604,7 +1631,7 @@ def _usbtest_verdict(board: Board, data: dict, out: str, passed: int, failed: in
# in the caller, and the two diverge as soon as a roster carries the
# optional `flasher_recover` key -- naming the wrong one sends the operator to the
# wrong probe. The wording stays on what usbtest actually reported ("still wedged"),
- # because unrecovered_hang is also set by the ambiguous/inconclusive aborts, where
+ # because unrecovered_hang is also set by the ambiguous abort, where
# nothing hung and the old text was false on both clauses.
board_wedged = (f'{board["name"]}: usbtest reports the device still wedged '
+ (f'after a recovery reflash via {rec_flasher["name"]}' if recovery
@@ -1615,7 +1642,7 @@ def _usbtest_verdict(board: Board, data: dict, out: str, passed: int, failed: in
notrun = int(data.get('notrun', 0))
total = passed + failed + notrun
if board_wedged and failed == 0 and notrun == 0:
- # Every case passed and the device STILL wedged -- usbtest's inconclusive/ambiguous
+ # Every case passed and the device STILL wedged -- usbtest's ambiguous
# abort fires after the last case, so nothing back-fills a BUDGET entry. Reporting
# the pass would exit 0 with a D-state holder on the rig and the board absent from
# the re-run spec. parsed=True: a retry re-pays the whole battery to re-observe a
@@ -1880,10 +1907,9 @@ def _tests_for(board: Board) -> list:
def test_board(board: Board) -> tuple:
- # (name, err_count, failed_tests, rows, duration[, blind, strays]) -- the board-LOCKED
- # early return is 5 wide, the normal one 7. _blind_note and _stray_note index 5 and 6
- # behind a len() guard, so a field inserted before them reads a WRONG SLOT rather than
- # raising: a duration would report as a stray count.
+ # (name, err_count, failed_tests, rows, duration[, strays]) -- the board-LOCKED early
+ # return is 5 wide, the normal one 6. _stray_note reads index 5 behind a len() guard,
+ # so a field inserted anywhere before it silently reports a duration as a stray count.
swept = False
name = board['name']
flasher = board['flasher']
@@ -2009,12 +2035,10 @@ def test_board(board: Board) -> tuple:
stray = hil_health.kill_own_children()
swept = True
- # LAST fields: whether this worker ran out of bounded-read budget, and what it could
- # not kill. Only the worker can answer either -- the blindness latch is
- # process-global and this is a separate process -- and the result tuple already
- # crosses back, so no Manager round-trip.
+ # LAST field: what this worker could not kill. Only the worker can answer it, and
+ # the result tuple already crosses back, so no Manager round-trip.
return (name, err_count, [] if board_wide_fail else sorted(set(failed_tests)),
- rows, t_total, hil_util.sysfs_blind(), stray)
+ rows, t_total, stray)
finally:
# A raise skips the sweep above, and maxtasksperchild=1 retires this process
# immediately afterwards -- reparenting its flasher to init and erasing the ppid
@@ -2133,7 +2157,7 @@ def _stray_note(mret: list) -> str:
runs AFTER accumulate_report on both abort paths, so a banner appended there was
written to a variable nobody read again.
"""
- dirty = [(r[0], r[6]) for r in mret if len(r) > 6 and r[6]]
+ dirty = [(r[0], r[5]) for r in mret if len(r) > 5 and r[5]]
if not dirty:
return ''
total = sum(n for _, n in dirty)
@@ -2142,28 +2166,6 @@ def _stray_note(mret: list) -> str:
f'{", ".join(f"{b} ({n})" for b, n in dirty)}.\n')
-def _blind_note(mret: list) -> str:
- """Name the boards whose worker went blind, for the report banner.
-
- A blind worker answers SYSFS_UNKNOWN for every attribute, so its "device not found" is
- "could not tell". That already reaches the log and the per-cell failure text, but the
- TABLE is what gets quoted -- and a red cell there is read as a broken board. Seen live
- (run 31794359407): four workers blind, several cells red because of it, and a report
- that said nothing.
-
- Per-board, not global: maxtasksperchild=1 gives every board a fresh worker, so a board
- that ran on a healthy one is not smeared by a neighbour's wedge. Rows synthesised by
- the timeout path are 5 fields wide and have nothing to report.
- """
- blind = [r[0] for r in mret if len(r) > 5 and r[5]]
- if not blind:
- return ''
- return (f'> **Not all verdicts are evidence.** {len(blind)} board(s) ran on a worker '
- f'that went blind on sysfs -- too many bounded reads stranded on a wedged '
- f'device -- so "not found" from them means "could not tell": '
- f'{", ".join(blind)}. See the usb-kernel-recover skill.\n')
-
-
# containment paths print through hil_health._p: stdout may already be a dead pipe (a
# dropped ssh session), and a BrokenPipeError there would skip os._exit
_p = hil_health._p
@@ -2261,8 +2263,8 @@ def _save_controller_hints(hints: dict, mret: list, uid_of: dict, cmap) -> None:
merged = {k: v for k, v in cur.items() if isinstance(v, dict)}
except (OSError, ValueError):
pass
- # onto what the CACHE now holds, not onto our startup snapshot: another HIL job may
- # have written a newer duration/pci for these boards since we read it
+ # overlay onto what the CACHE now holds, not onto our startup snapshot: another HIL
+ # job may have written a newer duration/pci for these boards since we read it
for name, *_ in mret:
uid = uid_of.get(name)
if uid is not None and uid in hints:
@@ -2296,41 +2298,49 @@ def _abort_report(reason: str, mret: list, config_boards: list, failed_fname: Pa
_write_failed_spec(failed_fname, report_dir,
[(n, 1, [], None, 0) for n in stuck]
+ [r for r in mret if r[1] > 0])
- except Exception as werr: # noqa: BLE001 - it mkdir()s and open()s the report dir
- # letting it raise here REPLACES the caller's RuntimeError, so the operator never
- # sees the 'pool timed out' line and no report is written at all
+ except Exception as werr: # noqa: BLE001 - it mkdir()s and open()s the very report dir
+ # the fallback below is FOR an unwritable/root-owned report dir; letting the spec
+ # raise here replaces the caller's RuntimeError, so the operator never sees the
+ # 'pool timed out' line and no report is written at all
print(f'warning: re-run spec failed: {type(werr).__name__}: {werr}', flush=True)
banner = (f"**HIL run {reason}.** {len(mret)} board(s) below finished and are this "
f"run's; {len(stuck)} never reported and are NOT in the table: "
f"{', '.join(stuck)}. The re-run spec covers those.\n")
try:
hil_report.accumulate_report(mret, report_dir, fresh, '',
- health_banner + _blind_note(mret)
- + _stray_note(mret), caveat=banner)
+ health_banner + _stray_note(mret), caveat=banner)
return
except Exception as rerr: # noqa: BLE001 - the caller's raise must still happen
print(f'warning: partial report failed: {type(rerr).__name__}: {rerr}'
- + ('; falling back to the board list' if timeout_secs else ''), flush=True)
- if timeout_secs is None:
- return
+ + '; falling back to the board list', flush=True)
try:
+ # banner=, or write_timeout_report's default caveat publishes 'No per-board
+ # results could be collected' onto a report where mret DID hold finished rows
+ # the CELL names the cause: a board the pool guard never reached did not
+ # "pool-timeout", and marking it so sends the reader after a guard that did not fire
hil_report.write_timeout_report(
report_dir, [b for b in config_boards if b['name'] in stuck],
- timeout_secs, prefix=health_banner)
+ timeout_secs or 0, banner=banner, prefix=health_banner,
+ cell=(hil_report.POOL_TIMEOUT_CELL if timeout_secs
+ else hil_report.RUN_ABORTED_CELL))
except Exception as re2: # noqa: BLE001
print(f'warning: fallback report failed too: {type(re2).__name__}: {re2}',
flush=True)
-def _start_pool(seed: str, hints_by_uid: dict):
- """(mgr, cmap, pool). Split out so main()'s try/finally reads as one shape.
+def _start_pool(mgr, seed: str, hints_by_uid: dict):
+ """(cmap, pool). Split out so main()'s try/finally reads as one shape.
+
+ The Manager is created by the CALLER and passed in: Pool() forks, and after a convoy
+ that fork is what hits EAGAIN/ENOMEM. Creating the Manager here too would leave main()
+ with `mgr` still None while a live SyncManager child exists -- os._exit skips its
+ finalizer and the orphan holds the runner's stdout, so the job step never completes.
maxtasksperchild=1: a fresh worker per board makes cross-board contamination
structural rather than dependent on every module global being reset by hand
(board_wedged, _current_fw, hil_flash's warn-once sets). The extra fork is noise
against a flash+test cycle.
"""
- mgr = Manager()
cmap = mgr.dict()
initargs = (Lock(), seed,
hil_lock.make_permit_sems(Semaphore, hil_lock.USBTEST_PARALLEL),
@@ -2338,7 +2348,7 @@ def _start_pool(seed: str, hints_by_uid: dict):
cmap, Lock(), hints_by_uid)
pool = Pool(processes=os.cpu_count() or 1, initializer=init_worker,
initargs=initargs, maxtasksperchild=1)
- return mgr, cmap, pool
+ return cmap, pool
def main() -> None:
@@ -2549,7 +2559,11 @@ def main() -> None:
(report_dir / f).unlink(missing_ok=True)
failed_fname.unlink(missing_ok=True)
try:
- mgr, cmap, pool = _start_pool(seed, hints_by_uid)
+ # BOUND FIRST, in main's own scope: a Pool fork failure inside _start_pool must
+ # still leave a live Manager reachable by the finally below, or its child is
+ # orphaned holding the runner's stdout.
+ mgr = Manager()
+ cmap, pool = _start_pool(mgr, seed, hints_by_uid)
# OUTER: encloses the pool block too, not just the reporting below. An exception
# escaping async_ret.get() (a worker exception, a Ctrl-C) runs the pool finally and
# then propagates straight out of main(); with _abandon_exit in a sibling try it
@@ -2648,8 +2662,7 @@ def main() -> None:
scoped = sorted(set(args.board) | set(board_test))
scope = f'{len(scoped)} board(s) — {", ".join(scoped)}' if scoped else ''
report = hil_report.accumulate_report(mret, report_dir, fresh, scope,
- health_banner + _blind_note(mret)
- + _stray_note(mret))
+ health_banner + _stray_note(mret))
print()
print(report)
print(f'\nReport written to {(report_dir / hil_report.REPORT_MD).resolve()}')