diff options
| author | hathach <[email protected]> | 2026-08-27 15:26:43 +0700 |
|---|---|---|
| committer | hathach <[email protected]> | 2026-08-28 14:18:41 +0700 |
| commit | b190840e1d85f806575f58635827502d714febcf (patch) | |
| tree | 76c6abf45f3670313d5812f3c453d849a98371f9 /test/hil/hil_test.py | |
| parent | a5938f71fbc704c85979121bb9e37d5d636ef20d (diff) | |
test/hil: drop the sysfs blindness subsystem and derive the recovery reserve
Two layers whose cost was a contract to reason about rather than an outcome.
SYSFS_UNKNOWN was a three-valued return five files had to keep apart, and
misreading unknown as absence was silent: a healthy board reported as a firmware
regression. What it guarded is real -- `serial` is served by usb_string_attr,
which takes usb_lock_device_interruptible (v6.12.96 sysfs.c:141-143), the same
lock a wedged usbfs ioctl holds -- so the BOUND stays, on every caller by
default. usb_scan reads `serial` on every device matching the VID, and hil_lock's
controller_of does that on essentially every board, so one wedged DUT would
otherwise stall every worker, not one. What goes is the third value.
read_sysfs now returns str or None, and the question the third value existed to
answer is asked directly instead, by two predicates that say which question they
answer: sysfs_stranded() is process-wide and sticky, for hil_pool_check's footer
("could anything here be the tool losing sight of healthy hardware?"), and
path_stranded(path) is per-device, which is what usbtest needs to tell a DUT
whose `serial` is held under device_lock from one that genuinely left the bus --
that difference decides whether it performs driver-registry writes that take the
uninterruptible device_lock.
Gone: _SysfsUnknown, SYSFS_UNKNOWN, sysfs_blind, sysfs_blind_note,
note_sysfs_strand, the cross-process blindness publishing and its report banner,
usb_scan's (list, bool) return, usbtest's inconclusive abort, _blind_note's slot
in the result tuple, and bounded_open, whose last caller went in the previous
commit.
The strand memo is rewritten around the one invariant that makes it safe to
reuse: it is keyed by the path's kernfs inode, captured BEFORE the read. A
busport does not change when a board returns to the same physical port, so a
path-only blacklist outlives the wedge and hil_pool_check's own recovery flow --
reset, reflash, wait_device polling that busport -- would never look at the
board again. A re-enumeration destroys the kernfs node and makes a new one, so a
changed inode is the all-clear. Two ceilings bound different things: per path
(_PATH_STRAND_MAX) for a board that flaps while still wedged, and per process
(_STRAND_MAX) as a backstop against RLIMIT_NOFILE, counted per PATH rather than
per reader because hil_pool_check runs four poll threads over one bus.
A board the pool guard never reached is now reported as run-aborted rather than
pool-timed-out, and outranks a stale board-locked cell for the same reason the
pool-timeout cell does.
Both predicates answer conservatively where they are consulted before something
irreversible. path_stranded() covers the paths read_sysfs answered None for
WITHOUT reading -- past _STRAND_MAX it declines to start another reader, and
vouching for a path nobody looked at hands usbtest's fail-CLOSED guard a
fabricated all-clear, running remove_id/unbind against a wedged device. usbtest's
startup lookup carries the same caveat hil_test's absent arm already did, because
its stderr is relayed verbatim into the report cell.
strand_note() survives the removal for the same reason master had it: every
caller that can say "not found" needs the same sentence, and the one site left to
re-invent it got missed -- a wedged-but-enumerated printer was reported as an
enumeration failure, sending a maintainer after firmware.
The two predicates are not interchangeable, and usbtest needs both. Its per-case
verdict is per-DUT -- a peer that stranded at case 2 must not make our board
report wedged at case 29 -- but the finally block's cleanup is process-wide:
remove_id plus an unbind of EVERY interface under the driver, including that
peer's, each taking the uninterruptible device_lock. So the verdict uses
path_stranded() and the global cleanup stays gated on sysfs_stranded().
USBTEST_RECOVERY_BUDGET was an independent 250s that could not actually contain
the ladder it reserved for, which is why usbtest.py carried a _time_left() gate
re-deciding before every step -- with a bare '- 35' for costs paid downstream
that nobody could re-derive. Between them the two produced a recovery that
skipped its own steps for most real hangs.
The reserve is now derived from the bounds usbtest itself declares, per flasher
and per target: a probe reset, a reflash, and the Rescue-DP POR plus retry a
wedged RP DAP needs, plus the settles and hil_util.REAP_GRACE for each bounded
step. The Rescue-DP legs are openocd-only and gated on the RP target cfg, and
a stub reset is screened out, so the reserve tracks each board's real ladder
instead of one fleet number: 390s for the two RP boards -- whose ladder the old
250 could not contain, which is exactly why the gates skipped their steps --
190s for the other seventeen probe-reset boards, and 150s for esptool and
lm4flash, whose reset is a no-op. Changing a bound in usbtest moves the
reserve with it, and a unit test asserts it covers the ladder.
With the room actually reserved, the child runs the ladder straight through:
recovery_steps, _time_left, the three per-step gates and the parsed-but-unused
--outer-timeout are gone. What stays is what decides outcomes -- the convoy_safe
gate, reset-before-reflash, the no_op screen so a stub that resets nothing is not
claimed, and wedged_pids() as the arbiter, because a clean flash only proves the
probe wrote the MCU.
hil_util.py 616 -> 514 lines.
Diffstat (limited to 'test/hil/hil_test.py')
| -rwxr-xr-x | test/hil/hil_test.py | 219 |
1 files changed, 116 insertions, 103 deletions
diff --git a/test/hil/hil_test.py b/test/hil/hil_test.py index 51c4667e6..ac89bdd95 100755 --- a/test/hil/hil_test.py +++ b/test/hil/hil_test.py @@ -63,6 +63,7 @@ from multiprocessing import TimeoutError as MpTimeoutError sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # PYTHONSAFEPATH drops it import hil_flash +import usbtest # for the recovery bounds only; hil_test runs it as a subprocess from helper import hil_health, hil_lock, hil_report, hil_util from helper.hil_util import device_tests, dual_tests, host_test @@ -220,13 +221,17 @@ class HilConfig(TypedDict): POOL_TIMEOUT = hil_util.pos_int_env('HIL_POOL_TIMEOUT', 3600) -# Headroom on top of a battery's own budget so ONE HUNG recovery (case timeout, SIGKILL -# wait, bounded reflash, settle) can finish. Only spent when cases actually time out. -USBTEST_RECOVERY_BUDGET = hil_util.pos_int_env('HIL_USBTEST_RECOVERY_BUDGET', 250) +# The post-hang recovery reserve is PER BOARD and lives in usbtest.recovery_reserve(), +# derived from the ladder that file itself declares. Reserved whole, which is what lets the +# child run the ladder straight through instead of asking "does the next step still fit?" +# before each step. It only ELAPSES when cases actually time out; a healthy battery returns +# in ~200s and never touches it. + # How long usbtest.py may keep starting new cases (--budget). The outer run_cmd timeout is -# always this PLUS the recovery headroom, never a separate literal, or lowering one eats -# the reserve the recovery needs. 0 is refused (usbtest.py reads it as "no limit"); the -# margin over a healthy battery (~200s) keeps contention from becoming BUDGET entries. +# always this PLUS the overshoot PLUS the recovery reserve when one can run, never a +# separate literal, or lowering one eats the room the other needs. 0 is refused (usbtest.py +# reads it as "no limit"); the margin over a healthy battery (~200s) keeps contention from +# becoming BUDGET entries. USBTEST_BATTERY_BUDGET = hil_util.pos_int_env('HIL_USBTEST_BATTERY_BUDGET', 260) # The battery checks its budget BEFORE dispatching a case, so it can overshoot by one @@ -484,12 +489,8 @@ def get_printer_dev(id: str, vendor_str, product_str, ifnum: int): product_str = product_str.replace(' ', '_') if product_str else '' for lp in glob.glob('/sys/class/usbmisc/lp*'): try: - # bounded: same device_lock() exposure as the sibling reads (see read_sysfs) sn = hil_util.read_sysfs(f'{lp}/device/../serial') - # UNKNOWN is not None: the sentinel has no __eq__, so an unanswered read - # would fall through both tests and read as 'not this board' -- the exact - # absence/unknown conflation read_sysfs exists to prevent. - if sn is None or sn is hil_util.SYSFS_UNKNOWN: + if sn is None: continue if sn == id: return f'/dev/usb/{os.path.basename(lp)}' @@ -506,7 +507,7 @@ def open_printer_dev(id: str, vendor_str, product_str, ifnum: int) -> str: lp_dev = wait_until(try_find) assert lp_dev, (f'Printer device not found for {id} if{ifnum:02d}' - + hil_util.sysfs_blind_note()) + + hil_util.strand_note()) return lp_dev @@ -905,9 +906,9 @@ def test_device_cdc_msc_freertos(board): def link_is_fs(speed) -> bool: - """Payload scaling from a `speed` attribute. Anything not positively read as high speed - counts as FS -- including None and SYSFS_UNKNOWN: the FS payload merely tests an HS - board less, while the HS payload hard-fails a healthy FS board.""" + """Payload scaling from a `speed` attribute. Anything not positively read as high + speed counts as FS, None included: the FS payload merely tests an HS board less, while + the HS payload hard-fails a healthy FS board.""" return speed not in ('480', '5000', '10000') @@ -946,16 +947,15 @@ def test_device_cdc_msc_throughput(board): # Detect speed (12 Mbps FS / 480 Mbps HS) for payload scaling; a device we never find # keeps the FS payload (see link_is_fs) - # usb_scan, not a private glob: it skips root hubs and remembers paths that already - # stranded, so one wedged peer cannot spend this worker's blindness budget four reads - # at a time. + # usb_scan, not a private glob: it skips root hubs and filters on the lock-free + # descriptor pair before touching `serial`. is_fs = True speed_known = False - devs, _ = hil_util.usb_scan(vid='cafe', serial=uid) + devs = hil_util.usb_scan(vid='cafe', serial=uid) if devs: speed = hil_util.read_sysfs(os.path.join(devs[0]['dir'], 'speed')) is_fs = link_is_fs(speed) - speed_known = speed not in (None, hil_util.SYSFS_UNKNOWN) + speed_known = speed is not None # Put tty in raw mode so dd sees pure binary throughput. rs = hil_util.run_cmd(f'timeout 30 stty -F {tty} raw -echo') @@ -1138,16 +1138,27 @@ def test_device_printer_to_cdc(board): ready = Path(tempfile.gettempdir()) / f'hil-lp-ready-{os.getpid()}-{size}' payload.write_bytes(test_data) ready.unlink(missing_ok=True) + # +5 like write_cdc's sibling wait below: the bound is on the OPEN, and the child + # must first fork, exec and boot CPython, which on a loaded rig routinely exceeds + # LP_OPEN_TIMEOUT on its own. A tighter wait here reports a slow interpreter start + # as a wedged node. + open_deadline = time.monotonic() + LP_OPEN_TIMEOUT + 5 + saw_ready = False def read_cdc(): # WAIT for the writer to have the node open, as Test 2's write_cdc does: the # child has to fork, exec and boot CPython, and reading before it starts just # burns the serial timeout. - deadline = time.monotonic() + LP_OPEN_TIMEOUT + 5 + # ONE deadline, shared with the child's bound below. Two different ones let + # the writer open after the parent gave up: it writes the whole payload with + # nobody reading, exits 0, and the byte-compare reports FIRMWARE DATA + # CORRUPTION for a board whose only problem was a slow open. + nonlocal saw_ready while not ready.exists(): - if time.monotonic() > deadline: - return # writer never opened; the rc/compare below reports it + if time.monotonic() > open_deadline: + return # never opened; the assert below reports THAT, not data time.sleep(0.02) + saw_ready = True # fullspeed devices may need extra time; ser.read is bounded by # SERIAL_READ_TIMEOUT, so an empty return means the stream went quiet while len(rd) < size: @@ -1163,9 +1174,32 @@ def test_device_printer_to_cdc(board): finally: ready.unlink(missing_ok=True) payload.unlink(missing_ok=True) - assert r.returncode == 0, (f'Printer->CDC writer failed ({size} bytes, rc ' - f'{r.returncode}): ' - f'{hil_util.cmd_stdout_text(r.stderr)[:200]}') + # rc 124 is run_alongside's kill, i.e. the open blocked -- and stderr is EMPTY + # there, so without the fallback the cell reads 'failed (32 bytes, rc 124):' and + # nothing, for the one failure this conversion exists to contain. An OSError is a + # FACT about the node (EBUSY from usblp's single-opener rule, ENOENT from a + # re-enumeration race) and must not send the operator to usb-kernel-recover. + # The bound covers the open AND the whole write, so rc 124 alone does not mean a + # wedged node. `ready` is written on the line after os.open() returns, so its + # ABSENCE is what says the open never completed -- the case that sends an operator + # to usb-kernel-recover. Anything else killed on the bound was a slow drain. + detail = hil_util.cmd_stdout_text(r.stderr).strip()[:200] + # FIRST: a child that exited on its OWN carries the concrete errno, and only one + # we KILLED (rc 124) can be diagnosed as an open that never completed. Asserting + # the marker before this reported EBUSY/ENOENT as a wedged node -- the conflation + # the comment above exists to prevent. rc is in the message because a child killed + # by a signal leaves `detail` empty. + assert r.returncode in (0, 124), ( + f'Printer->CDC writer failed ({size} bytes, rc {r.returncode}): {detail}') + # saw_ready, not ready.exists(): a marker that appeared AFTER read_cdc gave up + # means the child wrote with nobody reading, and the byte-compare below would call + # that firmware data corruption. Report the slow open instead. + assert saw_ready, (f'printer: {lp_dev} was not opened for write within ' + f'{LP_OPEN_TIMEOUT + 5}s (device wedged, or the writer never ' + f'started); rc {r.returncode}') + assert r.returncode == 0, ( + f'Printer->CDC writer killed on its bound after opening {lp_dev} ' + f'(rc {r.returncode}): the firmware stopped draining the OUT endpoint') assert bytes(rd) == test_data, (f'Printer->CDC wrong data ({size} bytes):\n' f' expected: {test_data[:64]}\n' f' received: {bytes(rd)[:64]}') @@ -1462,31 +1496,26 @@ def test_device_usbtest(board): uid = board['uid'] def usbtest_enumerated(): - """True, False, or None when a bounded read did not answer -- absence unproven.""" # vid_pid FIRST: right after flashing, the previous example's enumeration (same # serial, different PID) can linger and would fail usbtest.py's lookup -- and # filtering on the two lock-free descriptor fields rules out every other device - # on the bus before the one read that can block. usb_scan memoises paths that - # already stranded, so one wedged peer cannot spend the blindness budget here. - devs, unknown = hil_util.usb_scan(vid_pid=('cafe', '4010'), serial=uid) - if devs: - return True - return None if unknown else False + # on the bus before the one read that can block. + return bool(hil_util.usb_scan(vid_pid=('cafe', '4010'), serial=uid)) end = time.monotonic() + enum_timeout() seen = usbtest_enumerated() - while time.monotonic() < end and seen is not True: + while time.monotonic() < end and not seen: time.sleep(0.2) seen = usbtest_enumerated() # fail before usbtest_permit: an absent device would otherwise queue on the battery # mutex for minutes behind real batteries just to have usbtest.py report "no device" - if seen is not True: + if not seen: # 0/30 rather than a bare cell: the battery never ran (30 = standard case count) - raise TestFail( - f'no cafe:4010 device with serial {uid}' if seen is False else - f'cannot tell whether cafe:4010 {uid} is present: the bounded sysfs reads did ' - f'not answer{hil_util.sysfs_blind_note()}', - metric=f'{hil_report.REPORT_CELL["fail"]} 0/30') + # maxtasksperchild=1, so this worker only ever handled THIS board: a give-up here + # is about this device. Without the caveat a wedged-but-present DUT reads as a + # positive absence claim -- the conflation this whole path exists to avoid. + raise TestFail(f'no cafe:4010 device with serial {uid}{hil_util.strand_note()}', + metric=f'{hil_report.REPORT_CELL["fail"]} 0/30') # settle: right after flashing the enumeration can bounce once (and on dual-port parts # the other port's stale node — same serial and PID — lingers), and testusb run into # that gap sees the device drop mid-case @@ -1506,8 +1535,9 @@ def test_device_usbtest(board): # Post-hang recovery reflashes the DUT through its own probe, NEVER a root-port cycle # (one board reached instead of every fixture under the port; see usb-kernel-recover). # _current_fw is the artifact test_example flashed for THIS test: re-deriving it from - # board['name'] reflashes the wrong build on variant-only boards. --outer-timeout lets - # usbtest skip a reflash it cannot finish before our run_cmd kill, which would orphan + # board['name'] reflashes the wrong build on variant-only boards. Our run_cmd bound + # below RESERVES the whole ladder (usbtest.recovery_reserve), which is what lets the + # child run it straight through without an outer kill landing mid-flash and orphaning # the flasher (own session) on the probe. Never under --skip-flash -- and say so: a # HUNG case then holds the DUT's usbfs lock for the rest of the run, and a probe reset # is no substitute (the DWC2 pullup survives a core halt). @@ -1519,14 +1549,12 @@ def test_device_usbtest(board): # same probe convoy-safely without changing how the board is normally flashed. _rec_flasher = hil_flash.recover_flasher(board) recovery = bool(_current_fw and not skip_flash and hil_flash.convoy_safe(_rec_flasher)) - # ONE bound, computed here and used for BOTH the child's --outer-timeout and our own - # run_cmd kill below. Three separate expressions disagreed: --skip-flash appended no - # --outer-timeout at all (usbtest reads 0 as "no limit"), and the no-recovery branch - # narrowed only the CHILD's view while run_cmd still waited the full reserve -- so a - # board that cannot recover held a pool worker AND its battery permit idle for - # USBTEST_RECOVERY_BUDGET it had no way to spend, under a usbtest width of 2. - outer = USBTEST_BATTERY_BUDGET + (USBTEST_RECOVERY_BUDGET if recovery - else USBTEST_OVERSHOOT) + # ONE bound: run_cmd's kill below. It carries the recovery reserve only when a + # recovery can actually run, and only what THIS flasher's ladder can spend -- a board + # that cannot recover used to hold a pool worker AND its battery permit idle for a + # reserve it had no way to spend, under a usbtest width of 2. + outer = USBTEST_BATTERY_BUDGET + USBTEST_OVERSHOOT + ( + usbtest.recovery_reserve(_rec_flasher) if recovery else 0) if _current_fw and skip_flash: print('note: --skip-flash disables usbtest hang recovery; a HUNG case will leave ' 'the device wedged until it is reflashed', flush=True) @@ -1540,7 +1568,6 @@ def test_device_usbtest(board): # child side unaware that a second roster entry exists rb = json.dumps({'name': board['name'], 'flasher': _rec_flasher}) cmd += f' --recover-board {shlex.quote(rb)} --recover-fw {shlex.quote(_current_fw)}' - cmd += f' --outer-timeout {outer}' # The reserve above USBTEST_BATTERY_BUDGET exists because the battery can overrun by # one already-started case, and a hang there needs room for the recovery (whose reflash # is bounded by usbtest.RECOVER_FLASH_TIMEOUT, not HIL_CMD_TIMEOUT). Without it run_cmd @@ -1604,7 +1631,7 @@ def _usbtest_verdict(board: Board, data: dict, out: str, passed: int, failed: in # in the caller, and the two diverge as soon as a roster carries the # optional `flasher_recover` key -- naming the wrong one sends the operator to the # wrong probe. The wording stays on what usbtest actually reported ("still wedged"), - # because unrecovered_hang is also set by the ambiguous/inconclusive aborts, where + # because unrecovered_hang is also set by the ambiguous abort, where # nothing hung and the old text was false on both clauses. board_wedged = (f'{board["name"]}: usbtest reports the device still wedged ' + (f'after a recovery reflash via {rec_flasher["name"]}' if recovery @@ -1615,7 +1642,7 @@ def _usbtest_verdict(board: Board, data: dict, out: str, passed: int, failed: in notrun = int(data.get('notrun', 0)) total = passed + failed + notrun if board_wedged and failed == 0 and notrun == 0: - # Every case passed and the device STILL wedged -- usbtest's inconclusive/ambiguous + # Every case passed and the device STILL wedged -- usbtest's ambiguous # abort fires after the last case, so nothing back-fills a BUDGET entry. Reporting # the pass would exit 0 with a D-state holder on the rig and the board absent from # the re-run spec. parsed=True: a retry re-pays the whole battery to re-observe a @@ -1880,10 +1907,9 @@ def _tests_for(board: Board) -> list: def test_board(board: Board) -> tuple: - # (name, err_count, failed_tests, rows, duration[, blind, strays]) -- the board-LOCKED - # early return is 5 wide, the normal one 7. _blind_note and _stray_note index 5 and 6 - # behind a len() guard, so a field inserted before them reads a WRONG SLOT rather than - # raising: a duration would report as a stray count. + # (name, err_count, failed_tests, rows, duration[, strays]) -- the board-LOCKED early + # return is 5 wide, the normal one 6. _stray_note reads index 5 behind a len() guard, + # so a field inserted anywhere before it silently reports a duration as a stray count. swept = False name = board['name'] flasher = board['flasher'] @@ -2009,12 +2035,10 @@ def test_board(board: Board) -> tuple: stray = hil_health.kill_own_children() swept = True - # LAST fields: whether this worker ran out of bounded-read budget, and what it could - # not kill. Only the worker can answer either -- the blindness latch is - # process-global and this is a separate process -- and the result tuple already - # crosses back, so no Manager round-trip. + # LAST field: what this worker could not kill. Only the worker can answer it, and + # the result tuple already crosses back, so no Manager round-trip. return (name, err_count, [] if board_wide_fail else sorted(set(failed_tests)), - rows, t_total, hil_util.sysfs_blind(), stray) + rows, t_total, stray) finally: # A raise skips the sweep above, and maxtasksperchild=1 retires this process # immediately afterwards -- reparenting its flasher to init and erasing the ppid @@ -2133,7 +2157,7 @@ def _stray_note(mret: list) -> str: runs AFTER accumulate_report on both abort paths, so a banner appended there was written to a variable nobody read again. """ - dirty = [(r[0], r[6]) for r in mret if len(r) > 6 and r[6]] + dirty = [(r[0], r[5]) for r in mret if len(r) > 5 and r[5]] if not dirty: return '' total = sum(n for _, n in dirty) @@ -2142,28 +2166,6 @@ def _stray_note(mret: list) -> str: f'{", ".join(f"{b} ({n})" for b, n in dirty)}.\n') -def _blind_note(mret: list) -> str: - """Name the boards whose worker went blind, for the report banner. - - A blind worker answers SYSFS_UNKNOWN for every attribute, so its "device not found" is - "could not tell". That already reaches the log and the per-cell failure text, but the - TABLE is what gets quoted -- and a red cell there is read as a broken board. Seen live - (run 31794359407): four workers blind, several cells red because of it, and a report - that said nothing. - - Per-board, not global: maxtasksperchild=1 gives every board a fresh worker, so a board - that ran on a healthy one is not smeared by a neighbour's wedge. Rows synthesised by - the timeout path are 5 fields wide and have nothing to report. - """ - blind = [r[0] for r in mret if len(r) > 5 and r[5]] - if not blind: - return '' - return (f'> **Not all verdicts are evidence.** {len(blind)} board(s) ran on a worker ' - f'that went blind on sysfs -- too many bounded reads stranded on a wedged ' - f'device -- so "not found" from them means "could not tell": ' - f'{", ".join(blind)}. See the usb-kernel-recover skill.\n') - - # containment paths print through hil_health._p: stdout may already be a dead pipe (a # dropped ssh session), and a BrokenPipeError there would skip os._exit _p = hil_health._p @@ -2261,8 +2263,8 @@ def _save_controller_hints(hints: dict, mret: list, uid_of: dict, cmap) -> None: merged = {k: v for k, v in cur.items() if isinstance(v, dict)} except (OSError, ValueError): pass - # onto what the CACHE now holds, not onto our startup snapshot: another HIL job may - # have written a newer duration/pci for these boards since we read it + # overlay onto what the CACHE now holds, not onto our startup snapshot: another HIL + # job may have written a newer duration/pci for these boards since we read it for name, *_ in mret: uid = uid_of.get(name) if uid is not None and uid in hints: @@ -2296,41 +2298,49 @@ def _abort_report(reason: str, mret: list, config_boards: list, failed_fname: Pa _write_failed_spec(failed_fname, report_dir, [(n, 1, [], None, 0) for n in stuck] + [r for r in mret if r[1] > 0]) - except Exception as werr: # noqa: BLE001 - it mkdir()s and open()s the report dir - # letting it raise here REPLACES the caller's RuntimeError, so the operator never - # sees the 'pool timed out' line and no report is written at all + except Exception as werr: # noqa: BLE001 - it mkdir()s and open()s the very report dir + # the fallback below is FOR an unwritable/root-owned report dir; letting the spec + # raise here replaces the caller's RuntimeError, so the operator never sees the + # 'pool timed out' line and no report is written at all print(f'warning: re-run spec failed: {type(werr).__name__}: {werr}', flush=True) banner = (f"**HIL run {reason}.** {len(mret)} board(s) below finished and are this " f"run's; {len(stuck)} never reported and are NOT in the table: " f"{', '.join(stuck)}. The re-run spec covers those.\n") try: hil_report.accumulate_report(mret, report_dir, fresh, '', - health_banner + _blind_note(mret) - + _stray_note(mret), caveat=banner) + health_banner + _stray_note(mret), caveat=banner) return except Exception as rerr: # noqa: BLE001 - the caller's raise must still happen print(f'warning: partial report failed: {type(rerr).__name__}: {rerr}' - + ('; falling back to the board list' if timeout_secs else ''), flush=True) - if timeout_secs is None: - return + + '; falling back to the board list', flush=True) try: + # banner=, or write_timeout_report's default caveat publishes 'No per-board + # results could be collected' onto a report where mret DID hold finished rows + # the CELL names the cause: a board the pool guard never reached did not + # "pool-timeout", and marking it so sends the reader after a guard that did not fire hil_report.write_timeout_report( report_dir, [b for b in config_boards if b['name'] in stuck], - timeout_secs, prefix=health_banner) + timeout_secs or 0, banner=banner, prefix=health_banner, + cell=(hil_report.POOL_TIMEOUT_CELL if timeout_secs + else hil_report.RUN_ABORTED_CELL)) except Exception as re2: # noqa: BLE001 print(f'warning: fallback report failed too: {type(re2).__name__}: {re2}', flush=True) -def _start_pool(seed: str, hints_by_uid: dict): - """(mgr, cmap, pool). Split out so main()'s try/finally reads as one shape. +def _start_pool(mgr, seed: str, hints_by_uid: dict): + """(cmap, pool). Split out so main()'s try/finally reads as one shape. + + The Manager is created by the CALLER and passed in: Pool() forks, and after a convoy + that fork is what hits EAGAIN/ENOMEM. Creating the Manager here too would leave main() + with `mgr` still None while a live SyncManager child exists -- os._exit skips its + finalizer and the orphan holds the runner's stdout, so the job step never completes. maxtasksperchild=1: a fresh worker per board makes cross-board contamination structural rather than dependent on every module global being reset by hand (board_wedged, _current_fw, hil_flash's warn-once sets). The extra fork is noise against a flash+test cycle. """ - mgr = Manager() cmap = mgr.dict() initargs = (Lock(), seed, hil_lock.make_permit_sems(Semaphore, hil_lock.USBTEST_PARALLEL), @@ -2338,7 +2348,7 @@ def _start_pool(seed: str, hints_by_uid: dict): cmap, Lock(), hints_by_uid) pool = Pool(processes=os.cpu_count() or 1, initializer=init_worker, initargs=initargs, maxtasksperchild=1) - return mgr, cmap, pool + return cmap, pool def main() -> None: @@ -2549,7 +2559,11 @@ def main() -> None: (report_dir / f).unlink(missing_ok=True) failed_fname.unlink(missing_ok=True) try: - mgr, cmap, pool = _start_pool(seed, hints_by_uid) + # BOUND FIRST, in main's own scope: a Pool fork failure inside _start_pool must + # still leave a live Manager reachable by the finally below, or its child is + # orphaned holding the runner's stdout. + mgr = Manager() + cmap, pool = _start_pool(mgr, seed, hints_by_uid) # OUTER: encloses the pool block too, not just the reporting below. An exception # escaping async_ret.get() (a worker exception, a Ctrl-C) runs the pool finally and # then propagates straight out of main(); with _abandon_exit in a sibling try it @@ -2648,8 +2662,7 @@ def main() -> None: scoped = sorted(set(args.board) | set(board_test)) scope = f'{len(scoped)} board(s) — {", ".join(scoped)}' if scoped else '' report = hil_report.accumulate_report(mret, report_dir, fresh, scope, - health_banner + _blind_note(mret) - + _stray_note(mret)) + health_banner + _stray_note(mret)) print() print(report) print(f'\nReport written to {(report_dir / hil_report.REPORT_MD).resolve()}') |
