tinyusb.git/test/hil/helper/hil_pool_check.py, branch master

test/hil: drop the sysfs blindness subsystem and derive the recovery reserve

2026-08-28T07:18:41Z

Two layers whose cost was a contract to reason about rather than an outcome. SYSFS_UNKNOWN was a three-valued return five files had to keep apart, and misreading unknown as absence was silent: a healthy board reported as a firmware regression. What it guarded is real -- `serial` is served by usb_string_attr, which takes usb_lock_device_interruptible (v6.12.96 sysfs.c:141-143), the same lock a wedged usbfs ioctl holds -- so the BOUND stays, on every caller by default. usb_scan reads `serial` on every device matching the VID, and hil_lock's controller_of does that on essentially every board, so one wedged DUT would otherwise stall every worker, not one. What goes is the third value. read_sysfs now returns str or None, and the question the third value existed to answer is asked directly instead, by two predicates that say which question they answer: sysfs_stranded() is process-wide and sticky, for hil_pool_check's footer ("could anything here be the tool losing sight of healthy hardware?"), and path_stranded(path) is per-device, which is what usbtest needs to tell a DUT whose `serial` is held under device_lock from one that genuinely left the bus -- that difference decides whether it performs driver-registry writes that take the uninterruptible device_lock. Gone: _SysfsUnknown, SYSFS_UNKNOWN, sysfs_blind, sysfs_blind_note, note_sysfs_strand, the cross-process blindness publishing and its report banner, usb_scan's (list, bool) return, usbtest's inconclusive abort, _blind_note's slot in the result tuple, and bounded_open, whose last caller went in the previous commit. The strand memo is rewritten around the one invariant that makes it safe to reuse: it is keyed by the path's kernfs inode, captured BEFORE the read. A busport does not change when a board returns to the same physical port, so a path-only blacklist outlives the wedge and hil_pool_check's own recovery flow -- reset, reflash, wait_device polling that busport -- would never look at the board again. A re-enumeration destroys the kernfs node and makes a new one, so a changed inode is the all-clear. Two ceilings bound different things: per path (_PATH_STRAND_MAX) for a board that flaps while still wedged, and per process (_STRAND_MAX) as a backstop against RLIMIT_NOFILE, counted per PATH rather than per reader because hil_pool_check runs four poll threads over one bus. A board the pool guard never reached is now reported as run-aborted rather than pool-timed-out, and outranks a stale board-locked cell for the same reason the pool-timeout cell does. Both predicates answer conservatively where they are consulted before something irreversible. path_stranded() covers the paths read_sysfs answered None for WITHOUT reading -- past _STRAND_MAX it declines to start another reader, and vouching for a path nobody looked at hands usbtest's fail-CLOSED guard a fabricated all-clear, running remove_id/unbind against a wedged device. usbtest's startup lookup carries the same caveat hil_test's absent arm already did, because its stderr is relayed verbatim into the report cell. strand_note() survives the removal for the same reason master had it: every caller that can say "not found" needs the same sentence, and the one site left to re-invent it got missed -- a wedged-but-enumerated printer was reported as an enumeration failure, sending a maintainer after firmware. The two predicates are not interchangeable, and usbtest needs both. Its per-case verdict is per-DUT -- a peer that stranded at case 2 must not make our board report wedged at case 29 -- but the finally block's cleanup is process-wide: remove_id plus an unbind of EVERY interface under the driver, including that peer's, each taking the uninterruptible device_lock. So the verdict uses path_stranded() and the global cleanup stays gated on sysfs_stranded(). USBTEST_RECOVERY_BUDGET was an independent 250s that could not actually contain the ladder it reserved for, which is why usbtest.py carried a _time_left() gate re-deciding before every step -- with a bare '- 35' for costs paid downstream that nobody could re-derive. Between them the two produced a recovery that skipped its own steps for most real hangs. The reserve is now derived from the bounds usbtest itself declares, per flasher and per target: a probe reset, a reflash, and the Rescue-DP POR plus retry a wedged RP DAP needs, plus the settles and hil_util.REAP_GRACE for each bounded step. The Rescue-DP legs are openocd-only and gated on the RP target cfg, and a stub reset is screened out, so the reserve tracks each board's real ladder instead of one fleet number: 390s for the two RP boards -- whose ladder the old 250 could not contain, which is exactly why the gates skipped their steps -- 190s for the other seventeen probe-reset boards, and 150s for esptool and lm4flash, whose reset is a no-op. Changing a bound in usbtest moves the reserve with it, and a unit test asserts it covers the ladder. With the room actually reserved, the child runs the ladder straight through: recovery_steps, _time_left, the three per-step gates and the parsed-but-unused --outer-timeout are gone. What stays is what decides outcomes -- the convoy_safe gate, reset-before-reflash, the no_op screen so a stub that resets nothing is not claimed, and wedged_pids() as the arbiter, because a clean flash only proves the probe wrote the MCU. hil_util.py 616 -> 514 lines.

Add RTT console/capture tooling (tools/rtt.py), rtt skill, and HIL harness support (#3853)

2026-08-28T07:16:02Z

Promote SEGGER RTT from an inline debugging technique to a standalone skill backed by one stdlib-only implementation in tools/rtt.py: a CLI and importable module for console/capture over J-Link (RTTTelnetPort) and OpenOCD (rtt server) probes, with probe selection by serial or VID:PID, control-block address via --elf or --addr, bidirectional console, post-mortem ring dump, and --reset-before-attach for boot-time capture. The HIL harness reads a board's console over RTT when its probe has no VCOM ("logger": "rtt" plus a LOGGER=rtt variant define), covering device_info, pool-check aliveness, and CI wiring. Validated on 22 boards across both backends; 26 unit tests run in pre-commit.

test/hil: make main() readable and stop the suite sleeping (#3848)

2026-08-26T06:41:05Z

Three readability changes with no behaviour change on the healthy path — every pre-existing test passes untouched. main() was 368 lines with try/finally three deep, its two abort paths near-identical 40-line blocks; _abort_report holds that shape once, and the controller-hint cache and pool construction move to their own helpers. 368 -> 279, test_board 180 -> 151, test_device_usbtest 164 -> 125. test_hil_bounded.py cost 78s on every commit under test/hil/, mostly one 3s post-flash settle paid by ten tests against a fake rig. Now 37s. Fixes two pre-existing defects the extraction exposed: _write_failed_spec was unguarded inside the abort path, so an OSError there replaced the caller's RuntimeError and no report was written at all; _save_controller_hints overlaid a startup snapshot onto the re-read cache, clobbering a concurrent job's newer values. Also five comments that stated the opposite of the code, and both table renderers measuring width with len() against two-column status marks.

hil: express a board's always-on defines as a variant, dropping build.args

2026-08-21T07:23:40Z

The roster had two ways to pass a cmake -D to a board's build: `build.args`, applied to every variant, and `variant[].defines`, applied to one. They did the same thing, and only metro_m4_express used the first - for MAX3421_HOST=1, which is what makes it the one rig board that compiles hcd_max3421.c. A board whose define is always on now carries a single variant named after itself, which is exactly the shape `board.get('variant') or [{'name': name, 'flags': ''}]` already synthesises everywhere - so the build dir, the HIL report row and the variant-boundary handling are unchanged. raspberry_pi_pico has used that shape for its flags all along. Removes the BuildCfg type and the parallel code path from all four consumers: hil_test.build_board, hil_pool_check's two builders, hil_ci_set_matrix and ci_select.board_options. Verified: the hil-build matrix entry is byte-identical (`-b metro_m4_express -DMAX3421_HOST=1`), hil_test's build command is unchanged, ci_select still selects the board for a max3421 diff with MAX3421_HOST in its options, and a real build of dual/host_info_to_device_cdc and host/cdc_msc_hid on that board still compiles hcd_max3421.c.

hil, docs: reference toolchains by their official env vars, not one rig's paths

2026-08-20T10:43:44Z

~/code/pico/pico-sdk and $HOME/code/esp-idf/export.sh are the ci rig's private layout; written into instructions they silently stop being true on tusb, a dev PC, or any future rig. The docs now use the variables the vendors define -- PICO_SDK_PATH for the Pico SDK and IDF_PATH for ESP-IDF, activated explicitly as `. "$IDF_PATH/export.sh"` -- and leave where the checkouts live to each host's profile. The variables are only useful if the shells that agents actually get can see them, and `ssh 'cmd'` is non-interactive AND non-login: it reads no profile, and Debian's sshd-sourced ~/.bashrc returns at the interactive guard before most of the file. The ci rig already keeps its exports in the section ABOVE that early-return; IDF_PATH now sits there beside PICO_SDK_PATH, and the whole chain is verified from a plain non-interactive ssh: both variables visible, `. "$IDF_PATH/export.sh"` activates ESP-IDF v5.5.3 with idf.py on PATH -- no login shell, no alias, no hard-coded path. hil-pool-check documents that placement so the next rig is set up the same way.

hil-pool-check: document the probe power-cycle escalation; name the env script directly

2026-08-20T09:45:32Z

A probe whose firmware has wedged reports flash-failed with the probe present and "probe toggle unconfirmed". The tool's own recovery cannot fix that: an authorized toggle re-enumerates but never removes power, and hil_pool_check.py:304 already notes that ST-Link, WCH-Link, CP210x and picoprobe keep their sysfs kobject across one. So the check correctly gives up, and the operator was left to invent the next rung. Write it down, as what this rig's hardware actually does rather than what uhubctl advertises: the Renesas cards list their root hubs as ppps-capable but do not implement it (owner-confirmed - VBUS never drops, only D+/D-), so a root-port cycle is a harder forced re-enumeration that a wedged probe can ride out, worth exactly one attempt; and the AMD 0000:02:00.0, where the WCH-Links live, has no port-power switching at all - nothing to cycle, straight to a physical replug. Which card a probe hangs off decides which case applies, so the procedure starts from readlink. The ordering rules encode the shared-rig protocol: let the full run finish (a bounce re-enumerates siblings and corrupts checks still in flight), hold --all with this host's --config before the cycle (hil_lock.py hold validates nothing against the roster and nothing maps a sysfs busport to a board name, so a narrower hand-listed hold reserves nothing while reporting success - and --all defaults to tinyusb.json, which on the tusb rig would reserve 27 boards that do not exist there), release BEFORE the re-check (hil_pool_check.py self-locks every board it checks, so a hold still in place makes the verification report locked against your own hold and verify nothing), and drive the cycle through usb_recover.sh root-cycle by its full in-repo path - it is on no PATH and sudo's secure_path excludes the checkout. Never a bare `uhubctl -a cycle`: without -S it writes sysfs disable, whose disable_store takes the root hub's lock uninterruptibly and then usb_disconnect()s the wedged child - the one input that turns a probe wedge into a bus-wide wedge. Give the script the wedged probe's own busport, not the hub path: the serial guard and the success check both read the path you pass, and the hub's inode always changes when its own port cycles. Reporting asks for both passes: a final table showing every board healthy hides that a probe needed power-cycling to get there, which is the signal that it will recur. Also: the ESP-IDF env hints name `. $HOME/code/esp-idf/export.sh` instead of the `get-idf` alias, which lives only in interactive shells and fails from scripts.

test/hil, ci: contain a wedged USB stack instead of stranding the runner

2026-08-18T05:19:09Z

A wedged USB device used to take the whole HIL run with it. Every worker that touched the poisoned node blocked uninterruptibly, the pool could not be joined, map_async discarded every board's result, and the job ran to the GitHub ceiling with no report at all -- while the self-hosted runner's single job slot stayed occupied and every queued job waited behind it. Bound the calls a worker makes itself. read_sysfs, bounded_open and run_cmd all answer within a wall clock; read_sysfs distinguishes "absent" from "unknown", because a blocked read is not evidence of absence, and caps stranded readers at four (each costs a thread and an fd for the life of the process) after which the worker declares itself blind. mtype, the gio unmount, the libmtp session and the arecord/iperf reaps go through those bounds; the MTP session runs in a disposable subprocess, since libmtp's ctypes calls block unkillably in D state. Bound the run. A pool guard (HIL_POOL_TIMEOUT, 60 min) fires before any job ceiling and still writes a report. When the pool will not shut down, the sweep kills what the workers spawned -- descendants, not just direct children, since flashers run in their own session -- confirms each kill actually landed, and exits early so the runner is freed. Whatever survived is named in the report. Deliberately shallow past that point. We do not re-scan process groups, prove pid ownership, or escalate through sudo: a root-owned survivor is reported, not force-killed, because signalling a pid we cannot prove is ours is the worse failure, and the job ceiling backstops whatever this misses. A D-state holder was never killable anyway. Recover instead of reporting a wedge. A HUNG usbtest case reflashes its own DUT through its roster flasher, but only where the flasher can reach its probe past a poisoned node -- openocd pinned to a validated vid_pid, or esptool. Where it cannot, the run says so rather than reserving budget for a path that cannot fire. Raise the CI ceilings above the pool guard so the guard fires first and still writes its report, and pin --retry 1 on every HIL leg: the guard is a flat constant and does not scale with max_retry, so argparse's default of 3 would triple the serialized usbtest tail against an unchanged guard. Split the module: execution in hil_test/hil_flash/usbtest, infrastructure in helper/ (locking, health, selection, shared bounded IO), and the two matrix generators into .github/scripts/ -- ci_set_matrix.py sat in workflows/, where GitHub treats every file as a workflow definition. 193 tests cover the bounded paths, the kill ladder, the guard and the selector against synthetic /proc trees and PATH-injected fakes; a real wedge cannot be manufactured on demand.