| Age | Commit message (Collapse) | Author |
|---|
|
Two layers whose cost was a contract to reason about rather than an outcome.
SYSFS_UNKNOWN was a three-valued return five files had to keep apart, and
misreading unknown as absence was silent: a healthy board reported as a firmware
regression. What it guarded is real -- `serial` is served by usb_string_attr,
which takes usb_lock_device_interruptible (v6.12.96 sysfs.c:141-143), the same
lock a wedged usbfs ioctl holds -- so the BOUND stays, on every caller by
default. usb_scan reads `serial` on every device matching the VID, and hil_lock's
controller_of does that on essentially every board, so one wedged DUT would
otherwise stall every worker, not one. What goes is the third value.
read_sysfs now returns str or None, and the question the third value existed to
answer is asked directly instead, by two predicates that say which question they
answer: sysfs_stranded() is process-wide and sticky, for hil_pool_check's footer
("could anything here be the tool losing sight of healthy hardware?"), and
path_stranded(path) is per-device, which is what usbtest needs to tell a DUT
whose `serial` is held under device_lock from one that genuinely left the bus --
that difference decides whether it performs driver-registry writes that take the
uninterruptible device_lock.
Gone: _SysfsUnknown, SYSFS_UNKNOWN, sysfs_blind, sysfs_blind_note,
note_sysfs_strand, the cross-process blindness publishing and its report banner,
usb_scan's (list, bool) return, usbtest's inconclusive abort, _blind_note's slot
in the result tuple, and bounded_open, whose last caller went in the previous
commit.
The strand memo is rewritten around the one invariant that makes it safe to
reuse: it is keyed by the path's kernfs inode, captured BEFORE the read. A
busport does not change when a board returns to the same physical port, so a
path-only blacklist outlives the wedge and hil_pool_check's own recovery flow --
reset, reflash, wait_device polling that busport -- would never look at the
board again. A re-enumeration destroys the kernfs node and makes a new one, so a
changed inode is the all-clear. Two ceilings bound different things: per path
(_PATH_STRAND_MAX) for a board that flaps while still wedged, and per process
(_STRAND_MAX) as a backstop against RLIMIT_NOFILE, counted per PATH rather than
per reader because hil_pool_check runs four poll threads over one bus.
A board the pool guard never reached is now reported as run-aborted rather than
pool-timed-out, and outranks a stale board-locked cell for the same reason the
pool-timeout cell does.
Both predicates answer conservatively where they are consulted before something
irreversible. path_stranded() covers the paths read_sysfs answered None for
WITHOUT reading -- past _STRAND_MAX it declines to start another reader, and
vouching for a path nobody looked at hands usbtest's fail-CLOSED guard a
fabricated all-clear, running remove_id/unbind against a wedged device. usbtest's
startup lookup carries the same caveat hil_test's absent arm already did, because
its stderr is relayed verbatim into the report cell.
strand_note() survives the removal for the same reason master had it: every
caller that can say "not found" needs the same sentence, and the one site left to
re-invent it got missed -- a wedged-but-enumerated printer was reported as an
enumeration failure, sending a maintainer after firmware.
The two predicates are not interchangeable, and usbtest needs both. Its per-case
verdict is per-DUT -- a peer that stranded at case 2 must not make our board
report wedged at case 29 -- but the finally block's cleanup is process-wide:
remove_id plus an unbind of EVERY interface under the driver, including that
peer's, each taking the uninterruptible device_lock. So the verdict uses
path_stranded() and the global cleanup stays gated on sysfs_stranded().
USBTEST_RECOVERY_BUDGET was an independent 250s that could not actually contain
the ladder it reserved for, which is why usbtest.py carried a _time_left() gate
re-deciding before every step -- with a bare '- 35' for costs paid downstream
that nobody could re-derive. Between them the two produced a recovery that
skipped its own steps for most real hangs.
The reserve is now derived from the bounds usbtest itself declares, per flasher
and per target: a probe reset, a reflash, and the Rescue-DP POR plus retry a
wedged RP DAP needs, plus the settles and hil_util.REAP_GRACE for each bounded
step. The Rescue-DP legs are openocd-only and gated on the RP target cfg, and
a stub reset is screened out, so the reserve tracks each board's real ladder
instead of one fleet number: 390s for the two RP boards -- whose ladder the old
250 could not contain, which is exactly why the gates skipped their steps --
190s for the other seventeen probe-reset boards, and 150s for esptool and
lm4flash, whose reset is a no-op. Changing a bound in usbtest moves the
reserve with it, and a unit test asserts it covers the ladder.
With the room actually reserved, the child runs the ladder straight through:
recovery_steps, _time_left, the three per-step gates and the parsed-but-unused
--outer-timeout are gone. What stays is what decides outcomes -- the convoy_safe
gate, reset-before-reflash, the no_op screen so a stub that resets nothing is not
claimed, and wedged_pids() as the arbiter, because a clean flash only proves the
probe wrote the MCU.
hil_util.py 616 -> 514 lines.
|
|
support (#3853)
Promote SEGGER RTT from an inline debugging technique to a standalone skill
backed by one stdlib-only implementation in tools/rtt.py: a CLI and importable
module for console/capture over J-Link (RTTTelnetPort) and OpenOCD (rtt server)
probes, with probe selection by serial or VID:PID, control-block address via
--elf or --addr, bidirectional console, post-mortem ring dump, and
--reset-before-attach for boot-time capture. The HIL harness reads a board's
console over RTT when its probe has no VCOM ("logger": "rtt" plus a LOGGER=rtt
variant define), covering device_info, pool-check aliveness, and CI wiring.
Validated on 22 boards across both backends; 26 unit tests run in pre-commit.
|
|
Three readability changes with no behaviour change on the healthy path —
every pre-existing test passes untouched.
main() was 368 lines with try/finally three deep, its two abort paths
near-identical 40-line blocks; _abort_report holds that shape once, and the
controller-hint cache and pool construction move to their own helpers.
368 -> 279, test_board 180 -> 151, test_device_usbtest 164 -> 125.
test_hil_bounded.py cost 78s on every commit under test/hil/, mostly one 3s
post-flash settle paid by ten tests against a fake rig. Now 37s.
Fixes two pre-existing defects the extraction exposed: _write_failed_spec was
unguarded inside the abort path, so an OSError there replaced the caller's
RuntimeError and no report was written at all; _save_controller_hints overlaid
a startup snapshot onto the re-read cache, clobbering a concurrent job's newer
values. Also five comments that stated the opposite of the code, and both table
renderers measuring width with len() against two-column status marks.
|
|
The roster had two ways to pass a cmake -D to a board's build: `build.args`,
applied to every variant, and `variant[].defines`, applied to one. They did the
same thing, and only metro_m4_express used the first - for MAX3421_HOST=1, which
is what makes it the one rig board that compiles hcd_max3421.c.
A board whose define is always on now carries a single variant named after itself,
which is exactly the shape `board.get('variant') or [{'name': name, 'flags': ''}]`
already synthesises everywhere - so the build dir, the HIL report row and the
variant-boundary handling are unchanged. raspberry_pi_pico has used that shape
for its flags all along.
Removes the BuildCfg type and the parallel code path from all four consumers:
hil_test.build_board, hil_pool_check's two builders, hil_ci_set_matrix and
ci_select.board_options.
Verified: the hil-build matrix entry is byte-identical
(`-b metro_m4_express -DMAX3421_HOST=1`), hil_test's build command is unchanged,
ci_select still selects the board for a max3421 diff with MAX3421_HOST in its
options, and a real build of dual/host_info_to_device_cdc and host/cdc_msc_hid on
that board still compiles hcd_max3421.c.
|
|
~/code/pico/pico-sdk and $HOME/code/esp-idf/export.sh are the ci rig's private layout;
written into instructions they silently stop being true on tusb, a dev PC, or any
future rig. The docs now use the variables the vendors define -- PICO_SDK_PATH for the
Pico SDK and IDF_PATH for ESP-IDF, activated explicitly as `. "$IDF_PATH/export.sh"` --
and leave where the checkouts live to each host's profile.
The variables are only useful if the shells that agents actually get can see them, and
`ssh <rig> 'cmd'` is non-interactive AND non-login: it reads no profile, and Debian's
sshd-sourced ~/.bashrc returns at the interactive guard before most of the file. The ci
rig already keeps its exports in the section ABOVE that early-return; IDF_PATH now sits
there beside PICO_SDK_PATH, and the whole chain is verified from a plain non-interactive
ssh: both variables visible, `. "$IDF_PATH/export.sh"` activates ESP-IDF v5.5.3 with
idf.py on PATH -- no login shell, no alias, no hard-coded path. hil-pool-check documents
that placement so the next rig is set up the same way.
|
|
script directly
A probe whose firmware has wedged reports flash-failed with the probe present and
"probe toggle unconfirmed". The tool's own recovery cannot fix that: an authorized
toggle re-enumerates but never removes power, and hil_pool_check.py:304 already notes
that ST-Link, WCH-Link, CP210x and picoprobe keep their sysfs kobject across one. So
the check correctly gives up, and the operator was left to invent the next rung.
Write it down, as what this rig's hardware actually does rather than what uhubctl
advertises: the Renesas cards list their root hubs as ppps-capable but do not
implement it (owner-confirmed - VBUS never drops, only D+/D-), so a root-port cycle
is a harder forced re-enumeration that a wedged probe can ride out, worth exactly one
attempt; and the AMD 0000:02:00.0, where the WCH-Links live, has no port-power
switching at all - nothing to cycle, straight to a physical replug. Which card a
probe hangs off decides which case applies, so the procedure starts from readlink.
The ordering rules encode the shared-rig protocol: let the full run finish (a bounce
re-enumerates siblings and corrupts checks still in flight), hold --all with this
host's --config before the cycle (hil_lock.py hold validates nothing against the
roster and nothing maps a sysfs busport to a board name, so a narrower hand-listed
hold reserves nothing while reporting success - and --all defaults to tinyusb.json,
which on the tusb rig would reserve 27 boards that do not exist there), release
BEFORE the re-check (hil_pool_check.py self-locks every board it checks, so a hold
still in place makes the verification report locked against your own hold and verify
nothing), and drive the cycle through usb_recover.sh root-cycle by its full in-repo
path - it is on no PATH and sudo's secure_path excludes the checkout. Never a bare
`uhubctl -a cycle`: without -S it writes sysfs disable, whose disable_store takes the
root hub's lock uninterruptibly and then usb_disconnect()s the wedged child - the one
input that turns a probe wedge into a bus-wide wedge. Give the script the wedged
probe's own busport, not the hub path: the serial guard and the success check both
read the path you pass, and the hub's inode always changes when its own port cycles.
Reporting asks for both passes: a final table showing every board healthy hides that
a probe needed power-cycling to get there, which is the signal that it will recur.
Also: the ESP-IDF env hints name `. $HOME/code/esp-idf/export.sh` instead of the
`get-idf` alias, which lives only in interactive shells and fails from scripts.
|
|
A wedged USB device used to take the whole HIL run with it. Every worker that
touched the poisoned node blocked uninterruptibly, the pool could not be joined,
map_async discarded every board's result, and the job ran to the GitHub ceiling
with no report at all -- while the self-hosted runner's single job slot stayed
occupied and every queued job waited behind it.
Bound the calls a worker makes itself. read_sysfs, bounded_open and run_cmd all
answer within a wall clock; read_sysfs distinguishes "absent" from "unknown",
because a blocked read is not evidence of absence, and caps stranded readers at
four (each costs a thread and an fd for the life of the process) after which the
worker declares itself blind. mtype, the gio unmount, the libmtp session and the
arecord/iperf reaps go through those bounds; the MTP session runs in a disposable
subprocess, since libmtp's ctypes calls block unkillably in D state.
Bound the run. A pool guard (HIL_POOL_TIMEOUT, 60 min) fires before any job
ceiling and still writes a report. When the pool will not shut down, the sweep
kills what the workers spawned -- descendants, not just direct children, since
flashers run in their own session -- confirms each kill actually landed, and
exits early so the runner is freed. Whatever survived is named in the report.
Deliberately shallow past that point. We do not re-scan process groups, prove
pid ownership, or escalate through sudo: a root-owned survivor is reported, not
force-killed, because signalling a pid we cannot prove is ours is the worse
failure, and the job ceiling backstops whatever this misses. A D-state holder
was never killable anyway.
Recover instead of reporting a wedge. A HUNG usbtest case reflashes its own DUT
through its roster flasher, but only where the flasher can reach its probe past
a poisoned node -- openocd pinned to a validated vid_pid, or esptool. Where it
cannot, the run says so rather than reserving budget for a path that cannot fire.
Raise the CI ceilings above the pool guard so the guard fires first and still
writes its report, and pin --retry 1 on every HIL leg: the guard is a flat
constant and does not scale with max_retry, so argparse's default of 3 would
triple the serialized usbtest tail against an unchanged guard.
Split the module: execution in hil_test/hil_flash/usbtest, infrastructure in
helper/ (locking, health, selection, shared bounded IO), and the two matrix
generators into .github/scripts/ -- ci_set_matrix.py sat in workflows/, where
GitHub treats every file as a workflow definition. 193 tests cover the bounded
paths, the kill ladder, the guard and the selector against synthetic /proc trees
and PATH-injected fakes; a real wedge cannot be manufactured on demand.
|
