| Age | Commit message (Collapse) | Author |
|---|
|
Two layers whose cost was a contract to reason about rather than an outcome.
SYSFS_UNKNOWN was a three-valued return five files had to keep apart, and
misreading unknown as absence was silent: a healthy board reported as a firmware
regression. What it guarded is real -- `serial` is served by usb_string_attr,
which takes usb_lock_device_interruptible (v6.12.96 sysfs.c:141-143), the same
lock a wedged usbfs ioctl holds -- so the BOUND stays, on every caller by
default. usb_scan reads `serial` on every device matching the VID, and hil_lock's
controller_of does that on essentially every board, so one wedged DUT would
otherwise stall every worker, not one. What goes is the third value.
read_sysfs now returns str or None, and the question the third value existed to
answer is asked directly instead, by two predicates that say which question they
answer: sysfs_stranded() is process-wide and sticky, for hil_pool_check's footer
("could anything here be the tool losing sight of healthy hardware?"), and
path_stranded(path) is per-device, which is what usbtest needs to tell a DUT
whose `serial` is held under device_lock from one that genuinely left the bus --
that difference decides whether it performs driver-registry writes that take the
uninterruptible device_lock.
Gone: _SysfsUnknown, SYSFS_UNKNOWN, sysfs_blind, sysfs_blind_note,
note_sysfs_strand, the cross-process blindness publishing and its report banner,
usb_scan's (list, bool) return, usbtest's inconclusive abort, _blind_note's slot
in the result tuple, and bounded_open, whose last caller went in the previous
commit.
The strand memo is rewritten around the one invariant that makes it safe to
reuse: it is keyed by the path's kernfs inode, captured BEFORE the read. A
busport does not change when a board returns to the same physical port, so a
path-only blacklist outlives the wedge and hil_pool_check's own recovery flow --
reset, reflash, wait_device polling that busport -- would never look at the
board again. A re-enumeration destroys the kernfs node and makes a new one, so a
changed inode is the all-clear. Two ceilings bound different things: per path
(_PATH_STRAND_MAX) for a board that flaps while still wedged, and per process
(_STRAND_MAX) as a backstop against RLIMIT_NOFILE, counted per PATH rather than
per reader because hil_pool_check runs four poll threads over one bus.
A board the pool guard never reached is now reported as run-aborted rather than
pool-timed-out, and outranks a stale board-locked cell for the same reason the
pool-timeout cell does.
Both predicates answer conservatively where they are consulted before something
irreversible. path_stranded() covers the paths read_sysfs answered None for
WITHOUT reading -- past _STRAND_MAX it declines to start another reader, and
vouching for a path nobody looked at hands usbtest's fail-CLOSED guard a
fabricated all-clear, running remove_id/unbind against a wedged device. usbtest's
startup lookup carries the same caveat hil_test's absent arm already did, because
its stderr is relayed verbatim into the report cell.
strand_note() survives the removal for the same reason master had it: every
caller that can say "not found" needs the same sentence, and the one site left to
re-invent it got missed -- a wedged-but-enumerated printer was reported as an
enumeration failure, sending a maintainer after firmware.
The two predicates are not interchangeable, and usbtest needs both. Its per-case
verdict is per-DUT -- a peer that stranded at case 2 must not make our board
report wedged at case 29 -- but the finally block's cleanup is process-wide:
remove_id plus an unbind of EVERY interface under the driver, including that
peer's, each taking the uninterruptible device_lock. So the verdict uses
path_stranded() and the global cleanup stays gated on sysfs_stranded().
USBTEST_RECOVERY_BUDGET was an independent 250s that could not actually contain
the ladder it reserved for, which is why usbtest.py carried a _time_left() gate
re-deciding before every step -- with a bare '- 35' for costs paid downstream
that nobody could re-derive. Between them the two produced a recovery that
skipped its own steps for most real hangs.
The reserve is now derived from the bounds usbtest itself declares, per flasher
and per target: a probe reset, a reflash, and the Rescue-DP POR plus retry a
wedged RP DAP needs, plus the settles and hil_util.REAP_GRACE for each bounded
step. The Rescue-DP legs are openocd-only and gated on the RP target cfg, and
a stub reset is screened out, so the reserve tracks each board's real ladder
instead of one fleet number: 390s for the two RP boards -- whose ladder the old
250 could not contain, which is exactly why the gates skipped their steps --
190s for the other seventeen probe-reset boards, and 150s for esptool and
lm4flash, whose reset is a no-op. Changing a bound in usbtest moves the
reserve with it, and a unit test asserts it covers the ladder.
With the room actually reserved, the child runs the ladder straight through:
recovery_steps, _time_left, the three per-step gates and the parsed-but-unused
--outer-timeout are gone. What stays is what decides outcomes -- the convoy_safe
gate, reset-before-reflash, the no_op screen so a stub that resets nothing is not
claimed, and wedged_pids() as the arbiter, because a clean flash only proves the
probe wrote the MCU.
hil_util.py 616 -> 514 lines.
|
|
support (#3853)
Promote SEGGER RTT from an inline debugging technique to a standalone skill
backed by one stdlib-only implementation in tools/rtt.py: a CLI and importable
module for console/capture over J-Link (RTTTelnetPort) and OpenOCD (rtt server)
probes, with probe selection by serial or VID:PID, control-block address via
--elf or --addr, bidirectional console, post-mortem ring dump, and
--reset-before-attach for boot-time capture. The HIL harness reads a board's
console over RTT when its probe has no VCOM ("logger": "rtt" plus a LOGGER=rtt
variant define), covering device_info, pool-check aliveness, and CI wiring.
Validated on 22 boards across both backends; 26 unit tests run in pre-commit.
|
|
Every PR built all 74 legs (2494 example builds on GHA cmake alone) and flashed
all 30 rig boards, whatever it touched. One classifier now walks the PR diff twice
and answers three questions: which families to build, which examples per family,
and which boards run which tests. Fail-open throughout - anything no rule
classifies, any exception, any unusable output falls back to the full matrix, and
a master push always builds everything.
test/hil/helper/hil_select.py moves to tools/ci_select.py: it is no longer HIL-only,
and tools/ is where the build side can import it. test_hil_select.py follows it as
test_ci_select.py.
Rules (docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md holds the
full table): a port selects the families whose family.cmake references it, and its
role - a dcd change skips host examples and vice versa; a class selects only the
examples whose tusb_config.h enables its CFG_TU[DH]_ macro, following cross-class
includes; an example selects itself; hw/bsp selects its family or board; hw/mcu and
lib select whoever references them. CMake is the reference for all of it - make
follows whatever cmake decides, family.mk is never scanned.
Empty means empty (maintainer ruling): a rule that classifies a path to nothing
selects nothing. Ports no family references, classes no config enables, libs no
example builds and hw/mcu paths that resolve nowhere are all real - nothing
compiles them, so nothing can validate them, and the master-push build is the net.
Structural tests pin each such case with an explicit allowlist, so the day one
stops being empty it fails pre-commit instead of silently narrowing CI.
Per-example builds: build.py grows a repeatable -e, resolved against the targets
CMake actually registered and batched into one `cmake --build --target a b c`.
build_utils mirrors CMake's family_filter (the whole FAMILY_MCUS list, ${...} and
string(TOUPPER ...) resolved) for the cmake side, while the make side keeps
master's algorithm verbatim - the two build systems answer differently and a shared
answer breaks lpc54's make link. hil-build gains this even on a full selection:
1702 example builds become 515.
Transport: the selection travels as a file, never an argv or env var - a mass-sweep
diff selects 261 KB against a 128 KiB exec limit, and E2BIG would fail the step
before its own fallback could run. CircleCI carries the example map inside the
generated config (pipeline parameters cap at 512 chars), swapped into the parameter
defaults by sentinel match, and drops the scoping wholesale if that rewrite fails.
Every PR-derived value written to $GITHUB_ENV/$GITHUB_OUTPUT is character-screened.
Code metrics follow the scoping: metrics.py emits per-example totals, and
metrics_pair_compare compares the (board, example) pairs present on both sides
instead of a scoped run against a full-matrix average.
The selector's own suite gates it in both providers: a selector that exits 0 with
valid-but-wrong JSON is the one failure fail-open cannot catch, so a red suite
means the full matrix.
|
|
hil-validate ran one hil-operator per board. That parallelizes at the wrong layer:
hil_test.py already schedules boards across host controllers and budgets concurrent
flashes and usbtest batteries per controller (FLASH_PARALLEL/USBTEST_PARALLEL), and
those permits live in one process - N parallel runs multiply the budget onto the same
uPD720201 cards for no wall-clock gain over one run that already parallelizes. The
workflow now spawns ONE operator with every board as repeated -b.
The operator no longer retypes the report table. Four consecutive max-effort review
rounds found ~15 defects in this file and every one was in reconstructing board
identity from transcribed prose: report rows are named per VARIANT (nanoch32v203 only
ever produces -fsdev/-usbfs rows), a variant need not start with its board's name,
lock contention is a `board-locked` cell rather than a phrase, and each fix introduced
the next round's bug - including a fake-green test that asserted an invariant with the
one input shape that could not break it. The new helper test/hil/helper/hil_summary.py
does the join where the roster lives and emits one machine verdict per board
({board, ran, pass, locked, detail}); the operator returns that JSON verbatim plus
`wedged`, the only field it authors, and the workflow reads fields, never parses a
string. Its cell classifier mirrors hil_test.py's own tally exactly: failures are
always marked ('fail' or a ❌ prefix, TestFail's contract), everything unmarked is a
pass - a passing test may return a plain metric cell like '13443 KB/s', and the
mirrored rule is what keeps a green table from becoming a red verdict.
hil_ci.sh kept only the LAST -b, so multi-board remote runs staged one board's
binaries and every other board died on the rig after its lock and flash slot were
spent. It now parses every -b spelling argparse accepts (with the -bt arms ordered
first, longest-match, so the <config>.failed retry form is never read as a board named
"t..."), pre-flights roster membership and build dirs for ALL boards before anything
is wiped or staged, warns per declared variant with no build dir (which hil_test.py
would silently green-skip), forwards HIL_* knobs as export lines in one %q word the
remote evals ('; '-joined so it round-trips under dash - an authorized
HIL_NO_BOARD_LOCK force must not silently no-op), keeps HIL_REPORT_DIR local because
the copy-backs look in REMOTE_DIR, and copies hil_report.json and the .failed re-run
spec back beside the markdown, deleting stale local copies first so a green run cannot
leave last run's spec looking current.
Retries preserve the fleet: the documented path is the <config>.failed spec, which
already begins with --accumulate; a fresh scoped re-run would unlink the report and
collapse the whole-fleet table to the retried boards alone.
The risky logic is executable, not argued about: .claude/workflows/test-hil-validate.mjs
pins the lookup/verdict helpers and runs in pre-commit (hil-validate-logic); nine
staging tests drive hil_ci.sh through an ssh stub that models the real thing (argv
joined into one string the remote re-splits, heredoc on stdin - the naive echo-stub
passed while the feature was broken); and deliberate mutations of the verdict logic
are all caught.
Validated on the rig: a 2-board run (usbtest 30/30 on both; the pre-fix classifier,
replayed against that run's real report, fails the fully-green stm32f723disco on its
two passing '13443 KB/s' cells), the .failed --accumulate retry (merged report kept
every earlier row), and a 10-run soak over random subsets of a 22-board pool - 43
board-slots, every failure signature matched pre-existing CI state or known flake,
zero tooling failures, no locks left behind.
|
|
A wedged USB device used to take the whole HIL run with it. Every worker that
touched the poisoned node blocked uninterruptibly, the pool could not be joined,
map_async discarded every board's result, and the job ran to the GitHub ceiling
with no report at all -- while the self-hosted runner's single job slot stayed
occupied and every queued job waited behind it.
Bound the calls a worker makes itself. read_sysfs, bounded_open and run_cmd all
answer within a wall clock; read_sysfs distinguishes "absent" from "unknown",
because a blocked read is not evidence of absence, and caps stranded readers at
four (each costs a thread and an fd for the life of the process) after which the
worker declares itself blind. mtype, the gio unmount, the libmtp session and the
arecord/iperf reaps go through those bounds; the MTP session runs in a disposable
subprocess, since libmtp's ctypes calls block unkillably in D state.
Bound the run. A pool guard (HIL_POOL_TIMEOUT, 60 min) fires before any job
ceiling and still writes a report. When the pool will not shut down, the sweep
kills what the workers spawned -- descendants, not just direct children, since
flashers run in their own session -- confirms each kill actually landed, and
exits early so the runner is freed. Whatever survived is named in the report.
Deliberately shallow past that point. We do not re-scan process groups, prove
pid ownership, or escalate through sudo: a root-owned survivor is reported, not
force-killed, because signalling a pid we cannot prove is ours is the worse
failure, and the job ceiling backstops whatever this misses. A D-state holder
was never killable anyway.
Recover instead of reporting a wedge. A HUNG usbtest case reflashes its own DUT
through its roster flasher, but only where the flasher can reach its probe past
a poisoned node -- openocd pinned to a validated vid_pid, or esptool. Where it
cannot, the run says so rather than reserving budget for a path that cannot fire.
Raise the CI ceilings above the pool guard so the guard fires first and still
writes its report, and pin --retry 1 on every HIL leg: the guard is a flat
constant and does not scale with max_retry, so argparse's default of 3 would
triple the serialized usbtest tail against an unchanged guard.
Split the module: execution in hil_test/hil_flash/usbtest, infrastructure in
helper/ (locking, health, selection, shared bounded IO), and the two matrix
generators into .github/scripts/ -- ci_set_matrix.py sat in workflows/, where
GitHub treats every file as a workflow definition. 193 tests cover the bounded
paths, the kill ladder, the guard and the selector against synthetic /proc trees
and PATH-injected fakes; a real wedge cannot be manufactured on demand.
|
|
Pool/config:
- record real uids (ra8m1_ek), enable usbtest for espressif s3/p4, then
park ra6m5_ek and ra8m1_ek in boards-skip (ra6m5's usbtest/MSC traffic
can kill the uPD720201 host on its ROM firmware; ra8m1 USBHS bring-up
pending); max32666/nrf54lm20 stay enabled - their MosChip flakiness
never wedges
- re-enable device/usbtest on HS boards (mimxrt1064, ch32v307) now that
uPD720201 firmware 2.0.2.6 fixes the command-ring death; mimxrt1015
stays skipped - its HS battery killed the controller on both ROM and
2.0.2.6 firmware (board-specific); match the moved host-test bundles
(f723 <-> rt1064); skip never-passing tests on the new
nrf5340dk/nrf54lm20dk boards and the detached pico host bundle, each
documented with a comment
Host-controller quirk gating in usbtest.py (auto-skip, self-heals on a
healthy xHCI):
- MosChip MCS9990 EHCI: case 25 (int-OUT never scheduled, FRINDEX bug)
and case 11 (unlinked reads complete short/EREMOTEIO)
- Renesas uPD720201 xHCI: firmware-gated. The card must run firmware
>= 2.0.2.6 (RAM-uploaded - it reverts to ROM on every power cycle):
on older firmware the command ring dies under unlink stress (a
Configure Endpoint command stops completing; the hub worker deadlocks
holding the device lock; only a host power cycle recovers; three
boards reproduced it). usbtest.py reads the FW version register (PCI
config 0x6c) and refuses to run at all on older firmware - hil_test
surfaces that as a failed test with the reason. On current firmware
the full 30-case battery runs (validated FS+HS: metro_m4, f723,
f723-DMA all 30/30).
Scheduling (hil_test.py):
- Shuffle each (board, variant)'s test order with a seeded RNG
(HIL_SHUFFLE_SEED to replay) so usbtest batteries and flash churn spread
across the timeline instead of convoying on one controller.
- Per-controller usbtest + flash semaphores: HIL_USBTEST_PARALLEL
(default 4) concurrent usbtest batteries and HIL_FLASH_PARALLEL
(default 8) concurrent flashes per host controller. Profiled on
uPD720201 firmware 2.0.2.6 across 8/1..12/8: wall time falls
22.2/14.3/12.5/10.8 min at usbtest width 1/2/3/4 and plateaus there;
zero controller errors everywhere; first battery case failures
(leaf-hub bandwidth stretch) appear at 12/8, and flash width 12 only
amplifies flasher-hub contention flakes - so 8/4 is the optimum. A
separate battery-window flash throttle was profiled and dropped.
- Give every example a unique hardcoded USB PID (0x4001-0x4022, usbtest
keeps 0x4010) instead of the PID_MAP interface bitmap: different
examples now always re-enumerate back-to-back, even on boards whose
CPU reset does not drop D+ (WCH CH58x), so the EXAMPLE_PID table and
same-PID adjacency reordering in hil_test.py are gone; only the
variant-boundary same-example repeat needs a swap.
- Report matrix: stable columns with the metric-bearing tests pinned
first (usbtest, cdc_msc_throughput, msc_file_explorer[_freertos]),
the rest alphabetical.
Fail fast:
- enum wait budget 8 s on the first attempt, 4 s on retries; dfu waits
are deadline-based so dfu-util's own runtime counts against the
budget.
A device-absent failure now costs ~3-5x a passing test (20-30 s)
instead of 10-30x (47-150 s).
- CI runs hil_test with --retry 1 and no in-run second pass: a broken
fixture fails the job fast instead of holding the self-hosted runner
for hours and blocking other PRs' HIL jobs. hil_test still writes the
.skip sidecar, so a manual re-run attempt only retests what failed.
Review fixes (multi-agent adversarial review of this commit):
- tinyusb_win_usbser.inf: the PID rework moved five CDC examples onto
even PIDs the INF's odd-only DeviceList never matched (legacy-Windows
usbser binding) - appended 0x4006/4008/400a/4020/4022 to both lists.
- usbtest example: USBTEST_TIER is now overridable and the descriptors
and pumps are tier-conditional, so a board whose DCD cannot serve a
tier lowers it instead of skipping the whole example - RA2A1 (RUSB2
with no isochronous pipe) builds at tier 3 via its BOARD_ define; the
host battery follows the tier advertised in bcdDevice. Tier-4 output
verified byte-identical after the refactor.
- dynamic_configuration's second config derived USB_PID + 11 = 0x4018,
colliding with net_lwip_webserver - now USB_PID + 0x0100, outside the
per-example space. tools/check_example_pids.py (pre-commit hook)
enforces PID uniqueness incl. derived and literal idProduct values.
- usbtest.py firmware gate: matched by device ID (uPD720201/720202,
both use the 0x6c FW register), and an unreadable version (setpci
missing/denied) now refuses with its own message instead of
masquerading as "firmware 0x00000000"; noted the gate is necessary
but not sufficient (board-specific kills stay per-board skips).
- hil_test: deadline waits use time.monotonic(); multiprocessing
context pinned to fork (raw semaphores in Pool initargs); flash and
usbtest permits unified into one fail-closed, exception-safe
ctrl_permit (unknown controller takes every slot and logs a warning
instead of silently borrowing slot 0); an all-skipped battery
reports as skip, not "0/0" failure; slow-body polls (mtp, printer,
disk read) go through a shared deadline-based wait_until so their
bodies count against the enum budget; throughput's FS detection
compares serials case-insensitively like every other walk; a missing
MSC read-speed line now fails the host msc_file_explorer test
instead of passing with an empty metric.
Hardening:
- fail fast (15 s) when a driver-registry sysfs write blocks: a wedged
device otherwise turns every subsequent battery into an unkillable
D-state writer and silently hangs the whole run
- usb-recover skill: a VM reboot is not a reliable cure (MosChip hubs
latch up across the PCIe reset); full host power cycle is
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HeF2gZ1M7GWkz6Av4BpKPg
|
|
- changelog: fix 3 silent RST mis-renders carried over from the old monolith
— Markdown link in 0.7.0, mismatched/single backticks in 0.18.0 / 0.20.0
- make-release skill contributors one-liner: parallelize the gh fetch, anchor
the bot filter (was a substring that dropped handles like "abbott"), and join
with ", " (paste -sd cycles the delimiter -> "@a,@b @c"); finalize now stages
only the reviewed set (git add -A -- ':!.idea'); drop the obsolete CRLF gotcha
- make_release.py: emit LF not CRLF in the repository.yml insertion (fixes the
gotcha at the source)
- docs sidebar: drop the hardcoded furo component list; override brand.html to
include furo's own template + the sponsor button (decoupled from furo
internals, no upper-bound pin needed); move sponsor styles into custom.css
- .pre-commit-config.yaml: drop stale end-of-file-fixer excludes for the
deleted contributors/CoC include shims
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
|
|
|
|
|
|
|
* update chipidea dcd, remove manual ep_count and use DCCPARAMS to get number of endpoint instead
* add dcd dcache for chipidea
* add cmake for lpc18
* add makefile build for mcx
* use fork of mcu sdk
* fix ci build with nrf
* flash rp2040 with openocd
|
|
|
|
tuh_itf_info_t
|
|
|
|
|
|
|
|
temporarily disable codespell
|
|
- remove submodules init
- pre-commit: forbid new submodule, run in ci
|
|
|
|
|
