| Age | Commit message (Collapse) | Author |
|---|
|
The modules were 21% comment, much of it review-cycle argument rather than
guidance -- _kill_kids stated 'descendant by construction, no argv check needed'
twice, eight lines apart. Deleting such comments outright makes maintenance
worse: the next reader simplifies the thing the comment was defending. So the
history moves to the 2026-07-30 fleet-wedge design doc, which gains a trim
addendum recording what was removed, what was deliberately kept, and the rule
that decided each -- the CI ceiling bounds how long a run burns, and does nothing
about state that outlives it.
One comment was not merely long but WRONG: the report wipe carried 'The unlink is
DEFERRED to inside the pool try/except below', which is the opposite of what the
code does -- it sits before Manager() with its own comment explaining why. That
is the failure mode this pass is about, so it is deleted rather than reworded.
Kept everywhere: citations that refute a plausible wrong reading. That
usb_lock_device_interruptible is why the readers are killable, that usblp_mutex
is driver-global, that rawmidi honours O_NONBLOCK where usblp does not.
Two follow-ups are retired with them: pr3803-hil-blindness-reporting.md (there
is no blindness to report any more) and pr3803-usbtest-recovery-reserve.md (the
reserve is derived now). Kept: pr3803-flasher-recover.md, which PR #3832
implements, plus pr3803-pci-rebind-stranding.md and pr3803-hil-iar-rerun-spec.md,
both independent of this work.
|
|
Two layers whose cost was a contract to reason about rather than an outcome.
SYSFS_UNKNOWN was a three-valued return five files had to keep apart, and
misreading unknown as absence was silent: a healthy board reported as a firmware
regression. What it guarded is real -- `serial` is served by usb_string_attr,
which takes usb_lock_device_interruptible (v6.12.96 sysfs.c:141-143), the same
lock a wedged usbfs ioctl holds -- so the BOUND stays, on every caller by
default. usb_scan reads `serial` on every device matching the VID, and hil_lock's
controller_of does that on essentially every board, so one wedged DUT would
otherwise stall every worker, not one. What goes is the third value.
read_sysfs now returns str or None, and the question the third value existed to
answer is asked directly instead, by two predicates that say which question they
answer: sysfs_stranded() is process-wide and sticky, for hil_pool_check's footer
("could anything here be the tool losing sight of healthy hardware?"), and
path_stranded(path) is per-device, which is what usbtest needs to tell a DUT
whose `serial` is held under device_lock from one that genuinely left the bus --
that difference decides whether it performs driver-registry writes that take the
uninterruptible device_lock.
Gone: _SysfsUnknown, SYSFS_UNKNOWN, sysfs_blind, sysfs_blind_note,
note_sysfs_strand, the cross-process blindness publishing and its report banner,
usb_scan's (list, bool) return, usbtest's inconclusive abort, _blind_note's slot
in the result tuple, and bounded_open, whose last caller went in the previous
commit.
The strand memo is rewritten around the one invariant that makes it safe to
reuse: it is keyed by the path's kernfs inode, captured BEFORE the read. A
busport does not change when a board returns to the same physical port, so a
path-only blacklist outlives the wedge and hil_pool_check's own recovery flow --
reset, reflash, wait_device polling that busport -- would never look at the
board again. A re-enumeration destroys the kernfs node and makes a new one, so a
changed inode is the all-clear. Two ceilings bound different things: per path
(_PATH_STRAND_MAX) for a board that flaps while still wedged, and per process
(_STRAND_MAX) as a backstop against RLIMIT_NOFILE, counted per PATH rather than
per reader because hil_pool_check runs four poll threads over one bus.
A board the pool guard never reached is now reported as run-aborted rather than
pool-timed-out, and outranks a stale board-locked cell for the same reason the
pool-timeout cell does.
Both predicates answer conservatively where they are consulted before something
irreversible. path_stranded() covers the paths read_sysfs answered None for
WITHOUT reading -- past _STRAND_MAX it declines to start another reader, and
vouching for a path nobody looked at hands usbtest's fail-CLOSED guard a
fabricated all-clear, running remove_id/unbind against a wedged device. usbtest's
startup lookup carries the same caveat hil_test's absent arm already did, because
its stderr is relayed verbatim into the report cell.
strand_note() survives the removal for the same reason master had it: every
caller that can say "not found" needs the same sentence, and the one site left to
re-invent it got missed -- a wedged-but-enumerated printer was reported as an
enumeration failure, sending a maintainer after firmware.
The two predicates are not interchangeable, and usbtest needs both. Its per-case
verdict is per-DUT -- a peer that stranded at case 2 must not make our board
report wedged at case 29 -- but the finally block's cleanup is process-wide:
remove_id plus an unbind of EVERY interface under the driver, including that
peer's, each taking the uninterruptible device_lock. So the verdict uses
path_stranded() and the global cleanup stays gated on sysfs_stranded().
USBTEST_RECOVERY_BUDGET was an independent 250s that could not actually contain
the ladder it reserved for, which is why usbtest.py carried a _time_left() gate
re-deciding before every step -- with a bare '- 35' for costs paid downstream
that nobody could re-derive. Between them the two produced a recovery that
skipped its own steps for most real hangs.
The reserve is now derived from the bounds usbtest itself declares, per flasher
and per target: a probe reset, a reflash, and the Rescue-DP POR plus retry a
wedged RP DAP needs, plus the settles and hil_util.REAP_GRACE for each bounded
step. The Rescue-DP legs are openocd-only and gated on the RP target cfg, and
a stub reset is screened out, so the reserve tracks each board's real ladder
instead of one fleet number: 390s for the two RP boards -- whose ladder the old
250 could not contain, which is exactly why the gates skipped their steps --
190s for the other seventeen probe-reset boards, and 150s for esptool and
lm4flash, whose reset is a no-op. Changing a bound in usbtest moves the
reserve with it, and a unit test asserts it covers the ladder.
With the room actually reserved, the child runs the ladder straight through:
recovery_steps, _time_left, the three per-step gates and the parsed-but-unused
--outer-timeout are gone. What stays is what decides outcomes -- the convoy_safe
gate, reset-before-reflash, the no_op screen so a stub that resets nothing is not
claimed, and wedged_pids() as the arbiter, because a clean flash only proves the
probe wrote the MCU.
hil_util.py 616 -> 514 lines.
|
|
test_device_printer_to_cdc opened /dev/usb/lp* on the worker itself and let
hil_util.bounded_open abandon a thread when the open blocked. usblp allows one
opener -- usblp_open() returns -EBUSY while usblp->used (v6.12.96 usblp.c) --
so the abandoned thread's fd poisoned the node for every later test that worker
ran. The read half already avoided this by forking; the write half now does too,
via the same run_alongside, and a killed child takes its fd with it.
This removes the only production caller of bounded_open.
|
|
support (#3853)
Promote SEGGER RTT from an inline debugging technique to a standalone skill
backed by one stdlib-only implementation in tools/rtt.py: a CLI and importable
module for console/capture over J-Link (RTTTelnetPort) and OpenOCD (rtt server)
probes, with probe selection by serial or VID:PID, control-block address via
--elf or --addr, bidirectional console, post-mortem ring dump, and
--reset-before-attach for boot-time capture. The HIL harness reads a board's
console over RTT when its probe has no VCOM ("logger": "rtt" plus a LOGGER=rtt
variant define), covering device_info, pool-check aliveness, and CI wiring.
Validated on 22 boards across both backends; 26 unit tests run in pre-commit.
|
|
hil_test.py cannot run on Windows and never could: it imports helper.hil_lock,
whose module-level `import fcntl` is POSIX-only, so the harness fails at import
before a line of it executes. Past that it reads /sys/bus/usb, /dev/bus/usb,
/dev/serial/by-id and /proc, kills by process group, and takes flock board
locks -- none of which Windows has.
So the guards were protecting a platform the code cannot reach:
- run_cmd branched three ways on os.name to decide whether to set
start_new_session and whether to killpg. The non-POSIX arm called p.kill()
instead, which kills only the direct child -- exactly the semantics the whole
containment design rejects, since a flasher run through a shell reparents out
of reach. Dead code that documented the wrong answer.
- hil_test picked multiprocessing's default context on Windows "so it still
IMPORTS there". It does not import there.
- test_device_audio_test_freertos returned 'skipped' on nt before touching
ALSA, in a function only ever reached from a worker that cannot start there.
- Seven @unittest.skipIf(os.name == 'nt') decorators across the two suites.
These were the only ones with a real effect -- the unit tests DO import and
run on Windows, because they stub pyserial and mostly exercise pure logic --
but what they buy is a partially-green suite for a harness that cannot run,
and nothing verifies the set is correct: the hil-test hook only ever runs on
ubuntu-latest, so a missing guard fails silently until someone tries.
Removing them makes the POSIX assumption single and explicit rather than
scattered and half-honoured. Nothing changes on Linux: every removed branch was
the one already taken there.
Removing the run_cmd guards also removes their `else: p.kill()` arms. Those were
the Windows branches, and p.kill() reaches only the direct child -- a flasher run
through a shell keeps grandchildren it cannot touch, which is the semantics this
containment design rejects. RunCmdCleanupShape pins what is left: both cleanup
paths killpg, no try carries an else whose body would run when the kill
SUCCEEDED, and the BaseException path still re-raises. Structural rather than
behavioural because driving a real SIGINT into a blocked communicate() is
timing-dependent, and what actually breaks this block is an edit that rebinds a
branch -- which is a shape.
|
|
(#3852)
hid_generic_inout was the last unbounded blocking IO in the file. hidapi's
hidraw backend reads manufacturer/product via udev for each device reaching
create_device_info_for_device, both usb_string_attr served under the device
lock a wedged usbfs ioctl holds — and every DUT here is VID cafe, so a wedged
sibling stalls the walk.
A thread cannot bound it: cython-hidapi calls hid_open and hid_close bare
(0.15.0 hid.pyx), so they hold the GIL and the waiter can never resume.
Measured — a 1.0s bound never returned. run_cmd's killpg reaches a child
regardless; it gains an argv form for the -c body.
Filters on both ids: hidapi only runs the free uevent pre-check when ids are
passed (linux/hid.c:962), so an unfiltered walk sends every device straight to
the locked reads. Tests stall via ctypes.PyDLL, which unlike CDLL holds the
GIL — the shape a thread bound cannot cover.
|
|
Three readability changes with no behaviour change on the healthy path —
every pre-existing test passes untouched.
main() was 368 lines with try/finally three deep, its two abort paths
near-identical 40-line blocks; _abort_report holds that shape once, and the
controller-hint cache and pool construction move to their own helpers.
368 -> 279, test_board 180 -> 151, test_device_usbtest 164 -> 125.
test_hil_bounded.py cost 78s on every commit under test/hil/, mostly one 3s
post-flash settle paid by ten tests against a fake rig. Now 37s.
Fixes two pre-existing defects the extraction exposed: _write_failed_spec was
unguarded inside the abort path, so an OSError there replaced the caller's
RuntimeError and no report was written at all; _save_controller_hints overlaid
a startup snapshot onto the re-read cache, clobbering a concurrent job's newer
values. Also five comments that stated the opposite of the code, and both table
renderers measuring width with len() against two-column status marks.
|
|
hil_report.json and hil_report.md were written independently. Four writers
produced the markdown and three wrote no JSON at all -- and those three are the
paths where a run died, so a JSON consumer saw nothing exactly when it mattered:
the per-board verdicts an agent hands back reported the whole fleet as "no
report row" while a human read the real story from the markdown.
Every writer now goes through render_report(), so a table can never contain
something the JSON does not. The document gains `scope` (a three-board PR run
and a full run that lost 24 boards were indistinguishable) and `caveat` (how the
run ended). `banner` carries rig health across an --accumulate retry; `caveat`
records how a run ended and must not -- conflating them made a clean retry
publish an abandonment that never happened.
helper/hil_report.py owns the document end to end, dissolving the import cycle
that forced write_timeout_report to compose its own markdown and removing a
duplicate cell classifier kept in sync by hand. hil_summary.py is deleted; its
CLI moves there. hil_ci.sh uploads the sidecar so a remote --accumulate has a
merge bas
|
|
The roster had two ways to pass a cmake -D to a board's build: `build.args`,
applied to every variant, and `variant[].defines`, applied to one. They did the
same thing, and only metro_m4_express used the first - for MAX3421_HOST=1, which
is what makes it the one rig board that compiles hcd_max3421.c.
A board whose define is always on now carries a single variant named after itself,
which is exactly the shape `board.get('variant') or [{'name': name, 'flags': ''}]`
already synthesises everywhere - so the build dir, the HIL report row and the
variant-boundary handling are unchanged. raspberry_pi_pico has used that shape
for its flags all along.
Removes the BuildCfg type and the parallel code path from all four consumers:
hil_test.build_board, hil_pool_check's two builders, hil_ci_set_matrix and
ci_select.board_options.
Verified: the hil-build matrix entry is byte-identical
(`-b metro_m4_express -DMAX3421_HOST=1`), hil_test's build command is unchanged,
ci_select still selects the board for a max3421 diff with MAX3421_HOST in its
options, and a real build of dual/host_info_to_device_cdc and host/cdc_msc_hid on
that board still compiles hcd_max3421.c.
|
|
A wedged USB device used to take the whole HIL run with it. Every worker that
touched the poisoned node blocked uninterruptibly, the pool could not be joined,
map_async discarded every board's result, and the job ran to the GitHub ceiling
with no report at all -- while the self-hosted runner's single job slot stayed
occupied and every queued job waited behind it.
Bound the calls a worker makes itself. read_sysfs, bounded_open and run_cmd all
answer within a wall clock; read_sysfs distinguishes "absent" from "unknown",
because a blocked read is not evidence of absence, and caps stranded readers at
four (each costs a thread and an fd for the life of the process) after which the
worker declares itself blind. mtype, the gio unmount, the libmtp session and the
arecord/iperf reaps go through those bounds; the MTP session runs in a disposable
subprocess, since libmtp's ctypes calls block unkillably in D state.
Bound the run. A pool guard (HIL_POOL_TIMEOUT, 60 min) fires before any job
ceiling and still writes a report. When the pool will not shut down, the sweep
kills what the workers spawned -- descendants, not just direct children, since
flashers run in their own session -- confirms each kill actually landed, and
exits early so the runner is freed. Whatever survived is named in the report.
Deliberately shallow past that point. We do not re-scan process groups, prove
pid ownership, or escalate through sudo: a root-owned survivor is reported, not
force-killed, because signalling a pid we cannot prove is ours is the worse
failure, and the job ceiling backstops whatever this misses. A D-state holder
was never killable anyway.
Recover instead of reporting a wedge. A HUNG usbtest case reflashes its own DUT
through its roster flasher, but only where the flasher can reach its probe past
a poisoned node -- openocd pinned to a validated vid_pid, or esptool. Where it
cannot, the run says so rather than reserving budget for a path that cannot fire.
Raise the CI ceilings above the pool guard so the guard fires first and still
writes its report, and pin --retry 1 on every HIL leg: the guard is a flat
constant and does not scale with max_retry, so argparse's default of 3 would
triple the serialized usbtest tail against an unchanged guard.
Split the module: execution in hil_test/hil_flash/usbtest, infrastructure in
helper/ (locking, health, selection, shared bounded IO), and the two matrix
generators into .github/scripts/ -- ci_set_matrix.py sat in workflows/, where
GitHub treats every file as a workflow definition. 193 tests cover the bounded
paths, the kill ladder, the guard and the selector against synthetic /proc trees
and PATH-injected fakes; a real wedge cannot be manufactured on demand.
|
|
Fix HFP HIL reliability issue
|
|
by flasher extension (#3804)
test/hil: one openocd flasher, per-board verify and firmware extension
The four WCH boards move to `openocd`, leaving one flasher for all.
`verify` is now a per-board opt-out, not dropped fleet-wide: WCH cannot read flash back
over the WCH-Link sdi transport; the other seven openocd boards can, and say so explicitly.
FLASHER_SUFFIX decides each flasher's extension once — find_firmware returns the full path
and the flashers pass it through, so a build with only the wrong artifact is skipped rather
than failed mid-flash. --skip-flash bypasses the filter.
rescue_openocd() power-on-resets a wedged RP2040/RP2350 via its Rescue DP from the flash
retry; the probe has no reset line.
Drops unused openocd_adi, stflash, wlink_rs and uniflash, parks the unstable ra6m5_ek, and
tests that every roster flasher name dispatches.
|
|
|
|
hil, ci: scope HIL builds and tests to the boards a PR affects
Add test/hil/hil_select.py, a stdlib-only selector that maps a PR diff to the
rig boards, tests and BSP families a change can affect, and wire it into CI so
pull requests build and run only those. A port change picks its families' boards,
a class change picks the examples enabling that class, and device/host changes
prune the other role. Anything unclassified — infra, an unmapped port, a selector
error — falls back to the full matrix, and push/schedule runs are untouched.
Move the shared example lists to hil_examples.py; 54 hardware-free tests cover
the rules.
|
|
|
|
probes (#3794)
test/hil: add board-pool health check, split hil_test into focused modules (#3794)
Add test/hil/hil_pool_check.py: per-board rig health scan — probe presence,
light-example flash (dfu_runtime; device_info + serial check for host-only
boards), uid re-enumeration, safe recovery (probe authorized-toggle, board
reset), verified board_test re-park, USB topology report, and a markdown
summary table. Missing firmware is built on the spot (tools/build.py, idf.py
for espressif, one get_deps retry); row statuses: ok, flash-failed, failed,
locked. Board locks are always respected, never bypassed.
Refactor hil_test.py into hil_lock.py (flock protocol, controller permits,
hold/release/status CLI; replaces board_lock.py) and hil_flash.py (flashers,
find_firmware, run_cmd). Update WCH probe uids and the board roster in
tinyusb.json; add the hil-pool-check skill.
|
|
|
|
|
|
|
|
|
|
|
|
This reverts commit 80ffbff6e98a9c5053bba008ae2c5087f0351300.
|
|
bsp, hil: flash with the unified OpenOCD fork
https://github.com/hathach/openocd (branch tinyusb) is mainline plus every
config these boards need: RPi RP2350, ADI max32/max78, the MounRiver WCH
configs, and the wlinke adapter on mainline's riscv target. It is a superset
of the vendor forks, so one 'openocd' covers all boards; -DOPENOCD=/OPENOCD=
still select another, msdk's when MAXIM_PATH is set.
Drops family_flash_openocd_wch and the OPENOCD_WCH pair, dedups
family_flash_openocd_adi, aligns ch583's work area, and points hil at the
flasher's own config instead of generating one per probe.
Verified: HIL green on all four WCH boards and max32666fthr.
|
|
|
|
|
|
A re-run attempt merged into an empty base: another PR's HIL job ran
between attempt 1 and the retry and rewrote the shared hil_report.json,
so the run-stamp guard (correctly) refused the foreign base but the
full-fleet results were lost - the retry report contained only the
re-run cells.
Give each (run id, job) its own report dir instead:
- attempts of the same run share a dir, so the retry always finds its
own sidecar and .failed spec intact
- interleaved runs of other PRs/jobs write elsewhere and cannot clobber
- the run-stamp mechanism (.failed.run file) becomes redundant and is
removed
- stale per-run dirs are pruned after 2 weeks
|
|
Full-fleet profiling (HIL_PROFILE=1 instrumentation, included) showed each
uPD720201 controller's serialized usbtest battery chain dominates wall time,
and a board whose marginal device port bounces during concurrent batteries
can wedge or kill the controller ("xHCI host not responding to stop endpoint
command"). Every such death traced to mimxrt1015's port (its old "kills the
uPD720201" reputation) - it is removed from the config until recabled;
mimxrt1064's enum-retry stalls were a loose device cable (re-seated).
nrf54lm20dk moves to boards-skip until its failing J-Link probe is replugged.
With the hardware fixed both cards run width-4 batteries plus full flash
churn clean, so scheduling stays simple: two symmetric knobs, flashes and
batteries budgeted per controller.
- schedule_boards(): dispatch boards round-robin across host controllers from
a persisted hint cache (~/.cache/tinyusb-hil/ctrl_cache.json), learned and
merge-on-write refreshed each run (concurrent HIL jobs keep each other's
entries). Only the cached PCI address is consumed - dispatch order and
first-flash budgeting, never battery serialization (batteries resolve live
or fail closed to an all-slot permit).
- HIL_FLASH_PARALLEL (8) and HIL_USBTEST_PARALLEL (4) are budgeted per
controller via lock slots assigned on first sight.
- re-runs: a failed run writes <report dir>/<config>.failed with the exact
re-run spec (--accumulate -b <failed board> -bt <board>:<its failed
tests>) instead of the inverted --skip-board list of everything that
passed; --skip-board is gone, --flasher/--exclude-flasher scope a config
across CI jobs by flasher type (no board names hardcoded in workflows),
and -a/--accumulate merges a re-run into the existing report. The spec is
stamped with GITHUB_RUN_ID and cleared on fresh runs, so a retry can never
consume a spec left behind by a different run's dead or skipped attempt.
- CI: esp-idf firmware builds move out of hil-build into hil-build-esp, and
the esptool-flashed boards run in their own hil-tinyusb-esp job, so the
main hil-tinyusb run starts as soon as the fast toolchains finish instead
of waiting on the slow esp-idf build (an esp toolchain flake previously
skipped the whole rig run). Artifacts are namespaced per toolchain so the
esp job downloads only esp-idf binaries.
- HIL_PROFILE=1: timestamped log lines, per-flash durations, permit-wait
logging, uid->controller map dump for analysis.
- hil_report: per-variant test duration as a dedicated trailing column,
recorded only by full runs.
Validated on the ci rig (fixed seeds 20260716/777, full fleet at 8/4):
738s/780s walls with only known-flake failures and no controller deaths,
vs 1134-1211s serialized-battery baseline.
|
|
incompatible
The MosChip MCS9990 card is physically removed from the rig: delete its
cases-11/25 SKIP workaround (and the now-orphaned SKIP accounting) from
usbtest.py and refuse to run outright if a DUT ever sits behind one again.
usb_recover.sh gains `hub-cycle <busport>`: uhubctl VBUS cycle of the port
feeding the device, walking upstream (parent hub -> root port) until it
re-enumerates. Verified on the rig: leaf-level recovery (13-4.4 usbtest
device) and full walk to the root port on a dead branch. SKILL.md updated
for the action and the two-Renesas topology (root-port ppps is real; leaf
1a40:0201 hubs fake their "ganged" switching).
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WxUeX4Yn26KibfjvDg2pN9
|
|
# Conflicts:
# .claude/skills/hil/SKILL.md
# test/hil/hil_test.py
|
|
Pool/config:
- record real uids (ra8m1_ek), enable usbtest for espressif s3/p4, then
park ra6m5_ek and ra8m1_ek in boards-skip (ra6m5's usbtest/MSC traffic
can kill the uPD720201 host on its ROM firmware; ra8m1 USBHS bring-up
pending); max32666/nrf54lm20 stay enabled - their MosChip flakiness
never wedges
- re-enable device/usbtest on HS boards (mimxrt1064, ch32v307) now that
uPD720201 firmware 2.0.2.6 fixes the command-ring death; mimxrt1015
stays skipped - its HS battery killed the controller on both ROM and
2.0.2.6 firmware (board-specific); match the moved host-test bundles
(f723 <-> rt1064); skip never-passing tests on the new
nrf5340dk/nrf54lm20dk boards and the detached pico host bundle, each
documented with a comment
Host-controller quirk gating in usbtest.py (auto-skip, self-heals on a
healthy xHCI):
- MosChip MCS9990 EHCI: case 25 (int-OUT never scheduled, FRINDEX bug)
and case 11 (unlinked reads complete short/EREMOTEIO)
- Renesas uPD720201 xHCI: firmware-gated. The card must run firmware
>= 2.0.2.6 (RAM-uploaded - it reverts to ROM on every power cycle):
on older firmware the command ring dies under unlink stress (a
Configure Endpoint command stops completing; the hub worker deadlocks
holding the device lock; only a host power cycle recovers; three
boards reproduced it). usbtest.py reads the FW version register (PCI
config 0x6c) and refuses to run at all on older firmware - hil_test
surfaces that as a failed test with the reason. On current firmware
the full 30-case battery runs (validated FS+HS: metro_m4, f723,
f723-DMA all 30/30).
Scheduling (hil_test.py):
- Shuffle each (board, variant)'s test order with a seeded RNG
(HIL_SHUFFLE_SEED to replay) so usbtest batteries and flash churn spread
across the timeline instead of convoying on one controller.
- Per-controller usbtest + flash semaphores: HIL_USBTEST_PARALLEL
(default 4) concurrent usbtest batteries and HIL_FLASH_PARALLEL
(default 8) concurrent flashes per host controller. Profiled on
uPD720201 firmware 2.0.2.6 across 8/1..12/8: wall time falls
22.2/14.3/12.5/10.8 min at usbtest width 1/2/3/4 and plateaus there;
zero controller errors everywhere; first battery case failures
(leaf-hub bandwidth stretch) appear at 12/8, and flash width 12 only
amplifies flasher-hub contention flakes - so 8/4 is the optimum. A
separate battery-window flash throttle was profiled and dropped.
- Give every example a unique hardcoded USB PID (0x4001-0x4022, usbtest
keeps 0x4010) instead of the PID_MAP interface bitmap: different
examples now always re-enumerate back-to-back, even on boards whose
CPU reset does not drop D+ (WCH CH58x), so the EXAMPLE_PID table and
same-PID adjacency reordering in hil_test.py are gone; only the
variant-boundary same-example repeat needs a swap.
- Report matrix: stable columns with the metric-bearing tests pinned
first (usbtest, cdc_msc_throughput, msc_file_explorer[_freertos]),
the rest alphabetical.
Fail fast:
- enum wait budget 8 s on the first attempt, 4 s on retries; dfu waits
are deadline-based so dfu-util's own runtime counts against the
budget.
A device-absent failure now costs ~3-5x a passing test (20-30 s)
instead of 10-30x (47-150 s).
- CI runs hil_test with --retry 1 and no in-run second pass: a broken
fixture fails the job fast instead of holding the self-hosted runner
for hours and blocking other PRs' HIL jobs. hil_test still writes the
.skip sidecar, so a manual re-run attempt only retests what failed.
Review fixes (multi-agent adversarial review of this commit):
- tinyusb_win_usbser.inf: the PID rework moved five CDC examples onto
even PIDs the INF's odd-only DeviceList never matched (legacy-Windows
usbser binding) - appended 0x4006/4008/400a/4020/4022 to both lists.
- usbtest example: USBTEST_TIER is now overridable and the descriptors
and pumps are tier-conditional, so a board whose DCD cannot serve a
tier lowers it instead of skipping the whole example - RA2A1 (RUSB2
with no isochronous pipe) builds at tier 3 via its BOARD_ define; the
host battery follows the tier advertised in bcdDevice. Tier-4 output
verified byte-identical after the refactor.
- dynamic_configuration's second config derived USB_PID + 11 = 0x4018,
colliding with net_lwip_webserver - now USB_PID + 0x0100, outside the
per-example space. tools/check_example_pids.py (pre-commit hook)
enforces PID uniqueness incl. derived and literal idProduct values.
- usbtest.py firmware gate: matched by device ID (uPD720201/720202,
both use the 0x6c FW register), and an unreadable version (setpci
missing/denied) now refuses with its own message instead of
masquerading as "firmware 0x00000000"; noted the gate is necessary
but not sufficient (board-specific kills stay per-board skips).
- hil_test: deadline waits use time.monotonic(); multiprocessing
context pinned to fork (raw semaphores in Pool initargs); flash and
usbtest permits unified into one fail-closed, exception-safe
ctrl_permit (unknown controller takes every slot and logs a warning
instead of silently borrowing slot 0); an all-skipped battery
reports as skip, not "0/0" failure; slow-body polls (mtp, printer,
disk read) go through a shared deadline-based wait_until so their
bodies count against the enum budget; throughput's FS detection
compares serials case-insensitively like every other walk; a missing
MSC read-speed line now fails the host msc_file_explorer test
instead of passing with an empty metric.
Hardening:
- fail fast (15 s) when a driver-registry sysfs write blocks: a wedged
device otherwise turns every subsequent battery into an unkillable
D-state writer and silently hangs the whole run
- usb-recover skill: a VM reboot is not a reliable cure (MosChip hubs
latch up across the PCIe reset); full host power cycle is
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HeF2gZ1M7GWkz6Av4BpKPg
|
|
Confirmed by a 10-finder / 28-verifier adversarial review pass:
board_lock.py — the flock is now the sole authority: drop cmd_hold's
pid-liveness pre-gate (a live hil_test.py pool worker's stale record no
longer blocks a genuinely free board); cmd_release probes the flock and
only signals a verified holder, refuses to kill hil_test.py holders
(CI mid-test), handles PermissionError; the holder daemon truncates its
lock records on SIGTERM and keeps the success pipe clear of fds 0-2
(closed-stdio hold used to leave an orphan holder while reporting
failure); --config default resolves beside the script.
hil_test.py — truncate the lock record on per-board release (pool
workers outlive their flocks); warn instead of silently failing open
when the lock dir is unusable; error out on -b names absent from the
config (was a silent zero-test exit 0, readable as a green HIL run);
drop an emptied board row in accumulate_report (variant boards left a
blank ghost row).
workflows — remove the stray positional arg that made the validate size
stage exit 2 on every run; wrap JSON.parse(args) in all six scripts;
factor pr-babysit's drifted reply recipe into postReplyRecipe and dedup
refutation replies across cycles; validate args.pr and maxCycles;
driver-review rejects an empty dimensions list; hil-validate drops a
dead guard clause and retries diagnostics with -v -r 1.
agents/docs — port-dev scopes git clang-format to its own files
(concurrent workers reformatted each other in shared checkouts);
hil-operator/hil skill wording matches actual fail-fast output; the
implementation plan is now a DO-NOT-EXECUTE historical record (banner +
checked boxes) so plan-executing agents cannot revert shipped files.
Verified: lock storm 1-winner-in-10, stale-record hold, closed-stdio
hold, dead-pid cleanup, CI-holder refusal, ghost-row 4-scenario merge,
unknown-board exit 1, py_compile + check.sh on all six workflows,
pre-commit clean.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upj4hta5TNoAbidqeC1zZ6
|
|
Review-fix batch (owner-confirmed) on the multi-agent harness:
- board_lock: detach holder stdio so a captured `hold` cannot hang on the
daemon's inherited pipe; probe locks by holder-pid liveness instead of a
momentary flock, which could spuriously fail a concurrent acquirer
(storm-tested: 1 winner in 10, 0/15 acquire failures under probe storm)
- hil_test: locked board renders a visible board-locked fail row so the
report matches the exit code; stale marker cleared on a real re-run
- pr-babysit: autoPush now opt-in (default dry run); resolve recipe
paginates reviewThreads; post-push resolve gets issue-comment fallback
- validate: size stage honors non-default base via --base-branch; pvs
stage delegated to the new agent
- new static-analyzer agent (sonnet): PVS-Studio SAST+MISRA for one
board, structured findings gated on files changed vs base
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Rn1AN5DsTdFhRwhugfgKZi
|
|
Binds the kernel usbtest driver (gadget-zero profile), runs the tier-based
battery, auto-recovers kernel-side hangs, and skips cases the host
controller cannot run (MosChip MCS9990 EHCI int-OUT).
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HeF2gZ1M7GWkz6Av4BpKPg
|
|
Add worker agents (builder, port-dev, driver-reviewer, hil-operator,
pr-monitor), deterministic workflows (validate, fanout-dev, driver-review,
hil-validate, full-check, pr-babysit) and a /pre-pr gate skill, so sessions
can fan build/test/review/PR-triage work out to tiered subagents. pr-babysit
drives a PR to green: triage CI + bot reviews, fix validated findings, verify,
push, and reply-to + resolve each inline review thread (fixed or refuted).
Replace the stop-the-runner HIL discipline with per-board flock locks:
test/hil/board_lock.py plus a fail-open guard in hil_test.py let CI and dev
sessions share the rig per board (locked boards fail fast and re-run;
HIL_NO_BOARD_LOCK=1 is a user-authorized bypass). The actions-runner is
never stopped.
Design spec, implementation plan, and real-rig smoke evidence under
docs/superpowers/.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Rn1AN5DsTdFhRwhugfgKZi
|
|
Prefix each rig's hil_report.md matrix with a one-line tally
(passed / failed / skipped) so the number of failed tests is visible at a
glance in the PR comment without scanning the table. A metric-string cell
(e.g. throughput) counts as a pass; blank/not-run cells are excluded.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
|
|
|
Add the device-only CH582M-EVT (WCH USBFS via the shared dcd_ch32_usbfs.c),
riscv-gcc, flashed by openocd_wch probe 7FD88F0604B5, to tinyusb.json.
Also reorder device_tests to keep examples sharing a VID:PID non-adjacent:
cdc_msc and cdc_msc_throughput both use cafe:4003, and on boards whose
CPU-reset does not drop D+ (e.g. WCH CH58x via openocd) back-to-back same-PID
firmware leaves the host on the previous example's cached descriptors, so the
new example's CDC never enumerates and the test fails. Moving dfu (cafe:4000)
between them changes the PID and forces the host to re-enumerate.
Remote HIL on ci.lan: all device examples pass, including cdc_msc_throughput
(no skip needed).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
|
arecord (alsa-utils) is a documented HIL host requirement, like
mtools/libmtp9/iperf — none of which have a skip-if-missing guard. The
audio test was the exception: it silently returned 'skipped' when arecord
was absent, masking host misconfiguration. The ci.lan rig had been
silently skipping device/audio_test_freertos on every board because
alsa-utils was never installed.
Remove the shutil.which('arecord') guard so a missing package surfaces as
a failure, consistent with the other tool-dependent tests, and drop the
now-unused shutil import. Note in the host-setup comment that these
packages are required (a missing tool fails its test rather than skipping).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
|
dcd/musb: defer EP0 SETUP during DATA_IN/STATUS race
|
|
* hil: enable nanoch32v203 in CI with fsdev + usbfs variants
nanoch32v203 was parked in boards-skip; move it into the active pool now
that the board is wired to the ci.lan rig. Cover both USB device IPs as
build variants:
- nanoch32v203-fsdev: RHPORT_DEVICE=0 (USBD / stm32 FSDev IP)
- nanoch32v203-usbfs: RHPORT_DEVICE=1 (WCH USBFS IP)
|
|
pyserial's posix write() raises SerialTimeoutException after partial
progress with the byte count lost, so the retry loop re-sent from the
same offset and could duplicate bytes on the wire — surfacing as bogus
data-mismatch failures that look like device firmware bugs.
write_timeout is already a total per-call deadline, so the loop added
duplication risk without extending the budget: write once and treat a
timeout as fatal. Default bumped 2 -> 10 s to keep the old overall
bound; HIL_SERIAL_WRITE_DEADLINE removed.
The per-character CLI loops keep their existing pacing (the 2 ms sleep
between single-byte writes already spaces them on the wire); no
unbounded ser.flush()/tcdrain is added.
Review follow-up for #3643 (hil_test.py l.257/264 findings).
Co-Authored-By: Claude Fable 5 <[email protected]>
|
|
# Conflicts:
# test/hil/hil_test.py
|
|
* test/hil: replace build.flags_on with named variant schema
Boards declare build variants as `variant: [{name, flags}]` instead of
`build.flags_on`. The variant `name` is the build dir (cmake-build-<name>) and
the HIL report row; `flags` is the raw CFLAGS string (-D...=1) injected via
CFLAGS_CLI. No `variant` => a single build named after the board.
- build.py: --build-name <name> (dir) + --cflag=<token> (raw CFLAGS, repeatable,
=form survives the matrix's shell word-splitting); drop -f1/CFLAGS wrapping.
- hil_ci_set_matrix.py: emit one build arg per variant.
- hil_test.py: iterate variants; report row + build dir = variant name.
- hil_ci.sh: copy all cmake-build-<board>* dirs for -b runs.
- get_deps.py: accept (ignore) --build-name/--cflag from matrix args.
- tinyusb.json: migrate all 6 flags_on boards to variant.
* board_test: park CI build with busy spin instead of wfe
|
|
hil: park boards with idle board_test instead of erasing flash
|
|
Teardown: instead of flashing device/board_test (a USB-less blink loop that
keeps the MCU busy-looping), erase the first flash sector (vector table) so the
board faults to idle after its tests — no USB, lower power, faster. Per-flasher
erase_<name>: openocd/openocd_adi `flash erase_sector 0 0 0`; stlink `--erase
0`; jlink erases the sector at the flash origin read from the ELF (pure-Python,
new elf_flash_origin); esptool `erase_region 0x0 0x4000`; lm4flash writes a 4 KB
all-0xFF blank image (lm4flash erases before programming, so the first sector
ends up blank). device/board_test flash remains a fallback for flashers with no
erase_ function. The teardown is no longer a report column (it's cleanup).
Report: cdc_msc_throughput and msc_file_explorer[_freertos] now return a compact
read/write speed shown in their report cell instead of the pass tick (e.g.
"C 652k/422k M 1.1M/783k", "rd 1.2MB/s"). test_example returns an optional
metric; render_matrix shows it verbatim. Firmware lookup factored into
find_firmware (reused by the erase teardown).
Verified on the rig (stm32f723disco, jlink): erase disables the board in 0.8 s
and it disappears from the bus; the throughput cell shows live speeds.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
|
Neutral white circle for skipped, giving a ✅/❌/⚪ pass/fail/skip set.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
|
Colored emoji render green/red in the GitHub PR comment, far more visible than
the monochrome ✔/✖ dingbats. Skip stays ➖.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
|
The report sidecar lives in a persistent dir (it survives the CI workspace
clean so accumulation works across run attempts). A full run is "fresh" and
must not merge prior state, but previously fresh only avoided *loading* the
json at merge time — if a fresh run crashed before writing the report, the
stale json/md from an earlier run lingered and a retry (fresh=False) could
merge it, or the always() upload could post it. Delete hil_report.json/.md at
the start of a fresh run so prior results can never leak.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
|
hil_test.py persists results in a hil_report.json sidecar and regenerates
hil_report.md from it. A full run starts fresh; a re-run (--skip-board / -bt,
i.e. the .skip file) merges into the existing report so already-passed
boards/tests are preserved while only re-run cells update. The report dir is
configurable via HIL_REPORT_DIR.
build.yml: each HIL rig writes the report to a workspace-sibling dir that
survives the per-attempt workspace clean, and uploads it as an artifact. A new
hil-report job merges the rigs' reports into one sticky PR comment (marocchino)
with one table per rig.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
|
Per review (HiFiPhile): Ninja Multi-Config is needed for IAR, otherwise the
optimization level can't be lowered to none for debug. Revert gen_presets.py
back to Ninja Multi-Config (keeping only the cmake-build-<board> binaryDir
change), and instead teach hil_test.py to locate <ex>.elf whether it sits
directly in the example dir (single-config) or under a per-config subdir like
RelWithDebInfo/ (multi-config).
Verified: stm32u083nucleo passes 13/13 remote HIL with a multi-config preset
build (rsync preserves the RelWithDebInfo/ subdir; the resolver finds it).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
