| Age | Commit message (Collapse) | Author |
|---|
|
ci: an empty selection must build nothing, plus selector follow-ups
A PR whose build axis legitimately selected nothing rebuilt everything.
build.yml reads .build.families twice - as a |-joined regex, and implicitly
as "is anything selected" - but tested only -z "$FAMILY_REGEX", which an
empty list and a charset-rejected one both satisfy while meaning opposite
things. ci_set_matrix had already returned the correct all-empty matrix;
the fall-open branch discarded it. #3842 and #3840 each spent 74 cmake legs
on it. Branch on the two cases instead, rename FAM_* to FAMILY_*, and cover
the block with a test that extracts it from build.yml and executes it - it
had no test at all, which is how this shipped through two merges.
Follow-ups to the same machinery: glob.escape the repo root at five sites,
so a checkout path containing [ or * stops failing closed; drop the ci-full
label, read after the matrix was already computed and so never functional;
delete 13 mcu:MKL25ZXX / mcu:SAME5X skip tokens matching no board; carry the
rule table in the module docstring, guarded against drift; and pin six
selection behaviours a mutation pass proved untested.
Cut the selector's cost 1.8x (26.0s -> 14.6s) with 0 divergences over 260
paths, and stop scoping the membrowse upload by the PR example filter.
|
|
Every one of these dropped coverage silently - the worst failure mode here,
because the PR still goes green. Found by review, each reproduced first.
Selection rules:
* class_macros derived the config macro from the class DIRECTORY, so a change to
src/class/midi/midi2_device.c selected the midi_test examples (which do not
compile it) and never examples/device/midi2_device (the only one that enables
CFG_TUD_MIDI2, and the only one that does). The file's own macro is unioned in
where it differs - union, never replace: over-selecting costs a build,
under-selecting merges a break.
* the ${FAMILY_MCUS} fallback added for espressif fired on any family whose
_family_mcus came back empty, and _cmake_sets is if()-blind and keeps the FIRST
definition - so mcx/frdm_mcxn947 answered MCXA15, a token six examples' skip.txt
names, dropping 12 firmware images CMake builds. Limited now to families that
never spell set(FAMILY_MCUS ...) at all.
* lib_examples read only an example's top-level CMakeLists.txt/Makefile;
host/msc_file_explorer_freertos names lib/embedded-cli in src/CMakeLists.txt and
survived by luck. The whole example tree is scanned. (SEGGER_RTT and rt-thread
still resolve to nothing: all three references sit inside a LOGGER=rtt guard no
CI build sets - the documented ruling, not a miss.)
* get_family_boards applied ci_skip_boards/ci_preferred_boards only under
GITHUB_ACTIONS/CIRCLECI, so the selector answered differently on a laptop than
on a runner; _prune_buildable forces CI semantics. Its one-board pick also
abandoned the whole preferred list when entry one could not build the -e set,
and asked skip_example without the build's -D tokens.
* _config_enables and lib_examples still read with the locale encoding - under
LC_ALL=C the selector tracebacked on three tracked tusb_config.h files. The
whole selector and its suite run clean there now.
Workflows:
* the Membrowse Upload step omitted $EX_ARGS, but --one-first now picks the board
from the -e set, so it configured a different, empty build dir and uploaded
--identical for a board never compiled. It takes $EX_ARGS for the BOARD; the
target stays the aggregate, which has no DEPENDS and still records every example.
* blanking FAM_REGEX reset only build_filtered, leaving the build scoped while
code-metrics took the UNSCOPED branch and diffed a 1-family run against the full
averaged baseline. All three drop together now, as CircleCI's fall-open does.
* CircleCI's EX_ARGS had no character screen and is used unquoted, and its
code-metrics job still exit 1'd on an empty metrics set - which a scoped build
makes a legitimate outcome.
* a `ci-full` PR label now turns the scoping off for one PR. A selector bug
under-selects silently, and without a label the only ways back to a full matrix
are accidental.
Performance, since the selector gates every other job: family.cmake texts are read
once rather than per changed directory (a 6,000-file dep bump re-read 84 files
99,892 times) and _scrape_mcu is cached: 2.2s -> 0.29s there, 0.8s -> 0.33s on a
class diff.
Tests: a drift guard for hw/bsp families absent from ci_set_matrix.family_list
(they select zero legs now, where they used to ride the full matrix); the rule-4
port test asserted a SUBSET, which set() satisfies, so it could not fail on the
empty selection it exists to catch; the GITHUB_ENV guard test counted a SUM of two
guards. Drops metrics.py's --only-examples, which nothing called, and applies the
TOTAL scrub to the by-example branch that skipped it.
|
|
Every PR built all 74 legs (2494 example builds on GHA cmake alone) and flashed
all 30 rig boards, whatever it touched. One classifier now walks the PR diff twice
and answers three questions: which families to build, which examples per family,
and which boards run which tests. Fail-open throughout - anything no rule
classifies, any exception, any unusable output falls back to the full matrix, and
a master push always builds everything.
test/hil/helper/hil_select.py moves to tools/ci_select.py: it is no longer HIL-only,
and tools/ is where the build side can import it. test_hil_select.py follows it as
test_ci_select.py.
Rules (docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md holds the
full table): a port selects the families whose family.cmake references it, and its
role - a dcd change skips host examples and vice versa; a class selects only the
examples whose tusb_config.h enables its CFG_TU[DH]_ macro, following cross-class
includes; an example selects itself; hw/bsp selects its family or board; hw/mcu and
lib select whoever references them. CMake is the reference for all of it - make
follows whatever cmake decides, family.mk is never scanned.
Empty means empty (maintainer ruling): a rule that classifies a path to nothing
selects nothing. Ports no family references, classes no config enables, libs no
example builds and hw/mcu paths that resolve nowhere are all real - nothing
compiles them, so nothing can validate them, and the master-push build is the net.
Structural tests pin each such case with an explicit allowlist, so the day one
stops being empty it fails pre-commit instead of silently narrowing CI.
Per-example builds: build.py grows a repeatable -e, resolved against the targets
CMake actually registered and batched into one `cmake --build --target a b c`.
build_utils mirrors CMake's family_filter (the whole FAMILY_MCUS list, ${...} and
string(TOUPPER ...) resolved) for the cmake side, while the make side keeps
master's algorithm verbatim - the two build systems answer differently and a shared
answer breaks lpc54's make link. hil-build gains this even on a full selection:
1702 example builds become 515.
Transport: the selection travels as a file, never an argv or env var - a mass-sweep
diff selects 261 KB against a 128 KiB exec limit, and E2BIG would fail the step
before its own fallback could run. CircleCI carries the example map inside the
generated config (pipeline parameters cap at 512 chars), swapped into the parameter
defaults by sentinel match, and drops the scoping wholesale if that rewrite fails.
Every PR-derived value written to $GITHUB_ENV/$GITHUB_OUTPUT is character-screened.
Code metrics follow the scoping: metrics.py emits per-example totals, and
metrics_pair_compare compares the (board, example) pairs present on both sides
instead of a scoped run against a full-matrix average.
The selector's own suite gates it in both providers: a selector that exits 0 with
valid-but-wrong JSON is the one failure fail-open cannot catch, so a red suite
means the full matrix.
|
|
A wedged USB device used to take the whole HIL run with it. Every worker that
touched the poisoned node blocked uninterruptibly, the pool could not be joined,
map_async discarded every board's result, and the job ran to the GitHub ceiling
with no report at all -- while the self-hosted runner's single job slot stayed
occupied and every queued job waited behind it.
Bound the calls a worker makes itself. read_sysfs, bounded_open and run_cmd all
answer within a wall clock; read_sysfs distinguishes "absent" from "unknown",
because a blocked read is not evidence of absence, and caps stranded readers at
four (each costs a thread and an fd for the life of the process) after which the
worker declares itself blind. mtype, the gio unmount, the libmtp session and the
arecord/iperf reaps go through those bounds; the MTP session runs in a disposable
subprocess, since libmtp's ctypes calls block unkillably in D state.
Bound the run. A pool guard (HIL_POOL_TIMEOUT, 60 min) fires before any job
ceiling and still writes a report. When the pool will not shut down, the sweep
kills what the workers spawned -- descendants, not just direct children, since
flashers run in their own session -- confirms each kill actually landed, and
exits early so the runner is freed. Whatever survived is named in the report.
Deliberately shallow past that point. We do not re-scan process groups, prove
pid ownership, or escalate through sudo: a root-owned survivor is reported, not
force-killed, because signalling a pid we cannot prove is ours is the worse
failure, and the job ceiling backstops whatever this misses. A D-state holder
was never killable anyway.
Recover instead of reporting a wedge. A HUNG usbtest case reflashes its own DUT
through its roster flasher, but only where the flasher can reach its probe past
a poisoned node -- openocd pinned to a validated vid_pid, or esptool. Where it
cannot, the run says so rather than reserving budget for a path that cannot fire.
Raise the CI ceilings above the pool guard so the guard fires first and still
writes its report, and pin --retry 1 on every HIL leg: the guard is a flat
constant and does not scale with max_retry, so argparse's default of 3 would
triple the serialized usbtest tail against an unchanged guard.
Split the module: execution in hil_test/hil_flash/usbtest, infrastructure in
helper/ (locking, health, selection, shared bounded IO), and the two matrix
generators into .github/scripts/ -- ci_set_matrix.py sat in workflows/, where
GitHub treats every file as a workflow definition. 193 tests cover the bounded
paths, the kill ladder, the guard and the selector against synthetic /proc trees
and PATH-injected fakes; a real wedge cannot be manufactured on demand.
|
|
hil, ci: scope HIL builds and tests to the boards a PR affects
Add test/hil/hil_select.py, a stdlib-only selector that maps a PR diff to the
rig boards, tests and BSP families a change can affect, and wire it into CI so
pull requests build and run only those. A port change picks its families' boards,
a class change picks the examples enabling that class, and device/host changes
prune the other role. Anything unclassified — infra, an unmapped port, a selector
error — falls back to the full matrix, and push/schedule runs are untouched.
Move the shared example lists to hil_examples.py; 54 hardware-free tests cover
the rules.
|
|
A re-run attempt merged into an empty base: another PR's HIL job ran
between attempt 1 and the retry and rewrote the shared hil_report.json,
so the run-stamp guard (correctly) refused the foreign base but the
full-fleet results were lost - the retry report contained only the
re-run cells.
Give each (run id, job) its own report dir instead:
- attempts of the same run share a dir, so the retry always finds its
own sidecar and .failed spec intact
- interleaved runs of other PRs/jobs write elsewhere and cannot clobber
- the run-stamp mechanism (.failed.run file) becomes redundant and is
removed
- stale per-run dirs are pruned after 2 weeks
|
|
Full-fleet profiling (HIL_PROFILE=1 instrumentation, included) showed each
uPD720201 controller's serialized usbtest battery chain dominates wall time,
and a board whose marginal device port bounces during concurrent batteries
can wedge or kill the controller ("xHCI host not responding to stop endpoint
command"). Every such death traced to mimxrt1015's port (its old "kills the
uPD720201" reputation) - it is removed from the config until recabled;
mimxrt1064's enum-retry stalls were a loose device cable (re-seated).
nrf54lm20dk moves to boards-skip until its failing J-Link probe is replugged.
With the hardware fixed both cards run width-4 batteries plus full flash
churn clean, so scheduling stays simple: two symmetric knobs, flashes and
batteries budgeted per controller.
- schedule_boards(): dispatch boards round-robin across host controllers from
a persisted hint cache (~/.cache/tinyusb-hil/ctrl_cache.json), learned and
merge-on-write refreshed each run (concurrent HIL jobs keep each other's
entries). Only the cached PCI address is consumed - dispatch order and
first-flash budgeting, never battery serialization (batteries resolve live
or fail closed to an all-slot permit).
- HIL_FLASH_PARALLEL (8) and HIL_USBTEST_PARALLEL (4) are budgeted per
controller via lock slots assigned on first sight.
- re-runs: a failed run writes <report dir>/<config>.failed with the exact
re-run spec (--accumulate -b <failed board> -bt <board>:<its failed
tests>) instead of the inverted --skip-board list of everything that
passed; --skip-board is gone, --flasher/--exclude-flasher scope a config
across CI jobs by flasher type (no board names hardcoded in workflows),
and -a/--accumulate merges a re-run into the existing report. The spec is
stamped with GITHUB_RUN_ID and cleared on fresh runs, so a retry can never
consume a spec left behind by a different run's dead or skipped attempt.
- CI: esp-idf firmware builds move out of hil-build into hil-build-esp, and
the esptool-flashed boards run in their own hil-tinyusb-esp job, so the
main hil-tinyusb run starts as soon as the fast toolchains finish instead
of waiting on the slow esp-idf build (an esp toolchain flake previously
skipped the whole rig run). Artifacts are namespaced per toolchain so the
esp job downloads only esp-idf binaries.
- HIL_PROFILE=1: timestamped log lines, per-flash durations, permit-wait
logging, uid->controller map dump for analysis.
- hil_report: per-variant test duration as a dedicated trailing column,
recorded only by full runs.
Validated on the ci rig (fixed seeds 20260716/777, full fleet at 8/4):
738s/780s walls with only known-flake failures and no controller deaths,
vs 1134-1211s serialized-battery baseline.
|
|
Pool/config:
- record real uids (ra8m1_ek), enable usbtest for espressif s3/p4, then
park ra6m5_ek and ra8m1_ek in boards-skip (ra6m5's usbtest/MSC traffic
can kill the uPD720201 host on its ROM firmware; ra8m1 USBHS bring-up
pending); max32666/nrf54lm20 stay enabled - their MosChip flakiness
never wedges
- re-enable device/usbtest on HS boards (mimxrt1064, ch32v307) now that
uPD720201 firmware 2.0.2.6 fixes the command-ring death; mimxrt1015
stays skipped - its HS battery killed the controller on both ROM and
2.0.2.6 firmware (board-specific); match the moved host-test bundles
(f723 <-> rt1064); skip never-passing tests on the new
nrf5340dk/nrf54lm20dk boards and the detached pico host bundle, each
documented with a comment
Host-controller quirk gating in usbtest.py (auto-skip, self-heals on a
healthy xHCI):
- MosChip MCS9990 EHCI: case 25 (int-OUT never scheduled, FRINDEX bug)
and case 11 (unlinked reads complete short/EREMOTEIO)
- Renesas uPD720201 xHCI: firmware-gated. The card must run firmware
>= 2.0.2.6 (RAM-uploaded - it reverts to ROM on every power cycle):
on older firmware the command ring dies under unlink stress (a
Configure Endpoint command stops completing; the hub worker deadlocks
holding the device lock; only a host power cycle recovers; three
boards reproduced it). usbtest.py reads the FW version register (PCI
config 0x6c) and refuses to run at all on older firmware - hil_test
surfaces that as a failed test with the reason. On current firmware
the full 30-case battery runs (validated FS+HS: metro_m4, f723,
f723-DMA all 30/30).
Scheduling (hil_test.py):
- Shuffle each (board, variant)'s test order with a seeded RNG
(HIL_SHUFFLE_SEED to replay) so usbtest batteries and flash churn spread
across the timeline instead of convoying on one controller.
- Per-controller usbtest + flash semaphores: HIL_USBTEST_PARALLEL
(default 4) concurrent usbtest batteries and HIL_FLASH_PARALLEL
(default 8) concurrent flashes per host controller. Profiled on
uPD720201 firmware 2.0.2.6 across 8/1..12/8: wall time falls
22.2/14.3/12.5/10.8 min at usbtest width 1/2/3/4 and plateaus there;
zero controller errors everywhere; first battery case failures
(leaf-hub bandwidth stretch) appear at 12/8, and flash width 12 only
amplifies flasher-hub contention flakes - so 8/4 is the optimum. A
separate battery-window flash throttle was profiled and dropped.
- Give every example a unique hardcoded USB PID (0x4001-0x4022, usbtest
keeps 0x4010) instead of the PID_MAP interface bitmap: different
examples now always re-enumerate back-to-back, even on boards whose
CPU reset does not drop D+ (WCH CH58x), so the EXAMPLE_PID table and
same-PID adjacency reordering in hil_test.py are gone; only the
variant-boundary same-example repeat needs a swap.
- Report matrix: stable columns with the metric-bearing tests pinned
first (usbtest, cdc_msc_throughput, msc_file_explorer[_freertos]),
the rest alphabetical.
Fail fast:
- enum wait budget 8 s on the first attempt, 4 s on retries; dfu waits
are deadline-based so dfu-util's own runtime counts against the
budget.
A device-absent failure now costs ~3-5x a passing test (20-30 s)
instead of 10-30x (47-150 s).
- CI runs hil_test with --retry 1 and no in-run second pass: a broken
fixture fails the job fast instead of holding the self-hosted runner
for hours and blocking other PRs' HIL jobs. hil_test still writes the
.skip sidecar, so a manual re-run attempt only retests what failed.
Review fixes (multi-agent adversarial review of this commit):
- tinyusb_win_usbser.inf: the PID rework moved five CDC examples onto
even PIDs the INF's odd-only DeviceList never matched (legacy-Windows
usbser binding) - appended 0x4006/4008/400a/4020/4022 to both lists.
- usbtest example: USBTEST_TIER is now overridable and the descriptors
and pumps are tier-conditional, so a board whose DCD cannot serve a
tier lowers it instead of skipping the whole example - RA2A1 (RUSB2
with no isochronous pipe) builds at tier 3 via its BOARD_ define; the
host battery follows the tier advertised in bcdDevice. Tier-4 output
verified byte-identical after the refactor.
- dynamic_configuration's second config derived USB_PID + 11 = 0x4018,
colliding with net_lwip_webserver - now USB_PID + 0x0100, outside the
per-example space. tools/check_example_pids.py (pre-commit hook)
enforces PID uniqueness incl. derived and literal idProduct values.
- usbtest.py firmware gate: matched by device ID (uPD720201/720202,
both use the 0x6c FW register), and an unreadable version (setpci
missing/denied) now refuses with its own message instead of
masquerading as "firmware 0x00000000"; noted the gate is necessary
but not sufficient (board-specific kills stay per-board skips).
- hil_test: deadline waits use time.monotonic(); multiprocessing
context pinned to fork (raw semaphores in Pool initargs); flash and
usbtest permits unified into one fail-closed, exception-safe
ctrl_permit (unknown controller takes every slot and logs a warning
instead of silently borrowing slot 0); an all-skipped battery
reports as skip, not "0/0" failure; slow-body polls (mtp, printer,
disk read) go through a shared deadline-based wait_until so their
bodies count against the enum budget; throughput's FS detection
compares serials case-insensitively like every other walk; a missing
MSC read-speed line now fails the host msc_file_explorer test
instead of passing with an empty metric.
Hardening:
- fail fast (15 s) when a driver-registry sysfs write blocks: a wedged
device otherwise turns every subsequent battery into an unkillable
D-state writer and silently hangs the whole run
- usb-recover skill: a VM reboot is not a reliable cure (MosChip hubs
latch up across the PCIe reset); full host power cycle is
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HeF2gZ1M7GWkz6Av4BpKPg
|
|
* ci: post HIL report comment from workflow_run so it works on forked PRs
|
|
* hil: enable nanoch32v203 in CI with fsdev + usbfs variants
nanoch32v203 was parked in boards-skip; move it into the active pool now
that the board is wired to the ci.lan rig. Cover both USB device IPs as
build variants:
- nanoch32v203-fsdev: RHPORT_DEVICE=0 (USBD / stm32 FSDev IP)
- nanoch32v203-usbfs: RHPORT_DEVICE=1 (WCH USBFS IP)
|
|
* test/hil: replace build.flags_on with named variant schema
Boards declare build variants as `variant: [{name, flags}]` instead of
`build.flags_on`. The variant `name` is the build dir (cmake-build-<name>) and
the HIL report row; `flags` is the raw CFLAGS string (-D...=1) injected via
CFLAGS_CLI. No `variant` => a single build named after the board.
- build.py: --build-name <name> (dir) + --cflag=<token> (raw CFLAGS, repeatable,
=form survives the matrix's shell word-splitting); drop -f1/CFLAGS wrapping.
- hil_ci_set_matrix.py: emit one build arg per variant.
- hil_test.py: iterate variants; report row + build dir = variant name.
- hil_ci.sh: copy all cmake-build-<board>* dirs for -b runs.
- get_deps.py: accept (ignore) --build-name/--cflag from matrix args.
- tinyusb.json: migrate all 6 flags_on boards to variant.
* board_test: park CI build with busy spin instead of wfe
|
|
The Size Difference Report and HIL comments rendered their titles at h1, which
is oversized inside a PR comment. Use h2 for both titles (with subsections
demoted to h3 to keep the hierarchy), and rename the HIL comment from
"HIL test results" to "Hardware-in-the-loop (HIL) Test Report" for consistency.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
|
hil-hfp-iar runs hil_test.py on hfp.json built with IAR on its own rig. Upload
its report as the hil-report-hfp-iar artifact and add the job to the hil-report
combine job's needs, so the sticky comment shows a third table for the IAR rig
alongside tinyusb.json and hfp.json (gcc). The combine gate now runs if either
HIL job produced results.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
|
hil_test.py persists results in a hil_report.json sidecar and regenerates
hil_report.md from it. A full run starts fresh; a re-run (--skip-board / -bt,
i.e. the .skip file) merges into the existing report so already-passed
boards/tests are preserved while only re-run cells update. The report dir is
configurable via HIL_REPORT_DIR.
build.yml: each HIL rig writes the report to a workspace-sibling dir that
survives the per-attempt workspace clean, and uploads it as an artifact. A new
hil-report job merges the rigs' reports into one sticky PR comment (marocchino)
with one table per rig.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
|
* ci: carry metrics baseline forward on no-code-change pushes
The code-metrics job is gated on code_changed and only uploads the
metrics-tinyusb artifact on push, so a workflow/docs-only push to master (e.g.
removing an unrelated workflow) leaves the latest master Build run without a
baseline. PRs download the baseline from the latest master run, so the size
comparison then finds nothing and silently falls back to absolute sizes.
Add a small metrics-carry-forward job that, on a non-code-change push, downloads
the previous metrics-tinyusb artifact and re-publishes it, so the latest run
always carries a usable baseline. Carry-forward runs re-upload too, so the
baseline chains across consecutive no-code pushes (bounded by artifact retention).
|
|
|
|
Co-authored-by: Copilot <[email protected]>
Signed-off-by: Zixun LI <[email protected]>
|
|
|
|
Co-authored-by: Copilot <[email protected]>
Signed-off-by: HiFiPhile <[email protected]>
|
|
Co-authored-by: Copilot <[email protected]>
Signed-off-by: HiFiPhile <[email protected]>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
membrowse-upload upload with --identical if elf file does not exist
|
|
|
|
Co-authored-by: hathach <[email protected]>
|
|
Co-authored-by: Copilot <[email protected]>
|
|
register commit chain with membrowse
|
|
update build.yml only build with pr and push-to-master
|
|
|
|
|
|
|
|
|
|
* upload metrics.json and metrics compare to release page
* Adjust workflow comment handling for forks
|
|
|
|
|
|
|
|
* also render unchange table
* run metrics with circleci (full all boards build)
|
|
* run cmake ci build on github with a fixed set of board to keep the size stable
* Size Difference Report contain major >1% and minor <1& table
|
|
* fine tune ci to build more with circleci
* skip make for arm-iar, esp-idf
* skip make + clang for circleci since llvm-objcopy got killed due to memory issue.
|
|
|
|
fine tune ci matrix run
|
|
|
|
|
|
* change armgcc setup to manual download due to issue with action
* build windows, macos with cmake as well
|
