tinyusb.git/.github/scripts/ci_set_matrix.py, branch rework-metrics

ci-pinned-boards: curate the board set, esp upload via hil-build-esp, rp2040 fixes

2026-09-06T08:57:00Z

The cmake job builds and uploads every pinned board to membrowse on every run, PRs and pushes alike (--identical when code did not change) - it is the single owner of that upload, as before this branch touched it. Curate the board set itself. cxd56/spresense, da1469x/da1469x_dk_pro, fomu/fomu and ft9xx/mm900evxb are no longer pinned CI boards: none sit on either HIL roster and no other CI board carries their driver, so their size tracking was build-only with no hardware behind it - documented with honest `uncovered` reasons instead of silently dropping coverage. Swap lpc15/lpcxpresso1549 -> lpc11/lpcxpresso11u68 for dcd_lpc_ip3511, lpc17/lpcxpresso1769 -> lpc40/ea4088_quickstart for dcd_lpc17_40+ohci, and samd5x_e5x/same54_xplained -> metro_m4_express; adafruit_fruit_jam now also claims dcd_rp2040. All four changed boards build-verified (device/cdc_msc, cmake), and the rp2040 group's (family, board) sort order restored. Two stale notes the swap left behind are fixed: lpcxpresso11u68 is on neither roster, so its "HIL (htpc)" claim (copy-pasted from the entry it replaced) was false, and ea4088_quickstart is a maintainer bench board noted the same way frdm_kl25z and lpcxpresso1549 were. Rename the board file to .github/ci-pinned-boards.json, with matching --ci-pinned-boards/--ci-pinned-boards-only flags: it pins the board that represents each family in CI, and membrowse history is keyed on each pinned board's / target names, so the set is explicitly held rather than derived. Content and target names are unchanged; the per-entry "family" key is dropped - board names are unique across hw/bsp/*/boards, so it was redundant data that could drift from the tree. build.py's resolve_ci_boards() and the coverage checker derive the family from the board dir, and entries are sorted by board name, the only key left to sort on. espressif builds have no leg in the cmake job (esp-idf is by far the slowest toolchain, and its boards are already enumerated by name for the rig) - hil-build-esp keeps its own upload-membrowse + secrets: inherit instead, and hil-build-esp-identical covers the --identical row for a no-code-change push, the same gap hil-build-esp's own code-changed gate would otherwise leave. build_util.yml's Membrowse Upload step skips any leg whose args carry --build-name, an extra -D or --cflag - a HIL variant build - since it configures with the same -DBOARD= as the plain build and would otherwise fight it for the same upload target name; this protects hil-build-esp's DMA variants specifically. The step also does not scope by $EX_ARGS (kept on the Build step): scoping the upload by the PR's example filter would leave examples outside that filter with no upload attempt at all - a silent history gap rather than the --identical row the design intends. Fork PRs: GHA withholds secrets, so MEMBROWSE_API_KEY reaches membrowse_report.py empty. Master's CMake-expanded bare --api-key fell through to membrowse's GitHub tokenless auth; the wrapper now does the same by omitting --api-key entirely when the env var is empty, rather than exiting - the previous hard-exit was silently hidden behind the workflow's continue-on-error, so every fork PR lost its upload with no visible failure. Retiring ci_skip_boards let CircleCI's plain family sweep reach five rp2040 boards for the first time; two didn't build. pico_sdk fails dual/host_hid_to_device_cdc and dual/host_info_to_device_cdc - a meta-board with no board.h pin defines of its own has no PICO_DEFAULT_PIO_USB_* macros, which a dual-role rp2040 example needs for its PIO-USB host side - so each example gained a skip.txt entry naming the failure. feather_rp2040_max3421 failed host/bare_api on -Wmissing-prototypes for max3421_int_handler in hw/bsp/rp2040/family.c; it is only ever used as a same-file GPIO IRQ callback (no other caller), so it is marked static rather than skip-listing a zero-risk one-line fix. All 8 rp2040 boards now build clean, `-b ` and the bare `rp2040` family sweep alike. drivers_coverage_check.py type-checks `drivers` before iterating: a string previously iterated character-by-character into nonsense "X matches no driver source file" errors, and null raised an unhandled TypeError. Not fixed, flagged for a follow-up: MEMBROWSE_API_KEY reaches membrowse_report.py as a child-process argv (visible in /proc//cmdline on the runner, even though the logged command line is redacted) rather than via the environment. Final-review fixes: the espressif Membrowse Upload docker run only forwarded MEMBROWSE_API_KEY and CI into the container, so membrowse's --github metadata detection (GITHUB_EVENT_NAME/GITHUB_SHA/...) and its fork-PR tokenless auth never saw the actual event - add GITHUB_ACTIONS/GITHUB_EVENT_NAME/GITHUB_SHA/GITHUB_REF_NAME plus a read-only bind mount of GITHUB_EVENT_PATH (containers don't inherit runner env by default; the esp Build step needs none of this, since its `--target all` never builds the *-membrowse/-upload custom targets, so its now-unused `pip install membrowse` is also dropped). check-paths' code filter also gains tools/membrowse_report.py, so a PR touching only that file exercises the real build+upload matrix (ci_select rule 2d) instead of 74 no-op --identical legs. Review fixes: generate_membrowse_sizes()'s all-examples scope (-b with no -e) now averages each file's size across the elfs it appears in instead of summing, matching metrics.py's compute_avg() semantics from the linkermap engine this branch replaced - a file linked by N examples no longer reports ~N times its real size, and the Flash/RAM columns are a real binary's size again; single-example (-e) stays byte-identical (sum, same as before). The Membrowse Upload step (build_util.yml) and hil-build-esp-identical's loop (build.yml) now strip any -e pairs hil_ci_set_matrix.py bakes into a PR-scoped hil-build-esp leg's args before invoking, so a PR-scoped run still uploads every example of the pinned board rather than only the PR-selected subset - closing exactly the silent history gap the step's own comment already warned against. Also: build.py's espressif branch gates the idf.py build path on 'all' being present in build_targets rather than being exactly ['all'], so a combined --target invocation still runs the build instead of silently skipping on a fresh dir; membrowse_compare.py's layout-based bucketing unions the actual flash/ram buckets of a multi-region section's regions instead of assuming any 2+-region split is a flash+ram pair (e.g. RAM_D1/RAM_D2 now stays ram-only); membrowse_report.py's extract_defsyms() dedupes to match its own docstring. Post-validate fixes, folded in. The Membrowse Upload step threads the PR's -e filter through for BOARD ELIGIBILITY only: without it, resolve_ci_boards() ran with examples=None and still resolved the pinned board even when the Build step above had fallen back to a substitute, uploading --identical for a board that run never built. Under --ci-pinned-boards-only tools/build.py now nulls the example filter after board selection, so the upload still touches every example of whichever board got resolved - real rows for what was built, --identical for the rest. same7x earns its pin instead of assuming it: a full family sweep showed five *_freertos examples failing (no FreeRTOSConfig.h wired for same7x - skip.txt'd as family:same7x, naming the reason) and same70_qmtech's board.cmake/mk pointing at a same70n19b_flash.ld that never existed in the tree (the SDK ships none) - the board now carries its own, Q21B's script with the N19B's MEMORY sizes (512K flash / 256K ram per same70n19b.h), like the other boards with a board-local .ld. Both same7x boards now build their full example sets green under the real budget. cxd56 stays OUT of ci_set_matrix's family_list and its BSP reverts to master untouched: the same sweep showed NuttX-header failures across three examples, spresense is unpinned anyway, and a -Wno-error suppression for a family no CI leg builds silences a real diagnostic class for nothing. drivers_coverage_check.py gains two guards: a pinned family that no CI toolchain actually compiles is now a validity error (the dead-pin case the same7x episode demonstrated), and a non-dict entry in "boards" reports one clear error instead of a traceback. The spec and plan docs are updated to the shipped names (membrowse-targets.json / --board-pins / --pins-only never shipped; 71 occurrences), and stale test fixtures naming the deleted tinyusb_metrics target are repointed at examples-membrowse-upload. Validate-loop fixes, folded in. membrowse_compare.py's report_for_elf() now passes the elf's own linker scripts and --defsym values (read from its ninja build graph, exactly as the CI upload path does) to `membrowse report`: with none given membrowse falls back to its default Code/Data regions, which the layout bucketing cannot map, so `.data` lost its flash-side load image and was counted RAM-only (verified on stm32h743eval: region 'Data' without the script, ['FLASH','RAM_D1'] with it). _bucket_from_layout() also unions in the flash side for a BOTH_SECTIONS section whose layout lists only its RAM region - pico-sdk's script never lists `.data` under FLASH, undercounting raspberry_pi_pico's flash by 11.2K. membrowse_onboard.py: `membrowse onboard` checks out and `git clean -fdx`s every historical commit in whatever directory it runs, which would detach HEAD and wipe the deps symlinks in the repo root - so the wrapper runs it in a disposable worktree under cmake-metrics/, and the composed build script relinks deps and reconfigures the (also wiped) build dir before every historical build, not just the first. --build-dirs now includes the example's own directory, so a change to its sources no longer backfills as --identical. Not fixable in the wrapper: onboard applies one commit's linker scripts to every commit in the range (no per-commit override in its CLI).

membrowse tooling: fail loudly, dedup espressif, bucket by memory layout

2026-09-06T08:57:00Z

metrics_compare_base.py caught nothing when the `membrowse` CLI was absent, so the default engine died with a bare traceback after both builds had already run; it now points at `pip install membrowse` or --engine linkermap, documented in the skills. membrowse_report.py's ninja query returned '' on failure, starving the linker-script and --defsym extraction so membrowse silently reported default Code/Data regions and exited 0; it now exits with the failing invocation, and likewise when a successful query yields no linker script without --ld. membrowse_onboard.py passes the extracted --ld-scripts/--def so a backfill matches what CI uploads. Stale bare --ci/--combined examples in the tool and in CLAUDE.md are corrected. espressif was compiled twice per run - once by hil-build-esp for the rig, once by the cmake job's esp-idf leg - so it leaves ci-boards.json, the cmake matrix and ci_set_matrix's family_list (which also removes a never-run CircleCI leg that would have built all 15 espressif boards), and hil-build-esp carries the membrowse upload instead. A separate job, gated on the opposite condition and with no downstream dependents, does the --identical upload on no-code-change pushes without letting the HIL test jobs reach the rig. Flash-vs-RAM bucketing stops guessing from section names. TinyUSB places code and buffers in RAM under names no prefix list can enumerate - .time_critical.tinyusb, NonCacheable, .fast, .ccmram - so RAM deltas read as 0 on exactly the boards that do it. Cross-reference each symbol's section against every memory_layout region's sections list instead: present in two regions means a flash load copy and a RAM run copy (both budgets), one region is classified by region name or, if that is unrecognized, by the ELF section type membrowse already derived from sh_flags. The name table survives as a documented fallback, extended with the vendor flash and RAM section names verified in this repo's linker scripts. Note membrowse 1.2.9 reports every linker-script-derived region's type as UNKNOWN, so the region name is the only region-level signal it surfaces.

metrics: replace the linkermap CI pipeline with membrowse

2026-09-06T08:57:00Z

Membrowse becomes TinyUSB's size-analytics system, in CI and locally. .github/membrowse-targets.json names the boards CI builds per family and the drivers each covers; a checker enforced by pre-commit fails when a dcd/hcd driver (plus ehci/ohci) is neither claimed nor documented in `uncovered`, validates board and family names against hw/bsp, and requires a board claiming an hcd driver to build at least one host/dual example. tools/build.py resolves that file into the build matrix and gates uploads on it. Locally, tools/membrowse_compare.py diffs `membrowse report --json --all-symbols` per source file, keyed on object_file (membrowse 1.2.9 truncates source_file to a basename), and metrics_compare_base.py uses it as the default engine, with `--engine linkermap` as the legacy map.json fallback (still required for --combined/--ci). A guard errors instead of writing a silent all-zero table when the filters match nothing. linkermap stops running on every build: the POST_BUILD hook goes, the explicit -linkermap target and a new examples-linkermap aggregate stay, and family_add_linkermap() no-ops when the dep is not fetched. CI loses the whole linkermap pipeline - the code-metrics job, the sticky PR size comment, release metrics assets, per-family artifacts, metrics_pair_compare.py, the membrowse-onboard workflow and CircleCI's dead tinyusb_metrics lane. The cmake job builds the named boards (esp-idf included, membrowse wired into the IDF docker path via family_initialize_project and MEMBROWSE_LD_OVERRIDE) and uploads only those, keeping the --identical path for code-unchanged runs. ci_select follows: the size tooling no longer runs in any CI build, while the board list decides what a family's legs compile and so forces a full matrix. Adds the espressif, same7x, cxd56 and f1c100s families the board list needs, each build-verified. The upload itself moves out of cmake: family_add_membrowse()'s ~90-line inline bash becomes tools/membrowse_report.py, carrying ninja linker-script extraction with recursive INCLUDE resolution, --defsym collection, map-file detection, --identical fallback and upload composition. The API key is read from the environment at run time, passed via argv and redacted in logs - nothing is baked into build.ninja at configure time, which the CMake-expanded $ENV{MEMBROWSE_API_KEY} used to do. tools/membrowse_onboard.py wraps `membrowse onboard` for history backfill, deriving the build script, elf path and the CI-matching target name (/) so a backfill cannot land under a name CI never uploads to; it refuses to start on a dirty worktree, since onboard checks out past commits in place. A membrowse skill documents local reports, size diffs, board-list maintenance, uploads and backfill; the code-size skill records the new default engine and its linkermap fallback. Three follow-up handoffs capture what was deliberately split out: full linkermap removal, membrowse --combined support, and the dead build_filtered plumbing.

ci: an empty selection must build nothing, plus selector follow-ups (#3845)

2026-08-25T02:46:42Z

ci: an empty selection must build nothing, plus selector follow-ups A PR whose build axis legitimately selected nothing rebuilt everything. build.yml reads .build.families twice - as a |-joined regex, and implicitly as "is anything selected" - but tested only -z "$FAMILY_REGEX", which an empty list and a charset-rejected one both satisfy while meaning opposite things. ci_set_matrix had already returned the correct all-empty matrix; the fall-open branch discarded it. #3842 and #3840 each spent 74 cmake legs on it. Branch on the two cases instead, rename FAM_* to FAMILY_*, and cover the block with a test that extracts it from build.yml and executes it - it had no test at all, which is how this shipped through two merges. Follow-ups to the same machinery: glob.escape the repo root at five sites, so a checkout path containing [ or * stops failing closed; drop the ci-full label, read after the matrix was already computed and so never functional; delete 13 mcu:MKL25ZXX / mcu:SAME5X skip tokens matching no board; carry the rule table in the module docstring, guarded against drift; and pin six selection behaviours a mutation pass proved untested. Cut the selector's cost 1.8x (26.0s -> 14.6s) with 0 divergences over 260 paths, and stop scoping the membrowse upload by the PR example filter.

ci_set_matrix: fall open when no selected family builds anywhere

2026-08-21T09:08:25Z

family_list maps a family to the toolchains that build it, and seven hw/bsp families are in neither: cxd56, efm32, espressif, f1c100s, pic32mz, py32f0, same7x. Scoping to one of them intersected to nothing, so every toolchain key was [], every cmake leg skipped on `if: inputs.build-args != '[]'`, code-metrics took its no-metrics branch, and the PR went green from a build job that ran no compiler. The only signal was a stderr line nothing greps for. Not a coverage regression - master gave the same diff no compile coverage either, since none of the other families compiles same7x's board.h. What is new is that the gap used to be masked by the full matrix and is now the whole answer, and that green now means "ran no compiler" rather than "compiled 64 families". A selection whose families ALL miss is now unusable rather than empty: it prints UNSCOPED, which build.yml and .circleci/config.yml already grep to drop the build extras with it, and emits the full matrix. The two neighbouring cases keep their own answers - an explicit families: [] is still a legitimate nothing-selected, and a partial miss still scopes to the families that do build, noting the rest. The contract test pinned an exact count of fall-open markers, which this would have broken; it now pins the invariant (every message that emits the full matrix carries the marker) and was checked to still fail when a marker is removed. Also corrects the drift guard's note about espressif: hil-build-esp builds its boards by name, but that job is gated on repository_owner, so on a fork an espressif-only PR builds nowhere.

ci: scope the build matrix and the HIL run to what a PR affects

2026-08-21T04:07:27Z

Every PR built all 74 legs (2494 example builds on GHA cmake alone) and flashed all 30 rig boards, whatever it touched. One classifier now walks the PR diff twice and answers three questions: which families to build, which examples per family, and which boards run which tests. Fail-open throughout - anything no rule classifies, any exception, any unusable output falls back to the full matrix, and a master push always builds everything. test/hil/helper/hil_select.py moves to tools/ci_select.py: it is no longer HIL-only, and tools/ is where the build side can import it. test_hil_select.py follows it as test_ci_select.py. Rules (docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md holds the full table): a port selects the families whose family.cmake references it, and its role - a dcd change skips host examples and vice versa; a class selects only the examples whose tusb_config.h enables its CFG_TU[DH]_ macro, following cross-class includes; an example selects itself; hw/bsp selects its family or board; hw/mcu and lib select whoever references them. CMake is the reference for all of it - make follows whatever cmake decides, family.mk is never scanned. Empty means empty (maintainer ruling): a rule that classifies a path to nothing selects nothing. Ports no family references, classes no config enables, libs no example builds and hw/mcu paths that resolve nowhere are all real - nothing compiles them, so nothing can validate them, and the master-push build is the net. Structural tests pin each such case with an explicit allowlist, so the day one stops being empty it fails pre-commit instead of silently narrowing CI. Per-example builds: build.py grows a repeatable -e, resolved against the targets CMake actually registered and batched into one `cmake --build --target a b c`. build_utils mirrors CMake's family_filter (the whole FAMILY_MCUS list, ${...} and string(TOUPPER ...) resolved) for the cmake side, while the make side keeps master's algorithm verbatim - the two build systems answer differently and a shared answer breaks lpc54's make link. hil-build gains this even on a full selection: 1702 example builds become 515. Transport: the selection travels as a file, never an argv or env var - a mass-sweep diff selects 261 KB against a 128 KiB exec limit, and E2BIG would fail the step before its own fallback could run. CircleCI carries the example map inside the generated config (pipeline parameters cap at 512 chars), swapped into the parameter defaults by sentinel match, and drops the scoping wholesale if that rewrite fails. Every PR-derived value written to $GITHUB_ENV/$GITHUB_OUTPUT is character-screened. Code metrics follow the scoping: metrics.py emits per-example totals, and metrics_pair_compare compares the (board, example) pairs present on both sides instead of a scoped run against a full-matrix average. The selector's own suite gates it in both providers: a selector that exits 0 with valid-but-wrong JSON is the one failure fail-open cannot catch, so a red suite means the full matrix.

test/hil, ci: contain a wedged USB stack instead of stranding the runner

2026-08-18T05:19:09Z

A wedged USB device used to take the whole HIL run with it. Every worker that touched the poisoned node blocked uninterruptibly, the pool could not be joined, map_async discarded every board's result, and the job ran to the GitHub ceiling with no report at all -- while the self-hosted runner's single job slot stayed occupied and every queued job waited behind it. Bound the calls a worker makes itself. read_sysfs, bounded_open and run_cmd all answer within a wall clock; read_sysfs distinguishes "absent" from "unknown", because a blocked read is not evidence of absence, and caps stranded readers at four (each costs a thread and an fd for the life of the process) after which the worker declares itself blind. mtype, the gio unmount, the libmtp session and the arecord/iperf reaps go through those bounds; the MTP session runs in a disposable subprocess, since libmtp's ctypes calls block unkillably in D state. Bound the run. A pool guard (HIL_POOL_TIMEOUT, 60 min) fires before any job ceiling and still writes a report. When the pool will not shut down, the sweep kills what the workers spawned -- descendants, not just direct children, since flashers run in their own session -- confirms each kill actually landed, and exits early so the runner is freed. Whatever survived is named in the report. Deliberately shallow past that point. We do not re-scan process groups, prove pid ownership, or escalate through sudo: a root-owned survivor is reported, not force-killed, because signalling a pid we cannot prove is ours is the worse failure, and the job ceiling backstops whatever this misses. A D-state holder was never killable anyway. Recover instead of reporting a wedge. A HUNG usbtest case reflashes its own DUT through its roster flasher, but only where the flasher can reach its probe past a poisoned node -- openocd pinned to a validated vid_pid, or esptool. Where it cannot, the run says so rather than reserving budget for a path that cannot fire. Raise the CI ceilings above the pool guard so the guard fires first and still writes its report, and pin --retry 1 on every HIL leg: the guard is a flat constant and does not scale with max_retry, so argparse's default of 3 would triple the serialized usbtest tail against an unchanged guard. Split the module: execution in hil_test/hil_flash/usbtest, infrastructure in helper/ (locking, health, selection, shared bounded IO), and the two matrix generators into .github/scripts/ -- ci_set_matrix.py sat in workflows/, where GitHub treats every file as a workflow definition. 193 tests cover the bounded paths, the kill ladder, the guard and the selector against synthetic /proc trees and PATH-injected fakes; a real wedge cannot be manufactured on demand.