tinyusb.git/tools/ci_select.py, branch rework-metrics

ci-pinned-boards: curate the board set, esp upload via hil-build-esp, rp2040 fixes

2026-09-06T08:57:00Z

The cmake job builds and uploads every pinned board to membrowse on every run, PRs and pushes alike (--identical when code did not change) - it is the single owner of that upload, as before this branch touched it. Curate the board set itself. cxd56/spresense, da1469x/da1469x_dk_pro, fomu/fomu and ft9xx/mm900evxb are no longer pinned CI boards: none sit on either HIL roster and no other CI board carries their driver, so their size tracking was build-only with no hardware behind it - documented with honest `uncovered` reasons instead of silently dropping coverage. Swap lpc15/lpcxpresso1549 -> lpc11/lpcxpresso11u68 for dcd_lpc_ip3511, lpc17/lpcxpresso1769 -> lpc40/ea4088_quickstart for dcd_lpc17_40+ohci, and samd5x_e5x/same54_xplained -> metro_m4_express; adafruit_fruit_jam now also claims dcd_rp2040. All four changed boards build-verified (device/cdc_msc, cmake), and the rp2040 group's (family, board) sort order restored. Two stale notes the swap left behind are fixed: lpcxpresso11u68 is on neither roster, so its "HIL (htpc)" claim (copy-pasted from the entry it replaced) was false, and ea4088_quickstart is a maintainer bench board noted the same way frdm_kl25z and lpcxpresso1549 were. Rename the board file to .github/ci-pinned-boards.json, with matching --ci-pinned-boards/--ci-pinned-boards-only flags: it pins the board that represents each family in CI, and membrowse history is keyed on each pinned board's / target names, so the set is explicitly held rather than derived. Content and target names are unchanged; the per-entry "family" key is dropped - board names are unique across hw/bsp/*/boards, so it was redundant data that could drift from the tree. build.py's resolve_ci_boards() and the coverage checker derive the family from the board dir, and entries are sorted by board name, the only key left to sort on. espressif builds have no leg in the cmake job (esp-idf is by far the slowest toolchain, and its boards are already enumerated by name for the rig) - hil-build-esp keeps its own upload-membrowse + secrets: inherit instead, and hil-build-esp-identical covers the --identical row for a no-code-change push, the same gap hil-build-esp's own code-changed gate would otherwise leave. build_util.yml's Membrowse Upload step skips any leg whose args carry --build-name, an extra -D or --cflag - a HIL variant build - since it configures with the same -DBOARD= as the plain build and would otherwise fight it for the same upload target name; this protects hil-build-esp's DMA variants specifically. The step also does not scope by $EX_ARGS (kept on the Build step): scoping the upload by the PR's example filter would leave examples outside that filter with no upload attempt at all - a silent history gap rather than the --identical row the design intends. Fork PRs: GHA withholds secrets, so MEMBROWSE_API_KEY reaches membrowse_report.py empty. Master's CMake-expanded bare --api-key fell through to membrowse's GitHub tokenless auth; the wrapper now does the same by omitting --api-key entirely when the env var is empty, rather than exiting - the previous hard-exit was silently hidden behind the workflow's continue-on-error, so every fork PR lost its upload with no visible failure. Retiring ci_skip_boards let CircleCI's plain family sweep reach five rp2040 boards for the first time; two didn't build. pico_sdk fails dual/host_hid_to_device_cdc and dual/host_info_to_device_cdc - a meta-board with no board.h pin defines of its own has no PICO_DEFAULT_PIO_USB_* macros, which a dual-role rp2040 example needs for its PIO-USB host side - so each example gained a skip.txt entry naming the failure. feather_rp2040_max3421 failed host/bare_api on -Wmissing-prototypes for max3421_int_handler in hw/bsp/rp2040/family.c; it is only ever used as a same-file GPIO IRQ callback (no other caller), so it is marked static rather than skip-listing a zero-risk one-line fix. All 8 rp2040 boards now build clean, `-b ` and the bare `rp2040` family sweep alike. drivers_coverage_check.py type-checks `drivers` before iterating: a string previously iterated character-by-character into nonsense "X matches no driver source file" errors, and null raised an unhandled TypeError. Not fixed, flagged for a follow-up: MEMBROWSE_API_KEY reaches membrowse_report.py as a child-process argv (visible in /proc//cmdline on the runner, even though the logged command line is redacted) rather than via the environment. Final-review fixes: the espressif Membrowse Upload docker run only forwarded MEMBROWSE_API_KEY and CI into the container, so membrowse's --github metadata detection (GITHUB_EVENT_NAME/GITHUB_SHA/...) and its fork-PR tokenless auth never saw the actual event - add GITHUB_ACTIONS/GITHUB_EVENT_NAME/GITHUB_SHA/GITHUB_REF_NAME plus a read-only bind mount of GITHUB_EVENT_PATH (containers don't inherit runner env by default; the esp Build step needs none of this, since its `--target all` never builds the *-membrowse/-upload custom targets, so its now-unused `pip install membrowse` is also dropped). check-paths' code filter also gains tools/membrowse_report.py, so a PR touching only that file exercises the real build+upload matrix (ci_select rule 2d) instead of 74 no-op --identical legs. Review fixes: generate_membrowse_sizes()'s all-examples scope (-b with no -e) now averages each file's size across the elfs it appears in instead of summing, matching metrics.py's compute_avg() semantics from the linkermap engine this branch replaced - a file linked by N examples no longer reports ~N times its real size, and the Flash/RAM columns are a real binary's size again; single-example (-e) stays byte-identical (sum, same as before). The Membrowse Upload step (build_util.yml) and hil-build-esp-identical's loop (build.yml) now strip any -e pairs hil_ci_set_matrix.py bakes into a PR-scoped hil-build-esp leg's args before invoking, so a PR-scoped run still uploads every example of the pinned board rather than only the PR-selected subset - closing exactly the silent history gap the step's own comment already warned against. Also: build.py's espressif branch gates the idf.py build path on 'all' being present in build_targets rather than being exactly ['all'], so a combined --target invocation still runs the build instead of silently skipping on a fresh dir; membrowse_compare.py's layout-based bucketing unions the actual flash/ram buckets of a multi-region section's regions instead of assuming any 2+-region split is a flash+ram pair (e.g. RAM_D1/RAM_D2 now stays ram-only); membrowse_report.py's extract_defsyms() dedupes to match its own docstring. Post-validate fixes, folded in. The Membrowse Upload step threads the PR's -e filter through for BOARD ELIGIBILITY only: without it, resolve_ci_boards() ran with examples=None and still resolved the pinned board even when the Build step above had fallen back to a substitute, uploading --identical for a board that run never built. Under --ci-pinned-boards-only tools/build.py now nulls the example filter after board selection, so the upload still touches every example of whichever board got resolved - real rows for what was built, --identical for the rest. same7x earns its pin instead of assuming it: a full family sweep showed five *_freertos examples failing (no FreeRTOSConfig.h wired for same7x - skip.txt'd as family:same7x, naming the reason) and same70_qmtech's board.cmake/mk pointing at a same70n19b_flash.ld that never existed in the tree (the SDK ships none) - the board now carries its own, Q21B's script with the N19B's MEMORY sizes (512K flash / 256K ram per same70n19b.h), like the other boards with a board-local .ld. Both same7x boards now build their full example sets green under the real budget. cxd56 stays OUT of ci_set_matrix's family_list and its BSP reverts to master untouched: the same sweep showed NuttX-header failures across three examples, spresense is unpinned anyway, and a -Wno-error suppression for a family no CI leg builds silences a real diagnostic class for nothing. drivers_coverage_check.py gains two guards: a pinned family that no CI toolchain actually compiles is now a validity error (the dead-pin case the same7x episode demonstrated), and a non-dict entry in "boards" reports one clear error instead of a traceback. The spec and plan docs are updated to the shipped names (membrowse-targets.json / --board-pins / --pins-only never shipped; 71 occurrences), and stale test fixtures naming the deleted tinyusb_metrics target are repointed at examples-membrowse-upload. Validate-loop fixes, folded in. membrowse_compare.py's report_for_elf() now passes the elf's own linker scripts and --defsym values (read from its ninja build graph, exactly as the CI upload path does) to `membrowse report`: with none given membrowse falls back to its default Code/Data regions, which the layout bucketing cannot map, so `.data` lost its flash-side load image and was counted RAM-only (verified on stm32h743eval: region 'Data' without the script, ['FLASH','RAM_D1'] with it). _bucket_from_layout() also unions in the flash side for a BOTH_SECTIONS section whose layout lists only its RAM region - pico-sdk's script never lists `.data` under FLASH, undercounting raspberry_pi_pico's flash by 11.2K. membrowse_onboard.py: `membrowse onboard` checks out and `git clean -fdx`s every historical commit in whatever directory it runs, which would detach HEAD and wipe the deps symlinks in the repo root - so the wrapper runs it in a disposable worktree under cmake-metrics/, and the composed build script relinks deps and reconfigures the (also wiped) build dir before every historical build, not just the first. --build-dirs now includes the example's own directory, so a change to its sources no longer backfills as --identical. Not fixable in the wrapper: onboard applies one commit's linker scripts to every commit in the range (no per-commit override in its CLI).

ci-boards: one curated board list for CI, membrowse and HIL coverage

2026-09-06T08:57:00Z

The pinned-target file becomes .github/ci-boards.json - "pin" reads as GPIO pin in this repo - and with it the flags (--ci-boards, --ci-boards-only), the resolver (resolve_ci_boards()) and the JSON's own key (boards). It is now the only board-curation mechanism: tools/build.py's ci_skip_boards/ci_preferred_boards are gone, with samd2x_l2x and stm32h7 pinned to preserve today's CI picks, and resolve_ci_boards() applies the -e buildability filter so a scoped PR cannot pick a board that builds none of the selected examples and go green having compiled nothing. Curate the set itself: RP2350 boards (raspberry_pi_pico2, adafruit_fruit_jam for hcd_pio_usb) replace feather_rp2040_max3421, add max32666fthr, align the lpc55/nrf pins with rig boards, and drop the nuvoton trio, msp432e4, broadcom_64bit, f1c100s, gd32vf103 and mm32 - dwc2 stays covered, the rest move to uncovered with reasons. The checker moves to tools/drivers_coverage_check.py and reports coverage for both consumers without failing on gaps: membrowse boards (documented uncovered -> INFO, undocumented -> WARNING) and the HIL rosters (INFO), mapping boards to drivers through ci_select's rule-3/4 machinery including its role filter, so an hcd driver only counts host-role rig boards. Validity errors - unknown names, a driver both pinned and uncovered, an hcd claim on a board with no host example - still exit 1. Its chdir now scopes both cwd-relative calls, so the checker works from any directory, and the hook watches the inputs it actually reads.

metrics: replace the linkermap CI pipeline with membrowse

2026-09-06T08:57:00Z

Membrowse becomes TinyUSB's size-analytics system, in CI and locally. .github/membrowse-targets.json names the boards CI builds per family and the drivers each covers; a checker enforced by pre-commit fails when a dcd/hcd driver (plus ehci/ohci) is neither claimed nor documented in `uncovered`, validates board and family names against hw/bsp, and requires a board claiming an hcd driver to build at least one host/dual example. tools/build.py resolves that file into the build matrix and gates uploads on it. Locally, tools/membrowse_compare.py diffs `membrowse report --json --all-symbols` per source file, keyed on object_file (membrowse 1.2.9 truncates source_file to a basename), and metrics_compare_base.py uses it as the default engine, with `--engine linkermap` as the legacy map.json fallback (still required for --combined/--ci). A guard errors instead of writing a silent all-zero table when the filters match nothing. linkermap stops running on every build: the POST_BUILD hook goes, the explicit -linkermap target and a new examples-linkermap aggregate stay, and family_add_linkermap() no-ops when the dep is not fetched. CI loses the whole linkermap pipeline - the code-metrics job, the sticky PR size comment, release metrics assets, per-family artifacts, metrics_pair_compare.py, the membrowse-onboard workflow and CircleCI's dead tinyusb_metrics lane. The cmake job builds the named boards (esp-idf included, membrowse wired into the IDF docker path via family_initialize_project and MEMBROWSE_LD_OVERRIDE) and uploads only those, keeping the --identical path for code-unchanged runs. ci_select follows: the size tooling no longer runs in any CI build, while the board list decides what a family's legs compile and so forces a full matrix. Adds the espressif, same7x, cxd56 and f1c100s families the board list needs, each build-verified. The upload itself moves out of cmake: family_add_membrowse()'s ~90-line inline bash becomes tools/membrowse_report.py, carrying ninja linker-script extraction with recursive INCLUDE resolution, --defsym collection, map-file detection, --identical fallback and upload composition. The API key is read from the environment at run time, passed via argv and redacted in logs - nothing is baked into build.ninja at configure time, which the CMake-expanded $ENV{MEMBROWSE_API_KEY} used to do. tools/membrowse_onboard.py wraps `membrowse onboard` for history backfill, deriving the build script, elf path and the CI-matching target name (/) so a backfill cannot land under a name CI never uploads to; it refuses to start on a dirty worktree, since onboard checks out past commits in place. A membrowse skill documents local reports, size diffs, board-list maintenance, uploads and backfill; the code-size skill records the new default engine and its linkermap fallback. Three follow-up handoffs capture what was deliberately split out: full linkermap removal, membrowse --combined support, and the dead build_filtered plumbing.

tools: share Claude harness with Codex

2026-09-04T15:55:53Z

Add RTT console/capture tooling (tools/rtt.py), rtt skill, and HIL harness support (#3853)

2026-08-28T07:16:02Z

Promote SEGGER RTT from an inline debugging technique to a standalone skill backed by one stdlib-only implementation in tools/rtt.py: a CLI and importable module for console/capture over J-Link (RTTTelnetPort) and OpenOCD (rtt server) probes, with probe selection by serial or VID:PID, control-block address via --elf or --addr, bidirectional console, post-mortem ring dump, and --reset-before-attach for boot-time capture. The HIL harness reads a board's console over RTT when its probe has no VCOM ("logger": "rtt" plus a LOGGER=rtt variant define), covering device_info, pool-check aliveness, and CI wiring. Validated on 22 boards across both backends; 26 unit tests run in pre-commit.

ci_select: fix the membrowse test's env dependence, and stop HIL unit tests taking the rig (#3846)

2026-08-25T03:35:45Z

test_the_upload_board_can_diverge_from_the_built_board called get_family_boards without ci=True, so it pinned the developer's set, not the runner's: the CI skip lists move the one-first pick on three families. It held locally and went red on its first CI run. Pass ci=True, as _prune_buildable already does, and pin the runner's twelve. Rule 2 is a bare test/hil/ prefix, so the harness's own unit tests booked the full 27-board rig for diffs that cannot reach it. Carve test/hil/test/** out to rule 1b, beside test/{fuzz,unit-test}/**; the harness itself is untouched. A test pins that directory's file list, so anything added there that the rig does read fails rather than silently skipping hardware. Rule table updated in the spec and its carbon in the docstring.

ci: an empty selection must build nothing, plus selector follow-ups (#3845)

2026-08-25T02:46:42Z

ci: an empty selection must build nothing, plus selector follow-ups A PR whose build axis legitimately selected nothing rebuilt everything. build.yml reads .build.families twice - as a |-joined regex, and implicitly as "is anything selected" - but tested only -z "$FAMILY_REGEX", which an empty list and a charset-rejected one both satisfy while meaning opposite things. ci_set_matrix had already returned the correct all-empty matrix; the fall-open branch discarded it. #3842 and #3840 each spent 74 cmake legs on it. Branch on the two cases instead, rename FAM_* to FAMILY_*, and cover the block with a test that extracts it from build.yml and executes it - it had no test at all, which is how this shipped through two merges. Follow-ups to the same machinery: glob.escape the repo root at five sites, so a checkout path containing [ or * stops failing closed; drop the ci-full label, read after the matrix was already computed and so never functional; delete 13 mcu:MKL25ZXX / mcu:SAME5X skip tokens matching no board; carry the rule table in the module docstring, guarded against drift; and pin six selection behaviours a mutation pass proved untested. Cut the selector's cost 1.8x (26.0s -> 14.6s) with 0 divergences over 260 paths, and stop scoping the membrowse upload by the PR example filter.

ci_select: address Copilot review - anchor _META_RE, cover rule 12b

2026-08-22T16:10:14Z

Anchor the .github file alternatives. FUNDING.yml, labeler.yml and membrowse_pr_message.j2 sat inside a group whose only `$` belonged to the workflows/ branch, so they matched as prefixes: .github/labeler.yml.bak and .github/FUNDING.yml.old were classified as metadata and would have selected nothing. No such file exists today - the workflows/ alternative was already anchored and ISSUE_TEMPLATE/ is a directory prefix on purpose. Rule 12b had no test of its own: TestNoTrackedFileIsUnclassified only proved src/typec no longer reaches rule 17, not that the answer is right. TestTypecRule pins it - non-full, every selected example under typec/, all four src/typec files answering alike, no rig board, and the set derived from CFG_TUC_ENABLED rather than hardcoded, so it follows a new typec example on its own. Verified all four fail with rule 12b removed.

ci_select: classify the 254 files that were reaching rule 17

2026-08-21T15:17:51Z

Rule 17 (unclassified -> full on both axes) is the fail-open net for paths nobody anticipated, and it must stay that way: a wrong `full` costs runner minutes and is visible in the run, a wrong `empty` costs a merged regression and is invisible. But nothing in the tree should REACH it, and 254 tracked files did. The cost was real. PR #3842 changed a skill, a README and .gitignore; .gitignore matched no rule, so both axes went full and 74 cmake legs span up runners to do checkout + toolchain + get_deps before skipping the build, plus the whole 30-board rig. Three changes, none of which touch rule 17 itself: 1. _META_RE - repo metadata and tooling no Build step reads: .gitignore, .gitattributes, .clang-format, .codespellrc, .pre-commit-config.yaml, .readthedocs.yaml, .PVS-Studio/, .idea/, sonar-project.properties, the packaging manifests, CMakePresets, udev rules, test/{fuzz,unit-test} (their own jobs build those), the non-build .github/ files, and the tools/*.py scripts no build invokes. Deliberately NOT included, and still full: .circleci/**, .github/workflows/build*.yml, .github/actions/**, .github/scripts/**. The line is "does a Build step read this", not "is it source". 2. Rules 15 and 16 now match what they already claimed. Row 15 names examples//CMakeLists.txt and the regex never had it; row 16 says tools/build*.py but anchored tools/build\.py$. Both got the right answer only because rule 17 caught them on the way past. Also names their siblings - family_support.mk, family_rules.mk, src/CMakeLists.txt, src/tinyusb.mk - and .circleci/**, which generates the whole CircleCI matrix and was in no row at all. 3. src/typec/** gets row 12b. It is listed unconditionally by both build systems but its body is `#if CFG_TUC_ENABLED`, which only examples/typec/power_delivery sets - the same shape as the class rule, so the same answer: the examples that enable it (stm32g4 and stm32u5 after the buildability prune), and nothing on the rig, which runs no typec test. It was force-fulling 82 families and all 30 boards. TestNoTrackedFileIsUnclassified walks every tracked file and asserts none reaches rule 17, on both axes - 254 -> 0. Verified it fails when a new unclassified path appears. That turns 17 into what it should be: unreachable for anything in the tree, so it fires only for genuinely new shapes, and the author is told to write the row rather than letting the fall-through pick an answer for them. test_full_paths used sonar-project.properties as its stand-in for "unclassified"; that is now metadata, so the case moved to the new test_repo_metadata_is_not_a_build_input, with test_the_build_machinery_is_still_full pinning the other side of the line.

hil: express a board's always-on defines as a variant, dropping build.args

2026-08-21T07:23:40Z

The roster had two ways to pass a cmake -D to a board's build: `build.args`, applied to every variant, and `variant[].defines`, applied to one. They did the same thing, and only metro_m4_express used the first - for MAX3421_HOST=1, which is what makes it the one rig board that compiles hcd_max3421.c. A board whose define is always on now carries a single variant named after itself, which is exactly the shape `board.get('variant') or [{'name': name, 'flags': ''}]` already synthesises everywhere - so the build dir, the HIL report row and the variant-boundary handling are unchanged. raspberry_pi_pico has used that shape for its flags all along. Removes the BuildCfg type and the parallel code path from all four consumers: hil_test.build_board, hil_pool_check's two builders, hil_ci_set_matrix and ci_select.board_options. Verified: the hil-build matrix entry is byte-identical (`-b metro_m4_express -DMAX3421_HOST=1`), hil_test's build command is unchanged, ci_select still selects the board for a max3421 diff with MAX3421_HOST in its options, and a real build of dual/host_info_to_device_cdc and host/cdc_msc_hid on that board still compiles hcd_max3421.c.