ci: scope the build matrix and the HIL run to what a PR affects

Every PR built all 74 legs (2494 example builds on GHA cmake alone) and flashed
all 30 rig boards, whatever it touched. One classifier now walks the PR diff twice
and answers three questions: which families to build, which examples per family,
and which boards run which tests. Fail-open throughout - anything no rule
classifies, any exception, any unusable output falls back to the full matrix, and
a master push always builds everything.

test/hil/helper/hil_select.py moves to tools/ci_select.py: it is no longer HIL-only,
and tools/ is where the build side can import it. test_hil_select.py follows it as
test_ci_select.py.

Rules (docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md holds the
full table): a port selects the families whose family.cmake references it, and its
role - a dcd change skips host examples and vice versa; a class selects only the
examples whose tusb_config.h enables its CFG_TU[DH]_ macro, following cross-class
includes; an example selects itself; hw/bsp selects its family or board; hw/mcu and
lib select whoever references them. CMake is the reference for all of it - make
follows whatever cmake decides, family.mk is never scanned.

Empty means empty (maintainer ruling): a rule that classifies a path to nothing
selects nothing. Ports no family references, classes no config enables, libs no
example builds and hw/mcu paths that resolve nowhere are all real - nothing
compiles them, so nothing can validate them, and the master-push build is the net.
Structural tests pin each such case with an explicit allowlist, so the day one
stops being empty it fails pre-commit instead of silently narrowing CI.

Per-example builds: build.py grows a repeatable -e, resolved against the targets
CMake actually registered and batched into one `cmake --build --target a b c`.
build_utils mirrors CMake's family_filter (the whole FAMILY_MCUS list, ${...} and
string(TOUPPER ...) resolved) for the cmake side, while the make side keeps
master's algorithm verbatim - the two build systems answer differently and a shared
answer breaks lpc54's make link. hil-build gains this even on a full selection:
1702 example builds become 515.

Transport: the selection travels as a file, never an argv or env var - a mass-sweep
diff selects 261 KB against a 128 KiB exec limit, and E2BIG would fail the step
before its own fallback could run. CircleCI carries the example map inside the
generated config (pipeline parameters cap at 512 chars), swapped into the parameter
defaults by sentinel match, and drops the scoping wholesale if that rewrite fails.
Every PR-derived value written to $GITHUB_ENV/$GITHUB_OUTPUT is character-screened.

Code metrics follow the scoping: metrics.py emits per-example totals, and
metrics_pair_compare compares the (board, example) pairs present on both sides
instead of a scoped run against a full-matrix average.

The selector's own suite gates it in both providers: a selector that exits 0 with
valid-but-wrong JSON is the one failure fail-open cannot catch, so a red suite
means the full matrix.

1 files changed, 175 insertions, 36 deletions

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 8f6014f48..2ee124cb3 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -37,7 +37,10 @@ jobs:
               - 'hw/**'
               - 'test/hil/**'
               - 'tools/build.py'
+              - 'tools/build_utils.py'
+              - 'tools/ci_select.py'
               - 'tools/get_deps.py'
+              - 'tools/metrics.py'
               - '.github/actions/**'
               - '.github/workflows/build.yml'
               - '.github/workflows/build_util.yml'
@@ -48,6 +51,9 @@ jobs:
     outputs:
       json: ${{ steps.set-matrix-json.outputs.matrix }}
       hil_json: ${{ steps.set-matrix-json.outputs.hil_matrix }}
+      example_map: ${{ steps.set-matrix-json.outputs.example_map }}
+      build_filtered: ${{ steps.set-matrix-json.outputs.build_filtered }}
+      build_families_regex: ${{ steps.set-matrix-json.outputs.build_families_regex }}
       # one pair per rig job: hil-tinyusb (tinyusb.json minus esptool boards),
       # hil-tinyusb-esp (esptool boards only), hil-tinyusb (hfp.json)
       hil_args_tinyusb: ${{ steps.hil-select.outputs.args_tinyusb }}
@@ -62,7 +68,7 @@ jobs:
         with:
           fetch-depth: 0
 
-      - name: HIL selection (PR only)
+      - name: CI selection (PR only)
         id: hil-select
         if: github.event_name == 'pull_request'
         env:
@@ -79,55 +85,124 @@ jobs:
           # advisory workflow that nothing here can `needs:`. Test-failing selector =>
           # full matrix, same as a crashing one.
           SELECT_JSON=''
-          if ! python3 test/hil/test/test_hil_select.py; then
-            echo "::warning::hil_select unit suite failed - falling back to the full HIL matrix"
-          elif ! SELECT_JSON=$(python3 test/hil/helper/hil_select.py --base "origin/$BASE_REF" test/hil/tinyusb.json test/hil/hfp.json); then
-            echo "::warning::hil_select failed - falling back to the full HIL matrix"
+          if ! python3 test/hil/test/test_ci_select.py; then
+            echo "::warning::ci_select unit suite failed - falling back to the full HIL matrix"
+          elif ! SELECT_JSON=$(python3 tools/ci_select.py --base "origin/$BASE_REF" test/hil/tinyusb.json test/hil/hfp.json); then
+            echo "::warning::ci_select failed - falling back to the full HIL matrix"
             SELECT_JSON=''
           fi
 
+          # The selection is handed on as a FILE in the workspace, never as a step
+          # output/env var: it is ~KBs normally but a mass-sweep PR reaches hundreds of
+          # KB, and an env var that big makes the consuming exec fail with E2BIG BEFORE
+          # any fallback in it can run. Written here, ahead of its first reader.
+          # No file (non-PR event, or any fallback) = full matrix.
+          rm -f ci_select_out.json
+          if [ -n "$SELECT_JSON" ]; then
+            printf '%s' "$SELECT_JSON" > ci_select_out.json
+          fi
+
           # One args/run pair per rig job, split by flasher: a job whose own subset is
           # empty skips explicitly instead of running a board filter that matches zero
           # boards ("No tests were run." exits 0 and would read as a green HIL run).
           OUT=''
-          if [ -n "$SELECT_JSON" ]; then
-            OUT=$(SELECT_JSON="$SELECT_JSON" python3 -c '
-          import json, os
-          s = json.loads(os.environ["SELECT_JSON"])
+          if [ -s ci_select_out.json ]; then
+            OUT=$(python3 -c '
+          import json, re, sys
+          s = json.load(open("ci_select_out.json"))
+          # the same reading hil_ci_set_matrix.py applies: full false with no usable
+          # boards map is an UNUSABLE selection, not "nothing selected". Both must agree
+          # - one falling open to the whole roster while the other computes run=false
+          # buys a full 37-leg build and still zero hardware coverage.
+          if not s.get("full") and not isinstance(s.get("boards"), dict):
+              sys.exit("selection has full false but no usable boards map")
           tin = s.get("args_flasher", {}).get("tinyusb.json", {})
           legs = (("tinyusb", " ".join(a for f, a in sorted(tin.items()) if f != "esptool" and a)),
                   ("tinyusb_esp", tin.get("esptool", "")),
                   ("hfp", s.get("args", {}).get("hfp.json", "")))
           for key, a in legs:
+              # roster board names reach $GITHUB_OUTPUT as bare NAME=VALUE lines; a
+              # newline in one would inject extra run_* lines and flip which rig jobs run.
+              # ":" and "," are part of the normal shape - a partial filter is
+              # `-bt <board>:<test>,<test>` (ci_select._board_args)
+              if not re.fullmatch(r"[-A-Za-z0-9_/ .=+:,]*", a):
+                  sys.exit("unexpected characters in the " + key + " board filter")
               print("args_" + key + "=" + a)
               print("run_" + key + "=" + ("true" if (s.get("full") or a) else "false"))
           ') || OUT=''
             if [ -z "$OUT" ]; then
-              echo "::warning::hil_select output unusable - falling back to the full HIL matrix"
-              SELECT_JSON=''
+              echo "::warning::ci_select output unusable - falling back to the full HIL matrix"
+              # the same unusable selection must not stay behind for the build axis
+              rm -f ci_select_out.json
             fi
           fi
           if [ -z "$OUT" ]; then
             OUT=$(for k in tinyusb tinyusb_esp hfp; do printf 'args_%s=\nrun_%s=true\n' "$k" "$k"; done)
           fi
           echo "$OUT"
-          { echo "select=$SELECT_JSON"; echo "$OUT"; } >> $GITHUB_OUTPUT
+          echo "$OUT" >> $GITHUB_OUTPUT
 
       - name: Generate matrix json
         id: set-matrix-json
-        env:
-          SELECT: ${{ steps.hil-select.outputs.select }}
         run: |
-          # build matrix
-          MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py)
+          # Build matrix, scoped by the PR selection when one exists. Best-effort:
+          # ci_set_matrix falls back to the full matrix itself on unusable JSON,
+          # and a missing file (non-PR event, selector fallback) means no flags.
+          SELECT_FILE=ci_select_out.json
+          [ -s "$SELECT_FILE" ] || SELECT_FILE=''
+          BUILD_SELECT_FILE="$SELECT_FILE"
+          MATRIX_JSON=''
+          if [ -n "$SELECT_FILE" ]; then
+            # ci_set_matrix falls open on a selection it cannot use with rc 0 - it prints
+            # the full matrix and says UNSCOPED on stderr. The build extras below must
+            # not stay scoped when it did, or a nominally full build compiles 1 of 44
+            # examples per family and code-metrics compares that partial run against a
+            # full baseline. Only the BUILD axis is dropped: build.families being
+            # unusable says nothing about the boards map the HIL matrix reads.
+            MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py --select-file "$SELECT_FILE" 2>ci_set_matrix.err) || MATRIX_JSON=''
+            cat ci_set_matrix.err >&2
+            if [ -z "$MATRIX_JSON" ] || grep -q 'ci_set_matrix: UNSCOPED' ci_set_matrix.err; then
+              BUILD_SELECT_FILE=''
+            fi
+          fi
+          [ -z "$MATRIX_JSON" ] && MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py)
           echo "matrix=$MATRIX_JSON"
           echo "matrix=$MATRIX_JSON" >> $GITHUB_OUTPUT
 
+          # Build-axis extras: the per-family example map rides as a side channel
+          # (a value inside matrix entries would break CircleCI's family parameter
+          # and multiply GHA matrix legs). These stay step outputs - they are small
+          # derived values, unlike the selection they are read from. NOTE jq's //
+          # treats false like null, so .build.full is compared explicitly.
+          EXAMPLE_MAP='{}'
+          BUILD_FILTERED='false'
+          FAM_REGEX=''
+          if [ -n "$BUILD_SELECT_FILE" ]; then
+            EXAMPLE_MAP=$(jq -c '.build.family_examples // {}' "$BUILD_SELECT_FILE") || EXAMPLE_MAP='{}'
+            BUILD_FILTERED=$(jq -r 'if (.build? | type) == "object" and .build.full == false then "true" else "false" end' "$BUILD_SELECT_FILE") || BUILD_FILTERED='false'
+            if [ "$BUILD_FILTERED" = "true" ]; then
+              FAM_REGEX=$(jq -r '.build.families | join("|")' "$BUILD_SELECT_FILE") || FAM_REGEX=''
+              # family names come from hw/bsp dir names, which rule 6 reads straight out
+              # of the PR's diff path - and this is interpolated raw into a
+              # `name_is_regexp` artifact pattern, so a regex metacharacter there would
+              # silently match another family's baseline
+              case "$FAM_REGEX" in
+                *[!-A-Za-z0-9_\|]*)
+                  echo "::warning::unexpected characters in the family list - unscoped metrics"
+                  FAM_REGEX='' ;;
+              esac
+              [ -z "$FAM_REGEX" ] && BUILD_FILTERED='false'
+            fi
+          fi
+          echo "example_map=$EXAMPLE_MAP" >> $GITHUB_OUTPUT
+          echo "build_filtered=$BUILD_FILTERED" >> $GITHUB_OUTPUT
+          echo "build_families_regex=$FAM_REGEX" >> $GITHUB_OUTPUT
+
           # HIL matrix (merged from tinyusb + hifiphile configs), scoped on PRs.
           # Scoping is best-effort too: fall back to the unscoped (full) matrix.
           HIL_MATRIX_JSON=''
-          if [ -n "$SELECT" ]; then
-            HIL_MATRIX_JSON=$(python .github/scripts/hil_ci_set_matrix.py --select "$SELECT" test/hil/tinyusb.json test/hil/hfp.json) || HIL_MATRIX_JSON=''
+          if [ -n "$SELECT_FILE" ]; then
+            HIL_MATRIX_JSON=$(python .github/scripts/hil_ci_set_matrix.py --select-file "$SELECT_FILE" test/hil/tinyusb.json test/hil/hfp.json) || HIL_MATRIX_JSON=''
             if [ -z "$HIL_MATRIX_JSON" ]; then
               echo "::warning::scoped HIL matrix failed - falling back to the full HIL matrix"
             fi
@@ -162,6 +237,7 @@ jobs:
       toolchain: ${{ matrix.toolchain }}
       build-args: ${{ toJSON(fromJSON(needs.set-matrix.outputs.json)[matrix.toolchain]) }}
       build-options: '--one-first'
+      example-map: ${{ needs.set-matrix.outputs.example_map }}
       upload-metrics: true
       upload-artifacts: false
       upload-membrowse: true
@@ -169,8 +245,17 @@ jobs:
     secrets: inherit
 
   code-metrics:
-    needs: [ check-paths, cmake ]
-    if: needs.check-paths.outputs.code_changed == 'true'
+    needs: [ check-paths, cmake, set-matrix ]
+    # A scoped selection can empty every cmake toolchain (a test/hil-only PR). This
+    # job must still run then: skipping it leaves the sticky comment showing the
+    # PREVIOUS push's size table as if it were current. set-matrix must have
+    # SUCCEEDED though: !cancelled() alone let a failed set-matrix through, and this
+    # job would then overwrite the sticky comment with a wrong "built no families"
+    # diagnosis while reporting itself green.
+    if: |
+      !cancelled() && needs.check-paths.outputs.code_changed == 'true' &&
+      needs.set-matrix.result == 'success' &&
+      (needs.cmake.result == 'success' || needs.cmake.result == 'skipped')
     runs-on: ubuntu-latest
     permissions:
       pull-requests: write
@@ -187,8 +272,21 @@ jobs:
           pattern: metrics-*
           path: cmake-build
           merge-multiple: true
+          # download-artifact does not fail on a pattern that matches nothing, so a
+          # scoped PR that built no family simply lands here with an empty dir
+
+      - name: Detect empty metrics set
+        run: |
+          # No metrics at all => nothing to aggregate or compare. Write the marker the
+          # sticky comment will carry, so the size section says "skipped" for THIS push
+          # instead of silently keeping the previous push's table.
+          if ! ls cmake-build/*/metrics.json >/dev/null 2>&1; then
+            echo "_Code-size comparison skipped: PR selection built no families on this push._" > metrics_compare.md
+            echo "NO_METRICS=true" >> $GITHUB_ENV
+          fi
 
       - name: Aggregate Code Metrics
+        if: env.NO_METRICS != 'true'
         run: |
           python tools/get_deps.py
           python tools/metrics.py combine -j -m -f tinyusb/src cmake-build/*/metrics.json
@@ -201,7 +299,7 @@ jobs:
           path: metrics.json
 
       - name: Download Base Branch Metrics
-        if: github.event_name == 'pull_request' || github.event_name == 'workflow_dispatch'
+        if: env.NO_METRICS != 'true' && (github.event_name == 'pull_request' || github.event_name == 'workflow_dispatch') && needs.set-matrix.outputs.build_filtered != 'true'
         uses: dawidd6/action-download-artifact@v11
         with:
           workflow: build.yml
@@ -211,6 +309,29 @@ jobs:
           path: base-metrics
         continue-on-error: true
 
+      - name: Download base per-family metrics (scoped PR)
+        if: env.NO_METRICS != 'true' && github.event_name == 'pull_request' && needs.set-matrix.outputs.build_filtered == 'true'
+        uses: dawidd6/action-download-artifact@v11
+        with:
+          workflow: build.yml
+          workflow_conclusion: ''
+          search_artifacts: true      # a docs-only master push uploads no per-family artifacts
+          branch: ${{ github.base_ref }}
+          name: ^metrics-(${{ needs.set-matrix.outputs.build_families_regex }})$
+          name_is_regexp: true
+          path: base-family-metrics
+        continue-on-error: true
+
+      - name: Compare with Base Branch (scoped)
+        if: env.NO_METRICS != 'true' && github.event_name == 'pull_request' && needs.set-matrix.outputs.build_filtered == 'true'
+        run: |
+          # never fall back to the averaged metrics-tinyusb here: a scoped PR vs the
+          # 64-family/46-example average is exactly the mismatch this path prevents
+          python .github/scripts/metrics_pair_compare.py \
+            --base-dir base-family-metrics --new-dir cmake-build --out metrics_compare || \
+            echo "_Code-size comparison failed on the scoped path - see the code-metrics job log._" > metrics_compare.md
+          cat metrics_compare.md
+
       - name: Download Previous Release Asset
         if: github.event_name == 'release'
         env:
@@ -224,7 +345,7 @@ jobs:
           gh release download $PREV_TAG -p metrics.json -D base-metrics || echo "No metrics.json found in $PREV_TAG release"
 
       - name: Compare with Base Branch
-        if: github.event_name != 'push'
+        if: env.NO_METRICS != 'true' && github.event_name != 'push' && needs.set-matrix.outputs.build_filtered != 'true'
         run: |
           if [ -f base-metrics/metrics.json ]; then
             python tools/metrics.py compare -m -f tinyusb/src base-metrics/metrics.json metrics.json
@@ -252,6 +373,9 @@ jobs:
           path: |
             metrics_compare.md
             metrics.json
+          # metrics.json is absent when the selection built no family; the marker
+          # in metrics_compare.md is still what the sticky comment needs
+          if-no-files-found: ignore
 
       - name: Post Code Metrics as PR Comment
         if: (github.event_name == 'workflow_dispatch') || (github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == false)
@@ -627,32 +751,32 @@ jobs:
         run: |
           # Best-effort: this job is deliberately decoupled from set-matrix so unrelated
           # failures cannot kill hfp coverage - a selector failure here must likewise
-          # fall back to the full hfp matrix (no hil_select.json, no SEL_* vars), never
+          # fall back to the full hfp matrix (no ci_select.json, no SEL_* vars), never
           # fail the job.
-          if ! python3 test/hil/test/test_hil_select.py; then
-            echo "::warning::hil_select unit suite failed - running the full hfp matrix"
-            rm -f hil_select.json
+          if ! python3 test/hil/test/test_ci_select.py; then
+            echo "::warning::ci_select unit suite failed - running the full hfp matrix"
+            rm -f ci_select.json
             exit 0
           fi
-          if ! python3 test/hil/helper/hil_select.py --base "origin/$BASE_REF" test/hil/hfp.json > hil_select.json; then
-            echo "::warning::hil_select failed - running the full hfp matrix"
-            rm -f hil_select.json
+          if ! python3 tools/ci_select.py --base "origin/$BASE_REF" test/hil/hfp.json > ci_select.json; then
+            echo "::warning::ci_select failed - running the full hfp matrix"
+            rm -f ci_select.json
             exit 0
           fi
-          # hil_select.json is passed to hil_ci_set_matrix.py --select below to scope the
+          # ci_select.json is passed to hil_ci_set_matrix.py --select below to scope the
           # build; it already honours full=true by ignoring the board list.
           # The hil_test.py args go to a file, never to $GITHUB_ENV: they are derived
           # from roster board names, which a PR can edit. Only SEL_RUN (a literal
           # true/false computed here, needed by the step-level `if:`) goes to the env.
           if ! SEL_RUN=$(python3 -c '
           import json
-          s = json.load(open("hil_select.json"))
+          s = json.load(open("ci_select.json"))
           a = s["args"]["hfp.json"]
           open("hil_sel_args.txt", "w").write(a)
           print("true" if (s["full"] or a) else "false")
           '); then
-            echo "::warning::hil_select output unusable - running the full hfp matrix"
-            rm -f hil_select.json hil_sel_args.txt
+            echo "::warning::ci_select output unusable - running the full hfp matrix"
+            rm -f ci_select.json hil_sel_args.txt
             exit 0
           fi
           echo "SEL_RUN=$SEL_RUN"
@@ -661,9 +785,17 @@ jobs:
       - name: Get build boards
         if: env.SEL_RUN != 'false'
         run: |
-          if [ -f hil_select.json ]; then
-            MATRIX_JSON=$(python .github/scripts/hil_ci_set_matrix.py --select "$(cat hil_select.json)" test/hil/hfp.json)
-          else
+          # --select-file, never --select "$(cat ...)": a whole selection as one argv
+          # can exceed MAX_ARG_STRLEN on a big diff, and this job's design is to fall
+          # back to the full hfp matrix on any selector trouble, not to fail the step.
+          MATRIX_JSON=''
+          if [ -f ci_select.json ]; then
+            MATRIX_JSON=$(python .github/scripts/hil_ci_set_matrix.py --select-file ci_select.json test/hil/hfp.json) || MATRIX_JSON=''
+            if [ -z "$MATRIX_JSON" ]; then
+              echo "::warning::scoped hfp matrix failed - building the full hfp matrix"
+            fi
+          fi
+          if [ -z "$MATRIX_JSON" ]; then
             MATRIX_JSON=$(python .github/scripts/hil_ci_set_matrix.py test/hil/hfp.json)
           fi
           # Each variant carries its own --build-name/--cflag, which are global to a
@@ -672,6 +804,13 @@ jobs:
           echo "$MATRIX_JSON" | jq -r '.["arm-gcc"][]' > hil_build_entries.txt
           cat hil_build_entries.txt
           BUILD_ARGS=$(echo "$MATRIX_JSON" | jq -r '.["arm-gcc"] | join(" ")')
+          # board and example names are roster data a PR can edit, and jq -r un-escapes
+          # them: a newline here writes extra NAME=VALUE lines into GITHUB_ENV for every
+          # later step of a job that holds the IAR token. Refuse rather than guess.
+          case "$BUILD_ARGS" in
+            *[!-A-Za-z0-9_/\ .=+]*)
+              echo "::error::unexpected characters in the hfp build args"; exit 1 ;;
+          esac
           echo "BUILD_ARGS=$BUILD_ARGS"
           echo "BUILD_ARGS=$BUILD_ARGS" >> $GITHUB_ENV