From c514a6ee91a53927f55ba35e0a2933eddb5cefeb Mon Sep 17 00:00:00 2001 From: hathach Date: Thu, 20 Aug 2026 16:47:23 +0700 Subject: docs: spec, plan and outcome of the .claude instruction-surface audit Nothing checked the agents, workflows and skills against the code they describe, and the surface had drifted into stating incompatible rules. This records the protocol that found the defects and what it cost. Method: parallel subagents extract every falsifiable claim into JSONL ledgers; a validator re-reads each cited line and rejects any ledger whose quoted text is not there, so an extractor that paraphrases or hallucinates fails a script rather than reaching the verification queue - 1,387 claims, zero such failures. Verification runs only in the main session, and the same gate pointed at `citation` then checks the verifier's own work. Hard-earned evidence is source of truth: code is authoritative about code, experience about hardware, so claims get a fourth verdict, EARNED, and "no backing found" is never grounds for deletion. All 1,387 claims carry a verdict; the behavior sweep deliberately never emits CONFIRMED from a token match, because finding a claim's vocabulary proves presence, not truth. Every real defect came from cross-document comparison - none from any mechanical pass. A path-existence lint was built, measured (11 flags on the audited tree, all false positives, and the target defect is lexically identical to correct text elsewhere), and rejected; recorded so nobody rebuilds it. --- .../specs/2026-08-18-claude-doc-audit-design.md | 135 +++++++++++++++++++++ 1 file changed, 135 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-18-claude-doc-audit-design.md (limited to 'docs/superpowers/specs') diff --git a/docs/superpowers/specs/2026-08-18-claude-doc-audit-design.md b/docs/superpowers/specs/2026-08-18-claude-doc-audit-design.md new file mode 100644 index 000000000..5b150dc7b --- /dev/null +++ b/docs/superpowers/specs/2026-08-18-claude-doc-audit-design.md @@ -0,0 +1,135 @@ +# Audit of the `.claude/` instruction surface — design + +**Date:** 2026-08-18 +**Branch:** `claude/hil-doc-audit` + +## Why + +`hil-operator.md` told an operator two incompatible things at once: one rule forbade +pre-holding a board lock because `hil_test.py` self-locks, while a rule added in the same +revision made the lock the thing that keeps concurrent operators off each other's hardware — +so an operator following the second would take a hold that made its own run fail fast against +it. Both statements were fixed before this branch was folded, so neither survives in history; +what survives is the lesson that nothing checks these files against the code they describe. + +That is not an isolated slip. A scan of the 36 repo paths cited across `.claude/` flags 8 +that do not resolve. Five are legitimate — placeholders (`docs/changelog/X.Y.Z.md`, +`src/portable/x/dcd_x.c`, a `test_*.py` glob), a generated file +(`examples/cmake-build-pvs/compile_commands.json`), and a per-host gitignored config +(`test/hil/local.json`, whose absence the skill already handles). Three are drift: +`usbtest/SKILL.md:24,50` cites `src/usb_descriptors.h` and `src/tusb_config.h`, which are +example-relative but read as repo paths, and `:101` cites `tools/usb/testusb.c`, a Linux +kernel path presented like a repo file. + +Cross-references are in better shape: every `agentType` in a workflow resolves to an agent +in `.claude/agents/`, every `.claude/skills/` referenced by an agent or workflow +exists, and the workflow scripts call only harness functions that exist. The drift is in +**prose claims about behavior** — the class that made `hil-validate` parallelize at the +wrong layer, on top of a `hil_test.py` that already schedules boards across host +controllers under per-controller permits (`hil_lock.py:7,133-134`; `hil_test.py:2249,2419`). + +## Scope + +**In:** `.claude/agents/*.md` (7), `.claude/workflows/*.js` + `check.sh` (7), +`.claude/skills/*/SKILL.md` (16) and their 8 helper scripts, and the repo `CLAUDE.md`. +~4,700 lines (2,874 of prose, the rest helper scripts and `etm-trace/boards.md`). + +**Out:** `docs/superpowers/**` (historical records — correcting them rewrites history +rather than fixing what a future session executes), `.claude/settings*.json` and hooks, the +memory index, and any behavior change to the scripts themselves. + +## Claim taxonomy + +Only falsifiable classes get a verdict. Guidance ("bias toward caution") is checked solely +for contradiction with the classes below. + +| Class | Settled by | Example | +|---|---|---| +| Path | `ls`/`find`, with the base dir made explicit | `src/tusb_config.h` — example-relative, reads as repo-relative | +| Interface | argparse/grep in the named source | `-b` is `action='append'` (`hil_test.py:2249`) | +| Behavior | reading the implementing code, cited `file:line` | "permits are in-process semaphores" (`hil_lock.py:7`) | +| Number | the constant's definition | `FLASH_PARALLEL=4` (`hil_lock.py:133`) | +| Rig state | read-only `ssh ci.lan` probe | bus map, probe uids, sudoers entries, installed tools | +| Cross-doc | diffing the same rule's two statements | `hil-operator.md:18` vs `:37` | + +### Verdicts + +- **CONFIRMED** — current source says so. Cite `file:line`. Leave alone. +- **REFUTED** — current source says otherwise. Cite, correct the doc. +- **EARNED** — no source in scope settles it, and it is hard-earned rig knowledge. Stays in + the docs untouched; see the rule below. +- **UNVERIFIABLE** — no source in scope settles it and it is not earned knowledge either + (a placeholder, a generated file, a claim about something outside the repo). + +### Hard-earned evidence is source of truth + +A claim with no code backing is **not** a cut candidate when it is earned rig knowledge: +an observed hardware quirk, a failure mode paid for in rig downtime, a workaround whose +rationale lives only in the incident that produced it. Code is authoritative about code; +experience is authoritative about hardware, and the hardware does not document itself. + +Consequences: + +- Only a claim the **current source actively refutes** gets corrected. "I could not find + backing" is never grounds for deletion. +- Rig-state claims that have gone stale (a bus map, a probe uid) are **re-derived and + updated**, or converted into a derivation recipe ("buses renumber every boot — re-derive + with X"), never dropped. +- Where earned knowledge and current code disagree, that is a **finding to report**, not an + edit to make: one of them is a bug, and deciding which is out of this audit's scope. + +## Passes + +1. **Extraction (fan-out, 9 agents, no verdicts).** One agent per cluster, each writing a + ledger to the scratchpad and returning only a count and the ledger path. Per claim: + `file:line`, verbatim claim, class, what source would settle it, and a flag for + suspected hard-earned evidence. Agents return no judgments, so nothing arrives as a + verdict that would have to be unwound. +2. **Verification (mine).** Every claim checked against source myself: scripted checks for + paths/interfaces/numbers, code reading for behavior, read-only `ssh ci.lan` for rig + state (`ls`, `--help`, `which`, `lspci`, `lsusb`, `hil_lock.py status`, `sudo -l`, + `uname -r` — no locks, no flashing, no `uhubctl`, no recovery). Nothing acted on is + taken on an extractor's word. +3. **Cross-doc consistency (mine).** Build a rule inventory — board locks, timeouts, + output contracts, retry policy, config selection, forcing — and diff every place each + rule is stated. No per-file agent can do this pass; it is where the `hil-operator` + failure lived. +4. **Edits.** Delete only what is refuted by source, restates the command it precedes, or + duplicates a rule that has a canonical home elsewhere (keep one, reference it). Keep + every claim source confirms that changes behavior, every hard-earned observation, and + the "why" behind non-obvious rules. Structure stays as is. +5. **Gate.** Re-run the path and interface scans; `check.sh` on every workflow; `bash -n` + and `py_compile` on all 8 helper scripts; the four `test/hil` suites; + `pre-commit run --all-files`. + +## Extraction clusters + +| # | Cluster | Lines | +|---|---|---| +| 1 | `.claude/agents/*.md` (7 files) | 313 | +| 2 | `.claude/workflows/*.js` + `check.sh` | 659 | +| 3 | `hil`, `hil-pool-check` | 223 | +| 4 | `usb-kernel-recover`, `usb-kernel-debug` + 2 scripts | 253 + scripts | +| 5 | `target-debug`, `esp-target-debug` | 496 | +| 6 | `usbtest`, `usbmon`, `usb-sniffer` + `usbcap.sh` | 382 + script | +| 7 | `etm-trace` + `boards.md` + 2 scripts | 203 + files | +| 8 | `build-doc`, `code-size`, `pvs`, `make-release`, `read-doc`, `pre-pr` + 2 scripts | 345 + scripts | +| 9 | `CLAUDE.md` | 139 | + +## Deliverables + +Commits split by surface (agents / workflows / skills / CLAUDE.md) so review stays +tractable, on `claude/claude-doc-audit`. A findings report covering every REFUTED claim +with its citation, and every earned-knowledge-vs-code disagreement found in pass 2. + +A refuted claim whose *code* is the wrong half does not get a silent code edit: it becomes +a handoff doc under `docs/superpowers/followup/`, per the repo's deferred-work rule. + +## Success criteria + +- Every falsifiable claim in scope carries a verdict with a citation. +- No claim that current source refutes survives in the tree. +- No hard-earned observation is deleted; stale rig state is re-derived or turned into a + derivation recipe. +- No rule is stated in two places with two different meanings. +- The gate in pass 5 passes. -- cgit v1.3.1 From f96ddbaa1e11a98f8076df48ba73026191c28399 Mon Sep 17 00:00:00 2001 From: hathach Date: Fri, 21 Aug 2026 11:09:03 +0700 Subject: docs: record the CI selection design and its plan The binding rule table (17 rows x 3 answer columns), the measured effect per PR shape, and the reasoning behind the parts that look surprising: why empty means empty, why hw/mcu and lib are rules rather than full-matrix paths, why get_deps.py is diffed as data, and which build system is the reference. The plan is the task-by-task record of how it was built, kept as the origin trail. --- .../plans/2026-08-19-ci-build-family-filter.md | 1804 ++++++++++++++++++++ .../2026-08-19-ci-build-family-filter-design.md | 501 ++++++ 2 files changed, 2305 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-19-ci-build-family-filter.md create mode 100644 docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md (limited to 'docs/superpowers/specs') diff --git a/docs/superpowers/plans/2026-08-19-ci-build-family-filter.md b/docs/superpowers/plans/2026-08-19-ci-build-family-filter.md new file mode 100644 index 000000000..0d8b9cfa4 --- /dev/null +++ b/docs/superpowers/plans/2026-08-19-ci-build-family-filter.md @@ -0,0 +1,1804 @@ +# PR-Scoped CI Selection Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Promote `test/hil/helper/hil_select.py` to a repo-wide `tools/ci_select.py` whose one classification of a PR diff narrows three CI axes — build families, per-family example targets, and per-board HIL examples — wired into both GitHub Actions and CircleCI. + +**Architecture:** The selector gains an independent build classifier beside the untouched HIL one (17-rule table in the spec). `ci_set_matrix.py` filters the family matrix from the selector JSON; the per-family example map travels as a side channel (GHA job output / CircleCI pipeline parameter), resolved to `-e` flags per build job by a new `tools/build.py --example` filter. `hil_ci_set_matrix.py` appends `-e` per rig board. Code metrics gain per-example artifacts and a (family, example)-intersection compare. + +**Tech Stack:** Python 3 stdlib (selector must run on bare CI runners), GitHub Actions YAML, CircleCI dynamic config (continuation orb), jq, CMake/Ninja. + +**Spec:** `docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md` — read it first; every rule number below refers to its rule table. + +## Global Constraints + +- Commit messages: imperative mood, **no** `Co-Authored-By:` or `Claude-Session:` trailers (hathach is sole author — this overrides harness defaults). +- Never stage or touch `.idea/`. Always `git add` explicit paths, never `-A`. +- Bare-runner Python modules (`tools/ci_select.py`, `tools/build.py`, `tools/build_utils.py`, everything under `test/hil/helper/`) stay stdlib-only at module level — `test_hil_util.BottomLayer` enforces this; extend its lists, never work around them. +- `ci_select.py` stdout is machine-read JSON; every diagnostic goes to stderr. +- The family reference scan is **CMake-only** (`family.cmake` + espressif component `CMakeLists.txt`, never `family.mk`): CMake is the first-class build system, Make follows it. +- Fail-open everywhere: a selector/matrix-script failure must yield the full matrix, never a red job or a silently-empty one. +- Python style: match the existing modules (4-space indent in tools/ and test/hil/, terse targeted comments explaining *why*). +- YAML: 2-space indent, match surrounding style in `.github/workflows/` and `.circleci/`. +- Run suites from the repo root. Selector suite: `python3 test/hil/test/test_ci_select.py` (after Task 1). Full HIL-side suite: `python3 -m unittest discover -s test/hil/test`. + +--- + +### Task 1: Move the selector to `tools/ci_select.py` (mechanical, no behavior change) + +**Files:** +- Move: `test/hil/helper/hil_select.py` → `tools/ci_select.py` (git mv) +- Move: `test/hil/test/test_hil_select.py` → `test/hil/test/test_ci_select.py` (git mv) +- Modify: `test/hil/test/test_hil_util.py` (BottomLayer lists), `test/hil/hil_ci.sh` (scp list), `.pre-commit-config.yaml` (both hooks), `.github/workflows/build.yml` (4 path refs), `.claude/skills/pre-pr/SKILL.md`, `test/hil/helper/hil_util.py:21` (comment), `test/hil/hil_flash.py:297` (comment) + +**Interfaces:** +- Produces: module `tools/ci_select.py` importable as `ci_select` with `tools/` on `sys.path`; module attribute `_REPO_ROOT` (absolute repo root); CLI `python3 tools/ci_select.py --base REF|--diff-file F CONFIG.json...` — output JSON byte-compatible with today's `hil_select.py`. +- Consumes: `test/hil/helper/hil_util.py` rosters (unchanged). + +- [ ] **Step 1: git mv both files** + +```bash +git mv test/hil/helper/hil_select.py tools/ci_select.py +git mv test/hil/test/test_hil_select.py test/hil/test/test_ci_select.py +``` + +- [ ] **Step 2: Fix `tools/ci_select.py` imports and repo root** + +Replace the current path setup (line 24, `sys.path.insert(0, os.path.dirname(os.path.dirname(...)))` and its comment) with: + +```python +# tools/ -> repo root is ONE level up. Guarded by TestModuleMove.test_repo_root_guard: +# a wrong parent count here silently re-points every repo-relative glob (it happened +# at the helper/ move). +_REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +sys.path.insert(0, os.path.join(_REPO_ROOT, 'test', 'hil')) # for `from helper...` +from helper.hil_util import device_tests, dual_tests, host_test +``` + +In `main()`, replace the 4-level `repo_root` derivation (lines 503-505) with `repo_root = _REPO_ROOT`. Change the stderr prefix at line 519 from `hil_select:` to `ci_select:`. Update the module docstring: it now lives in `tools/`, serves HIL and (from Task 3) build selection; keep the fail-open sentence and the spec pointer, adding this spec's path. + +- [ ] **Step 3: Fix `test/hil/test/test_ci_select.py` imports** + +Replace the header import block (`from helper import hil_select`) so `REPO` is computed first, then: + +```python +REPO = os.path.dirname(os.path.dirname(os.path.dirname( + os.path.dirname(os.path.abspath(__file__))))) +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) # test/hil, for hil_flash/helper +sys.path.insert(0, os.path.join(REPO, 'tools')) +import hil_flash +import ci_select +from helper.hil_util import device_tests, dual_tests +``` + +Then `sed -i 's/\bhil_select\b/ci_select/g' test/hil/test/test_ci_select.py` and fix the header comment (file names, run command). Add the guard test: + +```python +class TestModuleMove(unittest.TestCase): + def test_repo_root_guard(self): + # __file__-derived root: moving the module without re-deriving the parent + # count re-points every scan at the wrong tree (it happened once already) + self.assertTrue(os.path.isdir(os.path.join(ci_select._REPO_ROOT, 'src'))) + self.assertTrue(os.path.isdir(os.path.join(ci_select._REPO_ROOT, 'hw', 'bsp'))) + self.assertEqual(os.path.realpath(ci_select._REPO_ROOT), os.path.realpath(REPO)) +``` + +- [ ] **Step 4: Update every reference** + +- `test/hil/test/test_hil_util.py` BottomLayer: in `test_bare_runner_modules_stay_stdlib_only`, replace `'hil_select'` with `'ci_select'` in the `local` set and replace `'helper/hil_select'` with `'../../tools/ci_select'` in the module-path tuple (the loop builds `hil_dir / f'{mod}.py'`, so a relative path out of test/hil works). Update the docstring sentence naming hil_select. +- `test/hil/hil_ci.sh`: delete the `"$ROOT_DIR/test/hil/helper/hil_select.py" \` scp line (nothing on the rig imports it). +- `.pre-commit-config.yaml`: rename hook `hil-select-test` → `ci-select-test`; `entry: python3 test/hil/test/test_ci_select.py`; `files: ^(hw/bsp/|src/|examples/|tools/ci_select\.py$)`. In the `hil-test` hook comment, s/test_hil_select/test_ci_select/. +- `.github/workflows/build.yml`: four call sites — lines ~82/84 (set-matrix) and ~632/637 (hil-hfp-iar): `test/hil/test/test_hil_select.py` → `test/hil/test/test_ci_select.py`, `test/hil/helper/hil_select.py` → `tools/ci_select.py`; s/hil_select/ci_select/ in the adjacent `::warning::` strings and comments (keep `hil_select.json` file names as `ci_select.json` for consistency — update both writers and both readers in the hfp-iar job). +- `.claude/skills/pre-pr/SKILL.md`: `python3 test/hil/helper/hil_select.py` → `python3 tools/ci_select.py`. +- Comments only: `test/hil/helper/hil_util.py:21` (hil_select → ci_select), `test/hil/hil_flash.py:297` (test_hil_select → test_ci_select). + +- [ ] **Step 5: Verify** + +```bash +python3 test/hil/test/test_ci_select.py # all pass +python3 -m unittest discover -s test/hil/test # all pass (~55 s) +python3 tools/ci_select.py --diff-file /dev/null test/hil/tinyusb.json | python3 -m json.tool >/dev/null +grep -rn "hil_select" --include='*.py' --include='*.yml' --include='*.yaml' --include='*.sh' --include='*.md' . | grep -v docs/superpowers | grep -v '\.worktrees' +``` + +Expected: suites green; last grep returns nothing (historical spec docs are the only allowed hits). + +- [ ] **Step 6: Commit** + +```bash +git add tools/ci_select.py test/hil/test/test_ci_select.py test/hil/test/test_hil_util.py \ + test/hil/hil_ci.sh .pre-commit-config.yaml .github/workflows/build.yml \ + .claude/skills/pre-pr/SKILL.md test/hil/helper/hil_util.py test/hil/hil_flash.py +git commit -m "tools: promote hil_select.py to tools/ci_select.py" +``` + +--- + +### Task 2: Generalize the family scan and re-rule `hw/mcu/**` (HIL side) + +**Files:** +- Modify: `tools/ci_select.py` (`port_families` → `path_families` + `mcu_families`, `_FULL_RE`, `_classify_one`) +- Test: `test/hil/test/test_ci_select.py` + +**Interfaces:** +- Produces: `path_families(rel_dir: str, repo_root: str) -> set[str]` — families whose `family.cmake`/espressif component CMakeLists reference `rel_dir` at a directory boundary; `mcu_families(path: str, repo_root: str) -> set[str]` — longest-resolving-prefix lookup for a changed `hw/mcu/...` path; `port_families(port_dir, repo_root)` kept as a thin wrapper (existing callers/tests unchanged). +- HIL JSON change: `hw/mcu/**` no longer forces `full: true`; it selects the resolved families' boards, all their tests (spec rule 7). + +- [ ] **Step 1: Write the failing tests** (append to `test_ci_select.py`) + +```python +class TestPathFamilies(unittest.TestCase): + def test_port_wrapper_unchanged(self): + self.assertEqual(ci_select.port_families('raspberrypi/rp2040', REPO), {'rp2040'}) + self.assertIn('stm32f4', ci_select.port_families('synopsys/dwc2', REPO)) + + def test_boundary_without_trailing_slash(self): + # hw/bsp/nrf/family.cmake writes `${TOP}/hw/mcu/nordic/nrfx` — no trailing + # slash; the match must accept a directory-boundary end-of-token + self.assertEqual(ci_select.path_families('hw/mcu/nordic/nrfx', REPO), {'nrf'}) + + def test_boundary_rejects_prefix_sibling(self): + # 'microchip/pic' must not inherit pic32mz's references (and pic32mz itself + # is family.mk-only, which the CMake-only scan never reads) + self.assertEqual(ci_select.port_families('microchip/pic', REPO), set()) + self.assertEqual(ci_select.port_families('microchip/pic32mz', REPO), set()) + + def test_mcu_families_prefix_walk(self): + self.assertEqual(ci_select.mcu_families('hw/mcu/nordic/nrf5x/nrf_clock.h', REPO), {'nrf'}) + self.assertEqual(ci_select.mcu_families('hw/mcu/dialog/da1469x/x.h', REPO), {'da1469x'}) + self.assertEqual(ci_select.mcu_families('hw/mcu/no_such_vendor/x.c', REPO), set()) + + +class TestMcuHilRule(unittest.TestCase): + def test_mcu_no_longer_forces_full(self): + s = ci_select.classify(['hw/mcu/nordic/nrf5x/nrf_clock.h'], REPO, ROSTERS) + self.assertFalse(s['full']) + self.assertIn('nrf', s['families']) # recorded even with no nrf rig board + + def test_mcu_selects_family_boards(self): + got = on_roster(self, 'feather_nrf52840_express', 'pca10056', 'pca10095') + s = ci_select.classify(['hw/mcu/nordic/nrf5x/nrf_clock.h'], REPO, real_rosters()) + self.assertFalse(s['full']) + for b in got: + self.assertIn(b, s['boards']) + + +class TestOrphanInvariant(unittest.TestCase): + ALLOW = {'microchip/pic', 'microchip/pic32mz'} # spec: known orphans, CMake builds neither + + def test_every_port_resolves_to_a_family(self): + for d in sorted(glob.glob(os.path.join(REPO, 'src/portable/*/*'))): + if not os.path.isdir(d): + continue + port = os.path.relpath(d, os.path.join(REPO, 'src/portable')).replace(os.sep, '/') + fams = ci_select.port_families(port, REPO) + if port in self.ALLOW: + self.assertEqual(fams, set(), f'{port}: no longer an orphan - drop it from ALLOW') + else: + self.assertTrue(fams, f'{port}: no family.cmake references it - wire it up or allowlist it') + + def test_tracked_mcu_vendors_resolve(self): + import subprocess as sp + r = sp.run(['git', 'ls-files', 'hw/mcu'], cwd=REPO, capture_output=True, text=True) + if r.returncode != 0: + self.skipTest('not a git checkout') + vendors = sorted({'/'.join(p.split('/')[:3]) for p in r.stdout.split()}) + for v in vendors: + self.assertTrue(ci_select.mcu_families(v + '/x.c', REPO), f'{v}: resolves to no family') +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 test/hil/test/test_ci_select.py TestPathFamilies -v` +Expected: FAIL/ERROR — `path_families`/`mcu_families` not defined. + +- [ ] **Step 3: Implement** + +In `tools/ci_select.py`, replace `port_families` with: + +```python +@functools.lru_cache(maxsize=None) +def path_families(rel_dir: str, repo_root: str) -> set: + """Board families whose family.cmake (or espressif component CMakeLists) + references rel_dir at a directory boundary. CMake only, on every axis: CMake + is the first-class build system and Make follows it, so family.mk is never + read - a port wired up in family.mk alone (microchip/pic32mz) is built by no + CI job and resolves to nothing. Boundary = '/', whitespace, quote, paren, + brace or end: `${TOP}/hw/mcu/nordic/nrfx` has no trailing slash, while bare + 'microchip/pic' must not match '.../microchip/pic32mz/...'.""" + fams = set() + bsp_root = os.path.join(repo_root, 'hw/bsp') + pat = re.compile(re.escape(rel_dir) + r'(?=[/\s"\')}]|$)', re.M) + for f in glob.glob(os.path.join(bsp_root, '*/family.cmake')) + \ + glob.glob(os.path.join(bsp_root, '*/components/*/CMakeLists.txt')): + try: + if pat.search(open(f).read()): + fams.add(os.path.relpath(f, bsp_root).split(os.sep, 1)[0]) + except OSError: + pass + return fams + + +def port_families(port_dir: str, repo_root: str) -> set: + return path_families('src/portable/' + port_dir, repo_root) + + +def mcu_families(path: str, repo_root: str) -> set: + """Families referencing a changed hw/mcu path: longest resolving dir prefix, + hw/mcu///... down to hw/mcu/.""" + parts = path.split('/') + for n in range(len(parts) - 1, 2, -1): + fams = path_families('/'.join(parts[:n]), repo_root) + if fams: + return fams + return set() +``` + +Keep the old docstring's CMake-only rationale for HIL (folded into the new one). Remove `hw/mcu/|` from `_FULL_RE`. In `_classify_one`, insert after the `hw/bsp/` block, before the `examples/` block: + +```python + if re.match(r'hw/mcu/', path): + fams = mcu_families(path, repo_root) + s.families.update(fams) + boards = [b['name'] for b in roster_boards + if board_family(b['name'], repo_root) in fams] + s.roles.update(('device', 'host')) + s.add(boards, 'all', f'{path}: mcu dir -> families {sorted(fams)} -> boards {boards}') + return +``` + +- [ ] **Step 4: Run tests** + +Run: `python3 test/hil/test/test_ci_select.py -v 2>&1 | tail -5` +Expected: all pass (the pre-existing port tests exercise the wrapper). + +- [ ] **Step 5: Commit** + +```bash +git add tools/ci_select.py test/hil/test/test_ci_select.py +git commit -m "ci_select: generalize family scan to hw/mcu, drop hw/mcu from HIL full-matrix rule" +``` + +--- + +### Task 3: Build classifier — rules 1-17, raw two-axis selection + +**Files:** +- Modify: `tools/ci_select.py` +- Test: `test/hil/test/test_ci_select.py` + +**Interfaces:** +- Produces: `classify_build(changed_files, repo_root) -> dict` with keys `full: bool`, `families: [str]` (sorted bsp-dir names), `family_examples: {family: [example]}` (key absent ⇒ that family builds all examples; examples as `role/name`), `reasons: [str]`. Also `all_examples(repo_root) -> tuple[str]`, `role_examples(repo_root, roles) -> set[str]`, `all_bsp_families(repo_root) -> list[str]`. Buildability pruning is Task 4 — this task emits the raw rule output. +- Consumes: `path_families`, `mcu_families`, `class_macros`, `class_include_edges`, `_config_enables`, `_NONCODE_RE` (all existing). + +- [ ] **Step 1: Write the failing tests** + +```python +class TestBuildClassifier(unittest.TestCase): + def b(self, files): + return ci_select.classify_build(files, REPO) + + def test_noncode_and_test_hil_contribute_nothing(self): # rules 1, 2 + s = self.b(['docs/info/index.rst', 'README.rst', 'test/hil/hil_test.py', '.claude/skills/hil/SKILL.md']) + self.assertFalse(s['full']) + self.assertEqual(s['families'], []) + self.assertEqual(s['family_examples'], {}) + + def test_port_device_rule(self): # rule 3 + s = self.b(['src/portable/raspberrypi/rp2040/dcd_rp2040.c']) + self.assertFalse(s['full']) + self.assertEqual(s['families'], ['rp2040']) + exs = s['family_examples']['rp2040'] + self.assertIn('device/cdc_msc', exs) + self.assertFalse(any(e.startswith(('host/', 'typec/')) for e in exs)) + # dual inclusion asserted on the pure role helper: whether a dual example + # survives Task 4's buildability pruning depends on the environment-gated + # CI board pick, so the classifier-output assertion must not rely on it + self.assertIn('dual/host_info_to_device_cdc', + ci_select.role_examples(REPO, ('device', 'dual'))) + self.assertNotIn('host/bare_api', ci_select.role_examples(REPO, ('device', 'dual'))) + + def test_port_host_rule(self): # rule 4 + s = self.b(['src/portable/analog/max3421/hcd_max3421.c']) + self.assertFalse(s['full']) + # max3421 is referenced only by the espressif component CMakeLists — and + # espressif is in no provider's family list, so this may prune to nothing + self.assertLessEqual(set(s['families']), {'espressif'}) + for exs in s['family_examples'].values(): + self.assertFalse(any(e.startswith(('device/', 'typec/')) for e in exs)) + + def test_port_shared_file_selects_all_examples(self): # rule 5 + s = self.b(['src/portable/synopsys/dwc2/dwc2_common.c']) + self.assertFalse(s['full']) + self.assertIn('stm32f4', s['families']) + self.assertNotIn('rp2040', s['families']) + self.assertNotIn('stm32f4', s['family_examples']) # 'all' => no map key + + def test_bsp_family_rule(self): # rule 6 + s = self.b(['hw/bsp/stm32f4/boards/stm32f407disco/board.h']) + self.assertEqual(s['families'], ['stm32f4']) + self.assertNotIn('stm32f4', s['family_examples']) + + def test_bsp_top_level_file_is_full(self): # rule 16 + self.assertTrue(self.b(['hw/bsp/board.c'])['full']) + self.assertTrue(self.b(['hw/bsp/family_support.cmake'])['full']) + + def test_mcu_rule(self): # rule 7 + s = self.b(['hw/mcu/nordic/nrf5x/nrf_clock.h']) + self.assertEqual(s['families'], ['nrf']) + s = self.b(['hw/mcu/no_such_vendor/x.c']) # empty means empty + self.assertFalse(s['full']) + self.assertEqual(s['families'], []) + + def test_class_device_rule(self): # rule 8 + s = self.b(['src/class/cdc/cdc_device.c']) + self.assertFalse(s['full']) + # near-all families (Task 4's pruning may drop a few); never equality + # against all_bsp_families — that's a tuple, and pruning shrinks the list + self.assertIn('stm32f4', s['families']) + self.assertGreater(len(s['families']), 50) + exs = s['family_examples']['stm32f4'] + self.assertIn('device/cdc_msc', exs) + self.assertNotIn('device/hid_composite', exs) + self.assertNotIn('host/cdc_msc_hid', exs) # TUH_CDC examples are rule 9's + + def test_class_host_rule(self): # rule 9 + s = self.b(['src/class/msc/msc_host.c']) + exs = s['family_examples']['stm32f4'] + self.assertIn('host/msc_file_explorer', exs) + self.assertNotIn('device/cdc_msc', exs) + + def test_class_shared_header_and_include_edge(self): # rule 10 + s = self.b(['src/class/audio/audio.h']) + exs = s['family_examples']['stm32f4'] + self.assertIn('device/audio_test', exs) + self.assertIn('device/midi_test', exs) # midi headers include audio.h + + def test_core_device_rule(self): # rule 11 + s = self.b(['src/device/usbd.c']) + exs = s['family_examples']['stm32f4'] + self.assertIn('device/cdc_msc', exs) + # no dual In-assertion: dual examples are only.txt-gated to max3421/pio-usb + # boards, so pruning legitimately drops them on a plain stm32f4 board + self.assertFalse(any(e.startswith(('host/', 'typec/')) for e in exs)) + + def test_core_host_rule(self): # rule 12 + s = self.b(['src/host/usbh.c']) + exs = s['family_examples']['stm32f4'] + self.assertFalse(any(e.startswith(('device/', 'typec/')) for e in exs)) + + def test_example_rule(self): # rules 13, 14 + s = self.b(['examples/device/cdc_msc/src/main.c']) + self.assertEqual(s['family_examples']['stm32f4'], ['device/cdc_msc']) + s = self.b(['examples/device/board_test/src/main.c']) + self.assertEqual(s['family_examples']['stm32f4'], ['device/board_test']) + s = self.b(['examples/device/no_such_example/src/main.c']) # deleted example: nothing + self.assertFalse(s['full']) + self.assertEqual(s['families'], []) + + def test_full_paths(self): # rules 15-17 + for p in ('src/common/tusb_fifo.c', 'src/osal/osal.h', 'src/tusb.c', + 'src/tusb_option.h', 'lib/SEGGER_RTT/RTT/SEGGER_RTT.c', + 'tools/build.py', 'tools/get_deps.py', 'tools/cmake/cpu/cortex-m4.cmake', + 'examples/CMakeLists.txt', 'examples/device/CMakeLists.txt', + 'examples/build_system/cmake/cpu.cmake', '.github/workflows/build.yml', + 'sonar-project.properties', 'some/unknown/path.c'): + self.assertTrue(self.b([p])['full'], p) + + def test_mixed_diff_unions_per_family(self): + s = self.b(['src/portable/raspberrypi/rp2040/dcd_rp2040.c', 'src/class/cdc/cdc_device.c']) + self.assertFalse(s['full']) + self.assertIn('stm32f4', s['families']) + self.assertGreater(len(s['families']), 50) + self.assertIn('device/hid_composite', s['family_examples']['rp2040']) # from the dcd rule + self.assertNotIn('device/hid_composite', s['family_examples']['stm32f4']) # cdc-only there + + def test_example_names_are_real_dirs(self): + for ex in ci_select.all_examples(REPO): + role, name = ex.split('/') + self.assertTrue(os.path.isdir(os.path.join(REPO, 'examples', role, name)), ex) + self.assertRegex(ex, r'^(device|dual|host|typec)/[A-Za-z0-9_]+$') +``` + +Note for `test_mixed_diff_unions_per_family`: it encodes the per-family union — rp2040 gets DEV+DUAL ∪ cdc-set, every other family only the cdc-set (spec §Two axes). Buildability pruning may later remove entries; these Task-3 tests use families/examples that survive pruning (stm32f4 and rp2040 build all the named examples), so they stay valid after Task 4. + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 test/hil/test/test_ci_select.py TestBuildClassifier -v 2>&1 | tail -3` +Expected: ERROR — `classify_build` not defined. + +- [ ] **Step 3: Implement** (append to `tools/ci_select.py`, after the HIL classifier) + +```python +# ------------------------------------------------------------- +# Build-axis classifier (spec rule table, docs/superpowers/specs/ +# 2026-08-19-ci-build-family-filter-design.md). Independent of the HIL +# classifier: same diff, second walk, its own fail-open. +# ------------------------------------------------------------- +_EX_ROLES = ('device', 'dual', 'host', 'typec') + + +@functools.lru_cache(maxsize=None) +def all_examples(repo_root: str) -> tuple: + """Every examples// with a CMakeLists.txt, as 'role/name'.""" + out = [] + for role in _EX_ROLES: + for d in sorted(glob.glob(os.path.join(repo_root, 'examples', role, '*/'))): + if os.path.isfile(os.path.join(d, 'CMakeLists.txt')): + out.append(f'{role}/{os.path.basename(d.rstrip(os.sep))}') + return tuple(out) + + +def role_examples(repo_root: str, roles) -> set: + want = set(roles) + return {e for e in all_examples(repo_root) if e.split('/', 1)[0] in want} + + +@functools.lru_cache(maxsize=None) +def all_bsp_families(repo_root: str) -> tuple: + return tuple(sorted(d for d in os.listdir(os.path.join(repo_root, 'hw/bsp')) + if os.path.isdir(os.path.join(repo_root, 'hw/bsp', d)))) + + +def _build_class_examples(cls: str, base: str, roles: set, repo_root: str) -> set: + """Examples (all 46, not the HIL lists) whose tusb_config.h enables the class's + macros for the given roles, plus classes that #include the changed header.""" + via = sorted(class_include_edges(repo_root).get(f'{cls}/{base}', ())) + out = set() + for prefix, role in (('TUD', 'device'), ('TUH', 'host')): + if role not in roles: + continue + macros = class_macros(cls, base, prefix) + \ + [m for c in via for m in class_macros(c, '', prefix)] + for ex in all_examples(repo_root): + cfg = os.path.join(repo_root, 'examples', ex, 'src', 'tusb_config.h') + if _config_enables(cfg, macros): + out.add(ex) + return out + + +class _BSel: + """family -> set(examples) | 'all', unioned per family.""" + def __init__(self): + self.full = False + self.fam_ex = {} + self.reasons = [] + + def add(self, fams, examples, reason): + self.reasons.append(reason) + for f in fams: + cur = self.fam_ex.get(f) + if examples == 'all' or cur == 'all': + self.fam_ex[f] = 'all' + else: + self.fam_ex[f] = (cur or set()) | set(examples) + + def force_full(self, reason): + self.full = True + self.reasons.append(reason) + + +def _classify_build_one(path, repo_root, s: _BSel): + base = os.path.basename(path) + if _NONCODE_RE.match(path): # rule 1 + return + if re.match(r'test/hil/', path): # rule 2 + s.reasons.append(f'{path}: HIL harness, no build contribution') + return + m = re.match(r'src/portable/((?:[^/]+/)?[^/]+)/', path) + if m: # rules 3-5 + port = m.group(1) + fams = port_families(port, repo_root) + if re.match(r'(dcd_|.*_device)', base): + exs = role_examples(repo_root, ('device', 'dual')) + elif re.match(r'(hcd_|.*_host)', base): + exs = role_examples(repo_root, ('host', 'dual')) + else: + exs = 'all' + s.add(fams, exs, f'{path}: port {port} -> families {sorted(fams)}') + return + if re.match(r'hw/bsp/[^/]+/', path): # rule 6 + fam = path.split('/')[2] + s.add({fam}, 'all', f'{path}: bsp family {fam}') + return + if re.match(r'hw/mcu/', path): # rule 7 + fams = mcu_families(path, repo_root) + s.add(fams, 'all', f'{path}: mcu -> families {sorted(fams)}') + return + m = re.match(r'src/class/([^/]+)/', path) + if m: # rules 8-10 + cls = m.group(1) + if re.search(r'_device\.[ch]$', base): + roles = {'device'} + elif re.search(r'_host\.[ch]$', base): + roles = {'host'} + else: + roles = {'device', 'host'} + exs = _build_class_examples(cls, base, roles, repo_root) + s.add(all_bsp_families(repo_root), exs, + f'{path}: class {cls} -> {sorted(exs)}') + return + m = re.match(r'src/(device|host)/', path) + if m: # rules 11-12 + role = m.group(1) + s.add(all_bsp_families(repo_root), role_examples(repo_root, (role, 'dual')), + f'{path}: core {role} stack') + return + m = re.match(r'examples/(device|dual|host|typec)/([^/]+)/', path) + if m: # rules 13-14 + ex = f'{m.group(1)}/{m.group(2)}' + if ex in all_examples(repo_root): + s.add(all_bsp_families(repo_root), {ex}, f'{path}: example {ex}') + else: + # a deleted example builds nothing; removing it from the role + # CMakeLists (rule 15) is what forces the full matrix + s.reasons.append(f'{path}: not an example dir, no build contribution') + return + s.force_full(f'{path}: unclassified -> full build matrix') # rules 15-17 + + +def classify_build(changed_files, repo_root): + s = _BSel() + for p in changed_files: + _classify_build_one(p, repo_root, s) + if s.full: + return {'full': True, 'families': list(all_bsp_families(repo_root)), + 'family_examples': {}, 'reasons': s.reasons} + fams, fam_ex = [], {} + for fam, exs in sorted(s.fam_ex.items()): + fams.append(fam) + if exs != 'all': + fam_ex[fam] = sorted(exs) + return {'full': False, 'families': fams, 'family_examples': fam_ex, + 'reasons': s.reasons} +``` + +Note: `examples//CMakeLists.txt` has no trailing slash after the second component, so the example regex misses it and it correctly falls through to `force_full` (rule 15) — `test_full_paths` pins this. + +- [ ] **Step 4: Run tests** + +Run: `python3 test/hil/test/test_ci_select.py TestBuildClassifier -v` +Expected: all pass. Then the full file: `python3 test/hil/test/test_ci_select.py 2>&1 | tail -3` — all pass. + +- [ ] **Step 5: Commit** + +```bash +git add tools/ci_select.py test/hil/test/test_ci_select.py +git commit -m "ci_select: add build-axis classifier (families x example targets)" +``` + +--- + +### Task 4: Buildability post-filter, `build` + `hil_examples` output keys + +**Files:** +- Modify: `tools/ci_select.py` (imports, post-filter, `main()`), `test/hil/test/test_hil_util.py` (BottomLayer lists) +- Test: `test/hil/test/test_ci_select.py` + +**Interfaces:** +- Produces: `classify_build` result is now pruned: each family's list intersected with what that family's CI board can build (`build_utils.skip_example`); family dropped when nothing survives; map key omitted when the kept set equals everything the board can build. `hil_examples(sel, rosters) -> {board: [example]}` — the board's selected tests (`sel['boards'][name]` when narrowed, else `board_tests`) plus always `device/board_test`. CLI JSON gains top-level `"build": {...}` (always) and `"hil_examples": {...}` (when rosters given; emitted even when `full` is true). +- Consumes: `tools/build_utils.skip_example(example, board)`; `tools/build.py:get_family_boards(family, one_random, one_first)` (module import — no behavior change to build.py yet). + +- [ ] **Step 1: Write the failing tests** + +```python +class TestBuildPostFilter(unittest.TestCase): + def test_kept_examples_are_buildable(self): + import build_utils, build as build_py + s = ci_select.classify_build(['src/class/msc/msc_host.c'], REPO) + self.assertFalse(s['full']) + # families that cannot build a single TUH_MSC example drop out entirely + self.assertNotIn('msp430', s['families']) + old = os.getcwd() + os.chdir(REPO) + try: + for fam, exs in s['family_examples'].items(): + board = build_py.get_family_boards(fam, False, True)[0] + for e in exs: + self.assertFalse(build_utils.skip_example(e, board), f'{fam}: {e}') + finally: + os.chdir(old) + + def test_unfiltered_family_has_no_map_key(self): + s = ci_select.classify_build(['hw/bsp/stm32f4/family.c'], REPO) + self.assertEqual(s['families'], ['stm32f4']) + self.assertEqual(s['family_examples'], {}) + + def test_no_stdout_pollution(self): + # get_family_boards prints on odd families; the selector's stdout is JSON + import io, contextlib + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + ci_select.classify_build(['src/class/msc/msc_host.c'], REPO) + self.assertEqual(buf.getvalue(), '') + + +class TestHilExamples(unittest.TestCase): + def test_board_test_always_present_and_full_emits(self): + s = ci_select.classify(['src/common/tusb_fifo.c'], REPO, ROSTERS) # full + he = ci_select.hil_examples(s, ROSTERS) + self.assertEqual(set(he), {b['name'] for b in ROSTER}) + for name, exs in he.items(): + self.assertIn('device/board_test', exs) + + def test_narrowed_board_gets_chosen_tests_only(self): + s = ci_select.classify(['examples/device/cdc_msc/src/main.c'], REPO, ROSTERS) + he = ci_select.hil_examples(s, ROSTERS) + self.assertEqual(he['stm32f407disco'], ['device/board_test', 'device/cdc_msc']) + + def test_full_board_gets_its_whole_test_list(self): + s = ci_select.classify(['hw/bsp/stm32f4/boards/stm32f407disco/board.h'], REPO, ROSTERS) + he = ci_select.hil_examples(s, ROSTERS) + want = set(ci_select.board_tests(ROSTER[1])) | {'device/board_test'} + self.assertEqual(set(he['stm32f407disco']), want) + self.assertNotIn('raspberry_pi_pico', he) # deselected board: no firmware needed + + +class TestCliJson(unittest.TestCase): + def test_build_key_without_rosters(self): + r = subprocess.run([sys.executable, os.path.join(REPO, 'tools/ci_select.py'), + '--diff-file', '/dev/null'], capture_output=True, text=True) + self.assertEqual(r.returncode, 0, r.stderr) + j = json.loads(r.stdout) + self.assertIn('build', j) + self.assertNotIn('hil_examples', j) # rosters not given + + def test_build_and_hil_keys_with_rosters(self): + import tempfile + with tempfile.NamedTemporaryFile('w', suffix='.txt', delete=False) as f: + f.write('src/portable/raspberrypi/rp2040/dcd_rp2040.c\n') + df = f.name + r = subprocess.run([sys.executable, os.path.join(REPO, 'tools/ci_select.py'), + '--diff-file', df, os.path.join(REPO, 'test/hil/tinyusb.json')], + capture_output=True, text=True) + os.unlink(df) + j = json.loads(r.stdout) + self.assertEqual(j['build']['families'], ['rp2040']) + self.assertIn('hil_examples', j) + for exs in j['hil_examples'].values(): + self.assertIn('device/board_test', exs) +``` + +(`subprocess`, `sys` are already imported in the test file.) + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 test/hil/test/test_ci_select.py TestBuildPostFilter TestHilExamples TestCliJson -v 2>&1 | tail -3` +Expected: FAIL — no pruning, no `hil_examples`, no `build` key. + +- [ ] **Step 3: Implement** + +In `tools/ci_select.py` module header, after the existing `helper` import, add: + +```python +import contextlib +import io + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # tools/, for build helpers +import build_utils +import build as build_py +``` + +(`contextlib`/`io` go into the stdlib import block at the top.) Add the pruning helpers and rewrite the tail of `classify_build`: + +```python +@contextlib.contextmanager +def _in_repo(repo_root): + """build_utils/build.py use repo-relative paths; scope a chdir around them. + get_family_boards also prints on an empty family - swallow stdout so the + selector's machine-read JSON stays clean (diagnostics belong on stderr).""" + old = os.getcwd() + os.chdir(repo_root) + try: + with contextlib.redirect_stdout(io.StringIO()): + yield + finally: + os.chdir(old) + + +def _prune_buildable(fams, fam_ex, repo_root): + """Intersect each family's selection with what its CI board can build + (build_utils.skip_example - the same skip.txt/only.txt data CMake's + family_filter reads). get_family_boards mirrors the build jobs' one-first + pick, CI preferred/skip lists included.""" + out_fams, out_ex = [], {} + allex = list(all_examples(repo_root)) + with _in_repo(repo_root): + for fam in fams: + boards = build_py.get_family_boards(fam, False, True) + if not boards: + out_fams.append(fam) # unknown layout: keep unfiltered + continue + board = boards[0] + buildable = [e for e in allex if not build_utils.skip_example(e, board)] + want = fam_ex.get(fam) + kept = buildable if want is None else [e for e in want if e in set(buildable)] + if not kept: + continue # this diff builds nothing for this family + out_fams.append(fam) + if set(kept) != set(buildable): + out_ex[fam] = kept + return out_fams, out_ex +``` + +Replace `classify_build`'s non-full return with: + +```python + fams = sorted(s.fam_ex) + fam_ex = {f: sorted(e) for f, e in s.fam_ex.items() if e != 'all'} + fams, fam_ex = _prune_buildable(fams, fam_ex, repo_root) + return {'full': False, 'families': fams, 'family_examples': fam_ex, + 'reasons': s.reasons} +``` + +Add `hil_examples` beside `selection_args`: + +```python +def hil_examples(sel, rosters): + """{board: examples hil-build must produce}: the board's selected tests plus + device/board_test, which hil_test.py flashes to park at every variant + boundary and at end-of-board teardown. Emitted for full selections too - the + HIL example universe is a fraction of the tree regardless of the diff.""" + by_name = {} + for _, boards in rosters: + for b in boards: + by_name.setdefault(b['name'], b) + if sel['full']: + chosen = {n: 'all' for n in by_name} + else: + chosen = sel['boards'] + out = {} + for name, tests in chosen.items(): + run = board_tests(by_name[name]) if tests == 'all' else list(tests) + out[name] = sorted(set(run) | {'device/board_test'}) + return out +``` + +In `main()`: change the configs argument to optional — `ap.add_argument('configs', nargs='*', help='rig roster JSON file(s); omit for the build view alone')` — so CircleCI (which never touches HIL) can run without rosters; with no configs, `rosters` is `[]`, the HIL keys degrade to empty, and `hil_examples` is omitted. Then after the `args_flasher` line: + +```python + if rosters: + s['hil_examples'] = hil_examples(s, rosters) + s['build'] = classify_build(files, repo_root) + for r in s['build']['reasons']: + print(f'ci_select[build]: {r}', file=sys.stderr) +``` + +Update `test/hil/test/test_hil_util.py` BottomLayer: add `'build'`, `'build_utils'` to the `local` allowed set and `'../../tools/build'`, `'../../tools/build_utils'` to the module-path tuple (ci_select now imports both on the bare runner). + +- [ ] **Step 4: Run tests + timing check** + +```bash +python3 test/hil/test/test_ci_select.py 2>&1 | tail -3 +python3 -m unittest discover -s test/hil/test 2>&1 | tail -3 +time python3 tools/ci_select.py --diff-file <(echo src/class/cdc/cdc_device.c) test/hil/tinyusb.json >/dev/null +``` + +Expected: suites pass; the timed run stays under ~5 s (skip_example over 75 families × 46 examples re-reads small files — if it exceeds that, memoize `skip_example` results per (example, board) inside `_prune_buildable`). + +- [ ] **Step 5: Commit** + +```bash +git add tools/ci_select.py test/hil/test/test_ci_select.py test/hil/test/test_hil_util.py +git commit -m "ci_select: prune build selection by example buildability, emit build + hil_examples keys" +``` + +--- + +### Task 5: `ci_set_matrix.py --select / --base` + +**Files:** +- Modify: `.github/scripts/ci_set_matrix.py` +- Test: `test/hil/test/test_ci_select.py` + +**Interfaces:** +- Produces: CLI `python .github/scripts/ci_set_matrix.py [--select JSON | --base REF]`. No flags → byte-identical to today's output. `--select`: families intersected with `select.build.families` unless `build.full`; unusable JSON → full matrix + stderr warning. `--base REF`: runs `tools/ci_select.py --base REF` itself and proceeds as `--select`. Output shape `{toolchain: [family]}` unchanged. + +- [ ] **Step 1: Write the failing tests** + +```python +SET_MATRIX = os.path.join(REPO, '.github/scripts/ci_set_matrix.py') + +class TestCiSetMatrix(unittest.TestCase): + def run_matrix(self, *args): + return subprocess.run([sys.executable, SET_MATRIX, *args], + capture_output=True, text=True) + + def test_no_flags_is_todays_output(self): + r = self.run_matrix() + self.assertEqual(r.returncode, 0, r.stderr) + self.baseline = json.loads(r.stdout) + self.assertIn('stm32f4', self.baseline['arm-gcc']) + + def test_select_full_is_identical(self): + base = json.loads(self.run_matrix().stdout) + sel = json.dumps({'build': {'full': True, 'families': [], 'family_examples': {}}}) + self.assertEqual(json.loads(self.run_matrix('--select', sel).stdout), base) + + def test_select_narrow_is_a_subset(self): + sel = json.dumps({'build': {'full': False, 'families': ['rp2040', 'stm32f4'], + 'family_examples': {}}}) + m = json.loads(self.run_matrix('--select', sel).stdout) + self.assertEqual(m['arm-gcc'], ['rp2040', 'stm32f4']) + self.assertEqual(m['riscv-gcc'], []) + self.assertEqual(set(m), set(json.loads(self.run_matrix().stdout))) # all keys kept + + def test_malformed_select_falls_open(self): + base = json.loads(self.run_matrix().stdout) + r = self.run_matrix('--select', 'not json {') + self.assertEqual(r.returncode, 0) + self.assertEqual(json.loads(r.stdout), base) + self.assertIn('full matrix', r.stderr) +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 test/hil/test/test_ci_select.py TestCiSetMatrix -v 2>&1 | tail -3` +Expected: FAIL — argparse rejects `--select`. + +- [ ] **Step 3: Implement** + +In `.github/scripts/ci_set_matrix.py`, add imports `argparse, os, subprocess, sys` and replace `set_matrix_json` + the main guard: + +```python +def set_matrix_json(select=None): + sel_fams = None + if select: + b = select.get('build') or {} + if b.get('full') is False: + sel_fams = set(b.get('families') or []) + matrix = {} + for toolchain in toolchain_list: + fams = [family for family, tc in family_list.items() if toolchain in tc] + if sel_fams is not None: + fams = [f for f in fams if f in sel_fams] + matrix[toolchain] = fams + print(json.dumps(matrix)) + + +def main(): + parser = argparse.ArgumentParser() + group = parser.add_mutually_exclusive_group() + group.add_argument('--select', help='tools/ci_select.py JSON; scopes families when build.full is false') + group.add_argument('--base', help='git ref: run tools/ci_select.py --base REF and scope from it') + args = parser.parse_args() + + select = None + try: + if args.select: + select = json.loads(args.select) + elif args.base: + root = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + r = subprocess.run([sys.executable, os.path.join(root, 'tools', 'ci_select.py'), + '--base', args.base], + capture_output=True, text=True, cwd=root, check=True) + select = json.loads(r.stdout) + except Exception as e: # fail-open: an unusable selection must never turn into a red job + print(f'ci_set_matrix: selection unusable ({e}) - full matrix', file=sys.stderr) + select = None + set_matrix_json(select) + + +if __name__ == '__main__': + main() +``` + +- [ ] **Step 4: Run tests** + +Run: `python3 test/hil/test/test_ci_select.py TestCiSetMatrix -v` — all pass. +Also: `python3 .github/scripts/ci_set_matrix.py | diff - <(git show HEAD:.github/scripts/ci_set_matrix.py | python3 -)` → no diff (byte-identical default output). + +- [ ] **Step 5: Extend the pre-commit hook scope and commit** + +In `.pre-commit-config.yaml`, `ci-select-test` hook: `files: ^(hw/bsp/|src/|examples/|tools/(ci_select|build|build_utils)\.py$|\.github/scripts/)`. + +```bash +git add .github/scripts/ci_set_matrix.py test/hil/test/test_ci_select.py .pre-commit-config.yaml +git commit -m "ci_set_matrix: scope the family matrix from a ci_select selection" +``` + +--- + +### Task 6: `hil_ci_set_matrix.py` emits `-e` per board + +**Files:** +- Modify: `.github/scripts/hil_ci_set_matrix.py` +- Test: `test/hil/test/test_ci_select.py` + +**Interfaces:** +- Produces: each build entry for board `B` gains ` -e ` for every entry of `select.hil_examples[B]` (before variant expansion, so all of a board's variants carry the same list). No `hil_examples` key (hand runs, old selectors) → output byte-identical to today. +- Consumed by: `hil-build` / `hil-build-esp` (via `build_util.yml` → `tools/build.py`), `hil-hfp-iar`'s inline build loop — all funnel into `tools/build.py`, which learns `-e` in Task 7. + +- [ ] **Step 1: Write the failing tests** + +```python +HIL_SET_MATRIX = os.path.join(REPO, '.github/scripts/hil_ci_set_matrix.py') + +class TestHilCiSetMatrixExamples(unittest.TestCase): + def run_matrix(self, *args): + r = subprocess.run([sys.executable, HIL_SET_MATRIX, *args, + os.path.join(REPO, 'test/hil/tinyusb.json')], + capture_output=True, text=True) + self.assertEqual(r.returncode, 0, r.stderr) + return r.stdout + + def test_no_hil_examples_is_byte_identical(self): + plain = self.run_matrix() + sel = json.dumps({'full': True, 'boards': {}}) + self.assertEqual(self.run_matrix('--select', sel), plain) + + def test_examples_appended_per_board(self): + board = on_roster(self, 'stm32f407disco')[0] + sel = json.dumps({'full': False, 'boards': {board: 'all'}, + 'hil_examples': {board: ['device/board_test', 'device/cdc_msc']}}) + m = json.loads(self.run_matrix('--select', sel)) + entries = [e for entries in m.values() for e in entries] + self.assertTrue(entries) + for e in entries: + self.assertIn(f'-b {board}', e) + self.assertIn('-e device/board_test', e) + self.assertIn('-e device/cdc_msc', e) +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 test/hil/test/test_ci_select.py TestHilCiSetMatrixExamples -v` +Expected: `test_examples_appended_per_board` FAILS (no `-e` in entries). + +- [ ] **Step 3: Implement** + +In `hil_ci_set_matrix.py` `main()`, after the `selected` computation add `ex_map = (sel or {}).get('hil_examples', {})`, and in the board loop, after the `build.args` append (line ~72): + +```python + # PR selection: build only the examples this board will run (its test + # list plus device/board_test, the parking firmware) - tools/build.py -e. + # Absent key (hand runs, full non-PR builds) keeps --target all. + for ex in ex_map.get(name, []): + build_board += f' -e {ex}' +``` + +- [ ] **Step 4: Run tests** + +Run: `python3 test/hil/test/test_ci_select.py -v 2>&1 | tail -3` — all pass. + +- [ ] **Step 5: Commit** + +```bash +git add .github/scripts/hil_ci_set_matrix.py test/hil/test/test_ci_select.py +git commit -m "hil_ci_set_matrix: append per-board -e example filters from the selection" +``` + +--- + +### Task 7: `tools/build.py --example` + +**Files:** +- Modify: `tools/build.py` +- Test: `test/hil/test/test_ci_select.py` + +**Interfaces:** +- Produces: repeatable `-e/--example role/name`. Without it, behavior is exactly today's (`--target all`). With it: cmake builds one `--target ` per requested example the board can build (`build_utils.skip_example`), mapping `all` → example names and `examples-membrowse-upload` → `-membrowse-upload` (the aggregate target `DEPENDS` every example — `hw/bsp/family_support.cmake:346-360` — and would rebuild the excluded ones); `tinyusb_metrics` and other targets pass through, order preserved. A board whose intersection is empty reports **skipped**. Make and espressif paths filter their example lists the same way. New helper `resolve_example_targets(build_targets, examples, board) -> list | None` (None = nothing buildable). + +- [ ] **Step 1: Write the failing tests** + +```python +class TestBuildPyExampleFilter(unittest.TestCase): + def setUp(self): + import build as build_py + self.build = build_py + self.old = os.getcwd() + os.chdir(REPO) # skip_example uses repo-relative paths + + def tearDown(self): + os.chdir(self.old) + + def test_all_maps_to_example_names(self): + t = self.build.resolve_example_targets(['all'], ['device/cdc_msc', 'device/dfu'], + 'stm32f407disco') + self.assertEqual(t, ['cdc_msc', 'dfu']) + + def test_membrowse_maps_per_example(self): + t = self.build.resolve_example_targets(['all', 'examples-membrowse-upload'], + ['device/cdc_msc'], 'stm32f407disco') + self.assertEqual(t, ['cdc_msc', 'cdc_msc-membrowse-upload']) + + def test_other_targets_pass_through_in_order(self): + t = self.build.resolve_example_targets(['all', 'tinyusb_metrics'], + ['device/cdc_msc'], 'stm32f407disco') + self.assertEqual(t, ['cdc_msc', 'tinyusb_metrics']) + + def test_unbuildable_examples_drop_and_empty_is_none(self): + # typec/power_delivery only builds on stm32g4-class parts, never on f4 + t = self.build.resolve_example_targets(['all'], + ['typec/power_delivery', 'device/cdc_msc'], + 'stm32f407disco') + self.assertEqual(t, ['cdc_msc']) + self.assertIsNone(self.build.resolve_example_targets(['all'], + ['typec/power_delivery'], + 'stm32f407disco')) +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 test/hil/test/test_ci_select.py TestBuildPyExampleFilter -v` +Expected: ERROR — `resolve_example_targets` not defined. + +- [ ] **Step 3: Implement** + +In `tools/build.py` add near `get_examples`: + +```python +def resolve_example_targets(build_targets, examples, board): + """Map generic targets onto per-example targets for a filtered build (-e). + 'all' -> the example executables; 'examples-membrowse-upload' -> per-example + upload targets (the aggregate DEPENDS on every example and would rebuild the + excluded ones); anything else (e.g. tinyusb_metrics) passes through. + Returns None when no requested example is buildable on this board.""" + buildable = [e for e in examples if not build_utils.skip_example(e, board)] + if not buildable: + return None + names = [e.split('/', 1)[1] for e in buildable] + out = [] + for t in build_targets: + if t == 'all': + out += names + elif t == 'examples-membrowse-upload': + out += [f'{n}-membrowse-upload' for n in names] + else: + out.append(t) + return list(dict.fromkeys(out)) +``` + +Thread `examples` (a list or `None`) through `main()` → `build_boards_list` → `cmake_board`/`make_board`: + +- `main()`: `parser.add_argument('-e', '--example', action='append', default=[], help='Only build these examples (role/name, repeatable). Default: all examples')`; pass `args.example or None` as a new final parameter of `build_boards_list`. +- `build_boards_list(..., examples=None)`: forward to both branches. +- `cmake_board(..., examples=None)`: in the espressif branch, after `all_examples = get_examples(family)` insert: + +```python + if examples is not None: + all_examples = [e for e in all_examples if e in examples] +``` + + In the generic branch, replace the target loop: + +```python + if rcmd.returncode == 0: + targets = build_targets + if examples is not None: + targets = resolve_example_targets(build_targets, examples, board) + if targets is None: + print_build_result(board, 'examples (PR filter)', 2, '-') + return [0, 0, 1] + cmd = ["cmake", "--build", build_dir, '--parallel', str(parallel_jobs)] + for target in targets: + rcmd = run_cmd(cmd + ['--target', target]) + if rcmd.returncode != 0: + break +``` + +- `make_board(..., examples=None)`: after `all_examples = get_examples(family)`: + +```python + if examples is not None: + all_examples = [e for e in all_examples if e in examples] + if not all_examples: + print_build_result(board, 'examples (PR filter)', 2, '-') + return [0, 0, 1] +``` + +- [ ] **Step 4: Run tests + a real filtered build** + +```bash +python3 test/hil/test/test_ci_select.py TestBuildPyExampleFilter -v +python3 tools/build.py -e device/cdc_msc -e device/cdc_dual_ports -b stm32f407disco +ls cmake-build/cmake-build-stm32f407disco/device/cdc_msc/cdc_msc.elf \ + cmake-build/cmake-build-stm32f407disco/device/cdc_dual_ports/cdc_dual_ports.elf +python3 tools/build.py -e typec/power_delivery -b stm32f407disco # expect: Skipped row, exit 0 +``` + +Expected: tests pass; both elfs exist; the typec run prints a Skipped result and exits 0. + +- [ ] **Step 5: Commit** + +```bash +git add tools/build.py test/hil/test/test_ci_select.py +git commit -m "build.py: add -e/--example filter with per-example target mapping" +``` + +--- + +### Task 8: `metrics.py --by-example` + by-example expansion + CMake wiring + +**Files:** +- Modify: `tools/metrics.py`, `examples/CMakeLists.txt`, `.pre-commit-config.yaml` +- Create + Test: `test/hil/test/test_ci_metrics.py` + +**Interfaces:** +- Produces: `metrics.py combine --by-example` additionally writes `_by_example.json` = `{"/": {"files": [...]}}`, the example id taken from the map.json's two parent dirs (`///*.map.json`). `combine` also accepts a by-example JSON as *input*, expanding each example to one data entry, with `--only-examples a,b` filtering which. `combine_files(input_files, filters=None, only_examples=None)`. Existing outputs byte-identical when the new flags are absent. +- Consumed by: `examples/CMakeLists.txt` `tinyusb_metrics` target (adds the flag), Task 9's pair-compare, Task 10's artifact upload. + +- [ ] **Step 1: Write the failing tests** (new file `test/hil/test/test_ci_metrics.py`) + +```python +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +# Unit tests for the by-example half of tools/metrics.py and the (family, example) +# pair-compare script. Stdlib only; synthetic map.json fixtures, no builds. +# python3 test/hil/test/test_ci_metrics.py +import json +import os +import subprocess +import sys +import tempfile +import unittest + +REPO = os.path.dirname(os.path.dirname(os.path.dirname( + os.path.dirname(os.path.abspath(__file__))))) +METRICS = os.path.join(REPO, 'tools', 'metrics.py') + + +def fake_map(path, files): + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, 'w') as f: + json.dump({'files': files}, f) + + +def entry(name, size, path_prefix='tinyusb/src'): + return {'file': name, 'path': f'{path_prefix}/{name}', 'size': size, + 'symbols': [{'name': f'{name}_fn', 'size': size}], 'sections': {'.text': size}} + + +class TestByExample(unittest.TestCase): + def build_tree(self, td): + fake_map(os.path.join(td, 'device', 'cdc_msc', 'cdc_msc.map.json'), + [entry('usbd.c', 100), entry('cdc_device.c', 50)]) + fake_map(os.path.join(td, 'host', 'bare_api', 'bare_api.map.json'), + [entry('usbh.c', 200)]) + + def test_by_example_output(self): + with tempfile.TemporaryDirectory() as td: + self.build_tree(td) + out = os.path.join(td, 'metrics') + r = subprocess.run([sys.executable, METRICS, 'combine', '-q', '-j', + '--by-example', '-o', out, + os.path.join(td, '*', '*', '*.map.json')], + capture_output=True, text=True) + self.assertEqual(r.returncode, 0, r.stderr) + by_ex = json.load(open(out + '_by_example.json')) + self.assertEqual(set(by_ex), {'device/cdc_msc', 'host/bare_api'}) + self.assertEqual({f['file'] for f in by_ex['device/cdc_msc']['files']}, + {'usbd.c', 'cdc_device.c'}) + # the plain averaged output is unchanged by the extra flag + avg = json.load(open(out + '.json')) + self.assertIn('files', avg) + + def test_by_example_json_roundtrips_as_combine_input(self): + with tempfile.TemporaryDirectory() as td: + self.build_tree(td) + out = os.path.join(td, 'metrics') + subprocess.run([sys.executable, METRICS, 'combine', '-q', '-j', '--by-example', + '-o', out, os.path.join(td, '*', '*', '*.map.json')], check=True) + out2 = os.path.join(td, 'sub') + r = subprocess.run([sys.executable, METRICS, 'combine', '-q', '-j', + '--only-examples', 'device/cdc_msc', + '-o', out2, out + '_by_example.json'], + capture_output=True, text=True) + self.assertEqual(r.returncode, 0, r.stderr) + sub = json.load(open(out2 + '.json')) + names = {f['file'] for f in sub['files']} + self.assertEqual(names, {'usbd.c', 'cdc_device.c'}) # bare_api filtered out + + +if __name__ == '__main__': + unittest.main() +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 test/hil/test/test_ci_metrics.py -v` +Expected: FAIL — argparse rejects `--by-example`. + +- [ ] **Step 3: Implement in `tools/metrics.py`** + +`combine_files` signature → `combine_files(input_files, filters=None, only_examples=None)`. Inside the `.json` branch, after `json.load`, insert the by-example expansion before the filter logic: + +```python + if 'files' not in json_data and json_data and \ + all(isinstance(v, dict) and 'files' in v for v in json_data.values()): + # a metrics_by_example.json: one data entry per example + for ex in sorted(json_data): + if only_examples and ex not in only_examples: + continue + sub = {'files': list(json_data[ex]['files'])} + if filters: + sub['files'] = [f for f in sub['files'] + if f.get('path') and any(x in f['path'] for x in filters)] + all_json_data['file_list'].append(f'{fin}:{ex}') + all_json_data['data'].append(sub) + continue +``` + +Add a writer near `write_json_output`: + +```python +def write_by_example(input_files, filters, path): + """{/: {files: [...]}} from map.json inputs laid out as + ///.map.json (examples/CMakeLists.txt's pattern).""" + out = {} + for fin in input_files: + d = os.path.dirname(os.path.abspath(fin)) + ex = f'{os.path.basename(os.path.dirname(d))}/{os.path.basename(d)}' + data = combine_files([fin], filters) + if data['data']: + out.setdefault(ex, {'files': []})['files'] += data['data'][0].get('files', []) + with open(path, 'w', encoding='utf-8') as f: + json.dump(out, f) +``` + +`cmd_combine`: pass `only_examples=set(args.only_examples.split(',')) if args.only_examples else None` into `combine_files`, and after the existing outputs: + +```python + if args.by_example: + write_by_example(input_files, args.filters, args.out + '_by_example.json') +``` + +Argparse additions on the combine subparser: + +```python + combine_parser.add_argument('--by-example', dest='by_example', action='store_true', + help='Also write _by_example.json: per-example file lists keyed by role/example') + combine_parser.add_argument('--only-examples', dest='only_examples', default='', + help='Comma-separated role/example ids to keep when reading by-example JSON inputs') +``` + +- [ ] **Step 4: Wire CMake + hooks** + +`examples/CMakeLists.txt` `tinyusb_metrics` target: change the command to +`combine -f tinyusb/src -j --by-example -o ${CMAKE_BINARY_DIR}/metrics` (one added flag). +`.pre-commit-config.yaml` `hil-test` hook: `files: ^(test/hil/|examples/device/mtp/src/|tools/metrics\.py$|\.github/scripts/metrics_pair_compare\.py$)`. + +- [ ] **Step 5: Run tests** + +```bash +python3 test/hil/test/test_ci_metrics.py -v # pass +python3 -m unittest discover -s test/hil/test 2>&1 | tail -3 # discovery picks the new file up +``` + +- [ ] **Step 6: Commit** + +```bash +git add tools/metrics.py examples/CMakeLists.txt test/hil/test/test_ci_metrics.py .pre-commit-config.yaml +git commit -m "metrics: emit and consume per-example size data (--by-example, --only-examples)" +``` + +--- + +### Task 9: `(family, example)`-intersection compare script + +**Files:** +- Create: `.github/scripts/metrics_pair_compare.py` +- Test: `test/hil/test/test_ci_metrics.py` + +**Interfaces:** +- Produces: CLI `metrics_pair_compare.py --base-dir D1 --new-dir D2 [--out metrics_compare]`. Each dir is searched recursively for `cmake-build-/metrics_by_example.json`; board → family via `hw/bsp/*/boards/`. Writes `.md`: the standard compare table over the intersection of `(family, example)` pairs, then a scope footer naming the compared families and any pairs missing on one side. Empty intersection → an explanatory one-line `.md`, exit 0. +- Consumes: `tools/metrics.py` internals `combine_files`/`compute_avg`-backed `compare_files` and `write_compare_markdown` (via `sys.path` import). + +- [ ] **Step 1: Write the failing tests** (append to `test_ci_metrics.py`) + +```python +PAIR_COMPARE = os.path.join(REPO, '.github/scripts/metrics_pair_compare.py') + + +def fake_by_example(root, board, data): + d = os.path.join(root, f'cmake-build-{board}') + os.makedirs(d, exist_ok=True) + with open(os.path.join(d, 'metrics_by_example.json'), 'w') as f: + json.dump(data, f) + + +class TestPairCompare(unittest.TestCase): + def test_intersection_compare(self): + with tempfile.TemporaryDirectory() as td: + base, new = os.path.join(td, 'base'), os.path.join(td, 'new') + # real board names so board->family resolution works against hw/bsp + fake_by_example(base, 'raspberry_pi_pico', + {'device/cdc_msc': {'files': [entry('usbd.c', 100)]}, + 'device/dfu': {'files': [entry('dfu_device.c', 10)]}}) + fake_by_example(new, 'raspberry_pi_pico', + {'device/cdc_msc': {'files': [entry('usbd.c', 120)]}}) + out = os.path.join(td, 'cmp') + r = subprocess.run([sys.executable, PAIR_COMPARE, '--base-dir', base, + '--new-dir', new, '--out', out], + capture_output=True, text=True) + self.assertEqual(r.returncode, 0, r.stderr) + md = open(out + '.md').read() + self.assertIn('usbd.c', md) + self.assertNotIn('dfu_device.c', md) # not on both sides + self.assertIn('rp2040', md) # scope footer + self.assertIn('device/dfu', md) # named as dropped + + def test_empty_intersection_writes_note(self): + with tempfile.TemporaryDirectory() as td: + base, new = os.path.join(td, 'base'), os.path.join(td, 'new') + fake_by_example(base, 'raspberry_pi_pico', {'device/dfu': {'files': [entry('a.c', 1)]}}) + fake_by_example(new, 'stm32f407disco', {'device/cdc_msc': {'files': [entry('b.c', 1)]}}) + out = os.path.join(td, 'cmp') + r = subprocess.run([sys.executable, PAIR_COMPARE, '--base-dir', base, + '--new-dir', new, '--out', out], + capture_output=True, text=True) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn('skipped', open(out + '.md').read()) +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 test/hil/test/test_ci_metrics.py TestPairCompare -v` +Expected: FAIL — script does not exist. + +- [ ] **Step 3: Implement `.github/scripts/metrics_pair_compare.py`** + +```python +#!/usr/bin/env python3 +"""Family+example-matched code-size compare for PR-scoped builds. + +The averaged metrics baseline (metrics-tinyusb) spans every family and example; +a scoped PR builds a subset, so comparing against it is apples-to-oranges. This +compares the intersection of (family, example) pairs present on BOTH sides, +averaged over exactly those pairs, and names what was dropped. See +docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md #code-metrics. +""" +import argparse +import glob +import json +import os +import sys +import tempfile + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', '..', 'tools')) +import metrics + + +def board_family(board, repo_root): + hits = glob.glob(os.path.join(repo_root, 'hw/bsp/*/boards', board)) + return os.path.basename(os.path.dirname(os.path.dirname(hits[0]))) if hits else None + + +def collect(root, repo_root): + """{(family, 'role/example'): [file entries]} from every + **/cmake-build-/metrics_by_example.json under root.""" + pairs = {} + pat = os.path.join(root, '**', 'metrics_by_example.json') + for f in sorted(glob.glob(pat, recursive=True)): + board = os.path.basename(os.path.dirname(f)) + if not board.startswith('cmake-build-'): + continue + fam = board_family(board[len('cmake-build-'):], repo_root) + if not fam: + print(f'pair_compare: no family for {board}, skipping', file=sys.stderr) + continue + try: + data = json.load(open(f)) + except (OSError, ValueError) as e: + print(f'pair_compare: unreadable {f} ({e}), skipping', file=sys.stderr) + continue + for ex, ent in data.items(): + pairs.setdefault((fam, ex), []).extend(ent.get('files', [])) + return pairs + + +def main(): + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument('--base-dir', required=True) + ap.add_argument('--new-dir', required=True) + ap.add_argument('--out', default='metrics_compare') + a = ap.parse_args() + repo_root = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + + base = collect(a.base_dir, repo_root) + new = collect(a.new_dir, repo_root) + common = sorted(set(base) & set(new)) + dropped = sorted(set(base) ^ set(new)) + + if not common: + with open(a.out + '.md', 'w') as f: + f.write('_Code-size comparison skipped: no (family, example) pair was built ' + 'on both the base branch and this PR._\n') + return + + def synth(pairs, path): + with open(path, 'w') as f: + json.dump({'files': [e for k in common for e in pairs[k]]}, f) + + with tempfile.TemporaryDirectory() as td: + b, n = os.path.join(td, 'base.json'), os.path.join(td, 'new.json') + synth(base, b) + synth(new, n) + comparison = metrics.compare_files(b, n, ['tinyusb/src']) + if comparison is None: + with open(a.out + '.md', 'w') as f: + f.write('_Code-size comparison failed to produce data._\n') + return + metrics.write_compare_markdown(comparison, a.out + '.md', 'name+') + + with open(a.out + '.md', 'a') as f: + fams = sorted({k[0] for k in common}) + f.write(f'\n_Scoped compare: {len(common)} (family, example) pairs across ' + f'{", ".join(fams)}._\n') + if dropped: + f.write('_Not compared (missing on one side): ' + + ', '.join(f'{fam}:{ex}' for fam, ex in dropped) + '._\n') + + +if __name__ == '__main__': + main() +``` + +- [ ] **Step 4: Run tests** + +Run: `python3 test/hil/test/test_ci_metrics.py -v` — all pass. + +- [ ] **Step 5: Commit** + +```bash +git add .github/scripts/metrics_pair_compare.py test/hil/test/test_ci_metrics.py +git commit -m "ci: add (family, example)-intersection code-size compare for scoped PRs" +``` + +--- + +### Task 10: GitHub Actions wiring (`build.yml` + `build_util.yml`) + +**Files:** +- Modify: `.github/workflows/build.yml`, `.github/workflows/build_util.yml` + +**Interfaces:** +- `set-matrix` new outputs: `example_map` (JSON `{family: [example]}`), `build_filtered` (`'true'`/`'false'`), `build_families_regex` (`fam1|fam2`, only when filtered). +- `build_util.yml` new input `example-map` (string, default `''`); when set, each leg resolves `-e` flags for its `matrix.arg` family and appends them (via env `$EX_ARGS`) to the Build and Membrowse invocations; metrics upload also grabs `metrics_by_example.json`. +- `code-metrics` gains `needs: set-matrix` and a scoped-baseline path. + +- [ ] **Step 1: Rename + thread the selection in `set-matrix`** + +Rename the step `HIL selection (PR only)` → `CI selection (PR only)` (id stays `hil-select`; renaming the id would touch every `steps.hil-select` reference — leave it). In the **Generate matrix json** step, replace the first three lines of the script (`MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py)` and the two echo lines) with: + +```bash + # Build matrix, scoped by the PR selection when one exists. Best-effort: + # ci_set_matrix falls back to the full matrix itself on unusable JSON, + # and an empty $SELECT (non-PR event, selector fallback) means no flags. + if [ -n "$SELECT" ]; then + MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py --select "$SELECT") || MATRIX_JSON='' + else + MATRIX_JSON='' + fi + [ -z "$MATRIX_JSON" ] && MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py) + echo "matrix=$MATRIX_JSON" + echo "matrix=$MATRIX_JSON" >> $GITHUB_OUTPUT + + # Build-axis extras: the per-family example map rides as a side channel + # (a value inside matrix entries would break CircleCI's family parameter + # and multiply GHA matrix legs). NOTE jq's // treats false like null, so + # .build.full is compared explicitly. + EXAMPLE_MAP=$(printf '%s' "${SELECT:-null}" | jq -c '.build.family_examples // {}') || EXAMPLE_MAP='{}' + BUILD_FILTERED=$(printf '%s' "${SELECT:-null}" | jq -r 'if (.build? | type) == "object" and .build.full == false then "true" else "false" end') || BUILD_FILTERED='false' + FAM_REGEX='' + if [ "$BUILD_FILTERED" = "true" ]; then + FAM_REGEX=$(printf '%s' "$SELECT" | jq -r '.build.families | join("|")') || FAM_REGEX='' + [ -z "$FAM_REGEX" ] && BUILD_FILTERED='false' + fi + echo "example_map=$EXAMPLE_MAP" >> $GITHUB_OUTPUT + echo "build_filtered=$BUILD_FILTERED" >> $GITHUB_OUTPUT + echo "build_families_regex=$FAM_REGEX" >> $GITHUB_OUTPUT +``` + +Add to the `set-matrix` job `outputs:` block: + +```yaml + example_map: ${{ steps.set-matrix-json.outputs.example_map }} + build_filtered: ${{ steps.set-matrix-json.outputs.build_filtered }} + build_families_regex: ${{ steps.set-matrix-json.outputs.build_families_regex }} +``` + +- [ ] **Step 2: `build_util.yml` — example-map input** + +Add the input: + +```yaml + example-map: + required: false + default: '' + type: string +``` + +Insert between **Get Dependencies** and **Build**: + +```yaml + - name: Resolve PR example filter + if: inputs.example-map != '' && inputs.example-map != '{}' + env: + # values are PR-derived - keep them out of ${{ }} script interpolation + # (env expansion word-splits but never re-parses shell metacharacters) + EXAMPLE_MAP: ${{ inputs.example-map }} + FAMILY: ${{ matrix.arg }} + run: | + # -e flags for this family; a family absent from the map builds everything + EX_ARGS=$(printf '%s' "$EXAMPLE_MAP" | jq -r --arg fam "$FAMILY" '(.[$fam] // []) | map("-e " + .) | join(" ")') || EX_ARGS='' + echo "EX_ARGS=$EX_ARGS" + echo "EX_ARGS=$EX_ARGS" >> $GITHUB_ENV +``` + +Append `$EX_ARGS` to all three `tools/build.py` invocations (the esp-idf docker line, the generic Build line, and the Membrowse line — build.py maps `examples-membrowse-upload` per example when `-e` is active, because the aggregate target rebuilds everything). Extend the metrics upload: + +```yaml + path: | + cmake-build/cmake-build-*/metrics.json + cmake-build/cmake-build-*/metrics_by_example.json +``` + +- [ ] **Step 3: `cmake` job passes the map** + +In the `cmake` job's `with:` block add `example-map: ${{ needs.set-matrix.outputs.example_map }}`. Do **not** add it to `hil-build`/`hil-build-esp`/`build-os` — hil legs carry `-e` inside their matrix entries; build-os keeps the full example set. + +- [ ] **Step 4: `code-metrics` scoped baseline** + +Verify the download action supports regexp names: +`curl -fsSL https://raw.githubusercontent.com/dawidd6/action-download-artifact/v11/action.yml | grep -n name_is_regexp` — expect a hit. (Fallback if absent: replace the download step below with a `gh run download`-based loop over `build_families_regex` split on `|`, using `gh api` to find the newest master run per artifact; keep the same directory layout.) + +Change `needs: [ check-paths, cmake ]` → `needs: [ check-paths, cmake, set-matrix ]`. Guard the two unscoped steps with the filtered flag: on **Download Base Branch Metrics** change the `if:` to + +```yaml + if: (github.event_name == 'pull_request' || github.event_name == 'workflow_dispatch') && needs.set-matrix.outputs.build_filtered != 'true' +``` + +and on **Compare with Base Branch** change `if: github.event_name != 'push'` to + +```yaml + if: github.event_name != 'push' && needs.set-matrix.outputs.build_filtered != 'true' +``` + +Insert after **Download Base Branch Metrics**: + +```yaml + - name: Download base per-family metrics (scoped PR) + if: github.event_name == 'pull_request' && needs.set-matrix.outputs.build_filtered == 'true' + uses: dawidd6/action-download-artifact@v11 + with: + workflow: build.yml + workflow_conclusion: '' + search_artifacts: true # a docs-only master push uploads no per-family artifacts + branch: ${{ github.base_ref }} + name: ^metrics-(${{ needs.set-matrix.outputs.build_families_regex }})$ + name_is_regexp: true + path: base-family-metrics + continue-on-error: true + + - name: Compare with Base Branch (scoped) + if: github.event_name == 'pull_request' && needs.set-matrix.outputs.build_filtered == 'true' + run: | + # never fall back to the averaged metrics-tinyusb here: a scoped PR vs the + # 64-family/46-example average is exactly the mismatch this path prevents + python .github/scripts/metrics_pair_compare.py \ + --base-dir base-family-metrics --new-dir cmake-build --out metrics_compare + cat metrics_compare.md +``` + +(The PR-side `cmake-build/` dir already holds this run's `metrics_by_example.json` files from the artifact download at the top of the job.) + +- [ ] **Step 5: Validate and commit** + +```bash +python3 -c "import yaml,sys; yaml.safe_load(open('.github/workflows/build.yml')); yaml.safe_load(open('.github/workflows/build_util.yml')); print('yaml ok')" +command -v actionlint >/dev/null && actionlint .github/workflows/build.yml .github/workflows/build_util.yml || true +git add .github/workflows/build.yml .github/workflows/build_util.yml +git commit -m "ci: scope the GHA build matrix and code-metrics baseline by PR selection" +``` + +--- + +### Task 11: CircleCI wiring + +**Files:** +- Modify: `.circleci/config.yml`, `.circleci/config2.yml` + +**Interfaces:** +- `config.yml` set-matrix: on PRs, runs the selector (gated on its own unit suite), scopes `MATRIX_JSON` via `--select`, skips empty toolchains, and forwards `example-map` + `build-filtered` to the continued workflow as pipeline parameters. +- `config2.yml`: declares those parameters; the `build` command resolves `-e` flags per family; `code-metrics` compare is bypassed with a note when filtered; a `no-op` job keeps the workflow valid when nothing is selected. + +- [ ] **Step 1: Verify the continuation orb accepts parameters** + +`curl -fsSL "https://circleci.com/developer/orbs/orb/circleci/continuation" | grep -io 'parameters' | head -1` — the `continuation/continue` command takes a `parameters` input (inline JSON or a file path). If the page is unreachable, proceed — the orb has carried this input since 0.2; the fallback is `parameters: '{"example-map": ...}'` inline via an env-composed string. + +- [ ] **Step 2: `config.yml` — selector + scoping + parameters** + +In the `Set matrix` run command, replace the first two lines (`MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py)` and its echo) with: + +```bash + # PR-scoped selection (best-effort: any failure falls back to the full + # matrix). CircleCI has no base-branch var; tinyusb PRs target master. + SELECT_JSON='' + if [ -n "${CIRCLE_PULL_REQUEST:-}" ]; then + git fetch --no-tags origin master || true + if python3 test/hil/test/test_ci_select.py >/dev/null 2>&1; then + SELECT_JSON=$(python3 tools/ci_select.py --base origin/master) || SELECT_JSON='' + else + echo "ci_select unit suite failed - using the full matrix" + fi + fi + MATRIX_JSON='' + if [ -n "$SELECT_JSON" ]; then + MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py --select "$SELECT_JSON") || MATRIX_JSON='' + fi + [ -z "$MATRIX_JSON" ] && MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py) + echo "MATRIX_JSON=$MATRIX_JSON" + + EXAMPLE_MAP=$(printf '%s' "${SELECT_JSON:-null}" | jq -c '.build.family_examples // {}') || EXAMPLE_MAP='{}' + BUILD_FILTERED=$(printf '%s' "${SELECT_JSON:-null}" | jq -r 'if (.build? | type) == "object" and .build.full == false then "true" else "false" end') || BUILD_FILTERED='false' + jq -n --arg map "$EXAMPLE_MAP" --arg filt "$BUILD_FILTERED" \ + '{"example-map": $map, "build-filtered": $filt}' > /tmp/continue_params.json +``` + +In the toolchain loop, after `FAMILY=$(echo $MATRIX_JSON | jq -r ".\"$toolchain\"")` add: + +```bash + if [ "$(echo "$FAMILY" | jq 'length')" = "0" ]; then + # an empty matrix parameter is a hard CircleCI config error, not a skip + echo "skip build-${build_system}-${toolchain}: no families selected" + continue + fi +``` + +(the `continue` also keeps the alias out of `BUILD_ALIASES`, so `code-metrics` never requires a job that was not generated). Guard the code-metrics emission and keep the workflow non-empty: + +```bash + if [ ${#BUILD_ALIASES[@]} -gt 0 ]; then + echo " - code-metrics:" >> .circleci/config2.yml + echo " requires:" >> .circleci/config2.yml + for alias in "${BUILD_ALIASES[@]}"; do + echo " - $alias" >> .circleci/config2.yml + done + else + # a workflow with zero jobs is invalid config + echo " - no-op" >> .circleci/config2.yml + fi +``` + +(replacing the current unconditional code-metrics block). Change the continuation call to: + +```yaml + - continuation/continue: + configuration_path: .circleci/config2.yml + parameters: /tmp/continue_params.json +``` + +- [ ] **Step 3: `config2.yml` — parameters, `-e` resolution, scoped-compare note, no-op job** + +At the top, after `version: 2.1`: + +```yaml +parameters: + example-map: + type: string + default: "{}" + build-filtered: + type: string + default: "false" +``` + +In the `build` command's **Build** step, before the toolchain if/else, insert: + +```bash + # PR example filter for this family ('{}' or a missing key = build all). + # The parameter is a JSON string composed by set-matrix from ci_select. + EX_ARGS=$(printf '%s' '<< pipeline.parameters.example-map >>' | jq -r --arg fam "<< parameters.family >>" '(.[$fam] // []) | map("-e " + .) | join(" ")' 2>/dev/null) || EX_ARGS='' +``` + +and append `$EX_ARGS` to both `tools/build.py` invocations (docker esp-idf and the generic one). In `code-metrics`, wrap the existing compare `when:` condition with the filter guard and add the note branch: + +```yaml + - when: + condition: + and: + - not: + equal: [ master, << pipeline.git.branch >> ] + - equal: [ "false", << pipeline.parameters.build-filtered >> ] + steps: + # ... the existing Download Base Branch Metrics + Compare + store_artifacts steps, unchanged ... + - when: + condition: + and: + - not: + equal: [ master, << pipeline.git.branch >> ] + - equal: [ "true", << pipeline.parameters.build-filtered >> ] + steps: + - run: + name: Scoped build - comparison unavailable + command: | + # CircleCI stores only the averaged metrics.json; the per-example + # baseline lives on GHA. See the GHA code-metrics PR comment. + echo "_Code-size comparison skipped on CircleCI: this PR built a scoped example set._" > metrics_compare.md + - store_artifacts: + path: metrics_compare.md + destination: metrics_compare.md +``` + +Add the no-op job beside the other job definitions: + +```yaml + no-op: + docker: + - image: cimg/base:current + resource_class: small + steps: + - run: + name: No families selected + command: echo "PR selection - no families to build on CircleCI" +``` + +- [ ] **Step 4: Validate and commit** + +```bash +python3 -c "import yaml; yaml.safe_load(open('.circleci/config.yml')); yaml.safe_load(open('.circleci/config2.yml')); print('yaml ok')" +command -v circleci >/dev/null && circleci config validate .circleci/config.yml || true +git add .circleci/config.yml .circleci/config2.yml +git commit -m "ci: scope the CircleCI build matrix and example set by PR selection" +``` + +--- + +### Task 12: End-to-end validation, review, hand-off + +**Files:** none new — verification only (fix-ups amend the relevant earlier area). + +- [ ] **Step 1: Full hooks + suites** + +```bash +pre-commit run --all-files # ~55 s; HIL hooks exercise real timeouts deliberately +``` + +Expected: all hooks pass (`ci-select-test` and `hil-test` among them). + +- [ ] **Step 2: Selector scenario table** + +```bash +for f in src/portable/raspberrypi/rp2040/dcd_rp2040.c src/class/cdc/cdc_device.c \ + src/host/usbh.c examples/device/cdc_msc/src/main.c test/hil/hil_test.py \ + src/common/tusb_fifo.c hw/mcu/nordic/nrf5x/x.h; do + echo "== $f" + python3 tools/ci_select.py --diff-file <(echo "$f") test/hil/tinyusb.json 2>/dev/null | \ + python3 -c "import json,sys; s=json.load(sys.stdin); b=s['build']; print('hil_full:', s['full'], ' build_full:', b['full'], ' fams:', len(b['families']), ' mapped:', len(b['family_examples']))" +done +``` + +Expected (spot-check against the spec's measured table): rp2040 → 1 family; cdc_device → all families, mapped lists; usbh → ~25 families; example → all families, 1-example lists; test/hil → 0 families, hil_full true; common → build_full true; hw/mcu → 1 family (`nrf`). + +- [ ] **Step 3: Matrix + build smoke** + +```bash +SEL=$(python3 tools/ci_select.py --diff-file <(echo src/portable/raspberrypi/rp2040/dcd_rp2040.c) test/hil/tinyusb.json 2>/dev/null) +python3 .github/scripts/ci_set_matrix.py --select "$SEL" | python3 -m json.tool | head +python3 .github/scripts/hil_ci_set_matrix.py --select "$SEL" test/hil/tinyusb.json | python3 -m json.tool | head +python3 tools/build.py -e device/cdc_msc -b stm32f407disco --target all --target tinyusb_metrics +python3 -c "import json; d=json.load(open('cmake-build/cmake-build-stm32f407disco/metrics_by_example.json')); print(sorted(d))" +``` + +Expected: matrix shows only rp2040 under arm-gcc; hil matrix entries carry `-e ... -e device/board_test`; the by-example JSON lists exactly `['device/cdc_msc']`. + +- [ ] **Step 4: Full example set for one board** (repo validation rule after tool changes) + +```bash +cd examples && cmake -B cmake-build-stm32f407disco -DBOARD=stm32f407disco -G Ninja -DCMAKE_BUILD_TYPE=MinSizeRel . \ + && cmake --build cmake-build-stm32f407disco && cd .. +``` + +Expected: builds green (objcopy warnings non-critical per CLAUDE.md). + +- [ ] **Step 5: Local review, then stop** + +Run the `/code-review` skill on the branch diff (user policy: every push carrying local changes gets a local review pass first) and fix what holds up, amending into the appropriate task commits. Then **stop and hand back to the user** — pushing `build-filter` and opening the PR is their call; note for the PR description that the workflow changes only fully prove out on a real PR run (first PR after merge-to-branch should be watched with `gh pr checks --watch`, and the `hil-select` step's warnings checked for silent fallbacks). + +--- + +## Self-Review Notes + +- Spec coverage: rule table (Tasks 2-4), CMake-only scan (Task 2), orphan invariant (Task 2), build/hil_examples JSON contract (Task 4), `ci_set_matrix` flags (Task 5), `hil_ci_set_matrix -e` (Task 6), `build.py -e` incl. membrowse aggregate-dependency workaround (Task 7), metrics by-example + intersection compare + never-fall-back rule (Tasks 8-10), GHA side channel + injection-safe env passing (Task 10), CircleCI empty-toolchain/alias/no-op fixes + parameters (Task 11), move fallout table (Task 1). +- Known deviation from the spec text, both directions justified inline: `hil_examples` uses the *narrowed* chosen test list when a board is narrowed (the spec's JSON example implies this; its prose says `board_tests` — the narrowed form is a strict subset and matches what the rig runs, and re-run specs are subsets of it). +- Spec's measured "hcd_max3421.c → 1 leg" is really 1 *bsp* family (`espressif`) that neither provider's family list builds → 0 CI legs; Task 3's rule-4 test therefore asserts shape, not that specific count. diff --git a/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md b/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md new file mode 100644 index 000000000..9fa358bee --- /dev/null +++ b/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md @@ -0,0 +1,501 @@ +# PR-scoped CI selection: promoting hil_select to tools/ci_select.py + +**Date:** 2026-08-19 +**Branch:** `build-filter` + +## Motivation + +Every PR builds every example on one board per family, on both CI providers: **74 legs / +2494 example-builds** on the GitHub Actions `cmake` job, and 129 family-legs per build system +on CircleCI (which runs cmake *and* make, plus clang/IAR). Most PRs touch one port, one class, +or one example, and a `hid_host.c` change cannot break an MSC device example on msp430. + +`hil-build` is worse in a different way: it builds **1702 example-builds** (37 board-builds × +46 examples, `--target all`) to run a test suite that needs at most **515**. The HIL example +universe is only 21 of the 46 examples in tree, and the median board needs 15 of them. + +`test/hil/helper/hil_select.py` already maps a PR diff to affected boards and per-board test +lists for HIL, and already owns both mappings the build matrix needs: port-to-family, and +class-macro-to-example +(`docs/superpowers/specs/2026-07-29-hil-pr-scoped-selection-design.md`). That design listed +"scoping the non-HIL build jobs" as an explicit non-goal; this is that follow-up. + +## Goal / non-goals + +**Goal:** promote the selector to a repo-wide `tools/ci_select.py` whose single classification +of a diff drives **all three** CI axes from one rule table — which families to build, which +example targets to build on each, and which rig boards run which tests — wired into +`ci_set_matrix.py` and `hil_ci_set_matrix.py` so both providers and the rig filter from one +source. Scoping applies to `pull_request` events only; push, release and `workflow_dispatch` +keep the full matrix. + +**Non-goals:** +- Variant-level or board-level selection below one-board-per-family on the build axis (all + variants of a selected HIL board still build and run). +- Changing `hil_test.py` behaviour. The selector only *composes* existing `-b` / `-bt` args. +- Changing which tests HIL decides to run. The HIL board/test decision is preserved except for + the single rule-7 change called out below. + +## The rule table + +One classification, three outputs. Every rule yields build families, build examples, and HIL +boards/tests. Pairs are unioned **per family** (build) and **per board** (HIL), so a mixed diff +never inflates one axis with another's breadth. + +`DEV` = 33 `examples/device/*`, `HOST` = 9, `DUAL` = 3, `TYPEC` = 1, `ALL` = 46. +`FAM` = the families whose `family.cmake` references the changed path (CMake only — see below). +"roster boards" = boards on `test/hil/{tinyusb,hfp}.json`. + +| # | Changed path | Build families | Build examples | HIL boards → tests | +| --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | +| 1 | `docs/`, `.claude/`, `*.md`, `*.rst`, `LICENSE` | — | — | — | +| 2 | `test/hil/**` | — | — | all boards → all tests | +| 2b | `tools/metrics.py`, `.github/scripts/metrics_*.py` | `ALL` (unchanged — `tinyusb_metrics` runs `metrics.py` as a build target) | `ALL` | — (nothing on the rig runs it) | +| 3 | `src/portable//dcd_*`, `*_device.[ch]` | `FAM` | `DEV`+`DUAL` | `FAM`'s device-role boards → device+dual tests | +| 4 | `src/portable//hcd_*`, `*_host.[ch]` | `FAM` | `HOST`+`DUAL` | `FAM`'s host-role boards → host+dual tests | +| 5 | `src/portable//**` (anything else) | `FAM` | `ALL` | `FAM`'s boards → all their tests | +| 5b | `src/portable//**` where `FAM` is empty | — | — | — (empty resolves to nothing on BOTH axes) | +| 6 | `hw/bsp//**` | that family | `ALL` | that family's boards → all tests (a `boards//` path narrows to that board) | +| 7 | `hw/mcu//**` | `FAM` — empty resolves to nothing (maintainer ruling) | `ALL` | `FAM`'s boards → all tests; empty resolves to nothing (maintainer ruling) ⚠ *see below* | +| 8 | `src/class//*_device.[ch]` | `ALL` | examples enabling `CFG_TUD_` | device-role boards → HIL tests enabling `CFG_TUD_` | +| 9 | `src/class//*_host.[ch]` | `ALL` | examples enabling `CFG_TUH_` | host-role boards → HIL tests enabling `CFG_TUH_` | +| 10 | `src/class//**` (shared header) | `ALL` | either, **plus include-edge classes** | both roles → same, plus include-edge classes | +| 11 | `src/device/**` | `ALL` | `DEV`+`DUAL` | device-role boards → device+dual tests | +| 12 | `src/host/**` | `ALL` | `HOST`+`DUAL` | host-role boards → host+dual tests | +| 13 | `examples///**` | `ALL` | just `` | if `` is a HIL test: all boards → that test; else nothing | +| 14 | `examples/device/board_test/**` | `ALL` | just `board_test` | all boards → all tests (HIL parking firmware) | +| 15 | `examples/build_system/**`, `examples/CMakeLists.txt`, `examples//CMakeLists.txt` | `ALL` | `ALL` | all boards → all tests | +| 16 | `src/common/`, `src/osal/`, `src/tusb.[ch]`, `src/tusb_option.h`, `tools/build*.py`, `tools/cmake/**`, `hw/bsp/{family_support.cmake,board.c,board_api.h,ansi_escape.h}`, `.github/**` | `ALL` | `ALL` | all boards → all tests | +| 16a | `lib//**` | `ALL` | examples whose own `CMakeLists.txt`/`Makefile` names `lib/` | those examples that are HIL tests, on all boards; empty resolves to nothing | +| 16b | `tools/get_deps.py` | families whose `deps_mandatory`/`deps_optional` entries changed | `ALL` | those families' boards → all tests; a logic change, an `'all'` entry, no base content or a changed token naming no family → full | +| 17 | anything unclassified | `ALL` | `ALL` | all boards → all tests (fail-open) | + +**Rule 2 is deliberately asymmetric.** A `test/hil/**` change is invisible to the family matrix +but is exactly what the rig exercises, so it builds nothing and runs everything. + +**Rule 7 is the one HIL-side behaviour change in this design.** Today `hw/mcu/` sits in +`hil_select`'s `_FULL_RE` and forces the full HIL matrix. Since the build axis now resolves +those paths to a family through the same scan, forcing full on the rig is inconsistent. The +path fires rarely — 4 commits in 3 years — so this is low-risk either way; if you would rather +keep the HIL view untouched, rule 7's HIL column becomes "all boards → all tests" and nothing +else in this design changes. + +Rules 8–10 reuse machinery `hil_select` already has — `class_macros`, `_config_enables`, +`class_include_edges` — applied over all 46 examples' `src/tusb_config.h` for the build axis +and over the HIL test list for the HIL axis. The include edges are why an `audio.h` change also +selects the MIDI examples (`midi{,2}_{device,host}.h` include `class/audio/audio.h`) and a +`cdc.h` change the net one. + +### Buildability post-filter (build axis) + +After the pairs are unioned, every `(family, examples)` pair is pruned with +`build_utils.skip_example(example, )` — the same `skip.txt` / `only.txt` +data CMake's `family_filter` uses (40 `skip.txt`, 13 `only.txt` in tree). Examples the family +cannot build are dropped; a family left with none is dropped entirely. + +This is where most of the host-side saving comes from: only 23 of 75 CI families can build +`host/bare_api` at all, and 2 can build `typec/power_delivery`. + +### Measured effect + +GHA `cmake` job, baseline **74 legs / 2494 example-builds**; `hil-build`, baseline **1702 +example-builds** across 37 board-builds. + +| PR shape | Build legs | Build ex-builds | HIL boards | hil-build ex-builds | +| ---------------------------- | ---------: | --------------: | ---------: | ------------------: | +| `dcd_rp2040.c` | 1 | 35 | 2 | 32 | +| `hcd_max3421.c` | 1 | 10 | 7 | 36 | +| `hw/bsp/stm32f4/**` | 1 | 45 | 1 | 15 | +| `dcd_dwc2.c` | 20 | 646 | 10 | 184 | +| `hid_host.c` | 24 | 68 | — | — | +| `msc_host.c` | — | — | 9 | 40 | +| `usbh.c` | 25 | 217 | 10 | 57 | +| `msc_device.c` | 74 | 350 | — | — | +| `cdc_device.c` | 74 | 588 | 27 | 192 | +| `examples/device/cdc_msc/**` | 73 | 73 | 25 | 60 | +| `usbd.c` | 74 | 2297 | 27 | 472 | +| `src/common/**` (full) | 74 | 2494 | 30 | 515 | +| `test/hil/**` only | 0 | 0 | 30 | 515 | + +The full-matrix row is the headline for `hil-build`: even with **no** PR narrowing, per-board +example selection takes it from 1702 to 515. + +### Why "empty means empty" + +Both views answer an empty `FAM` the same way (rule 5b): nothing. The HIL view used to force +the full 30-board rig there, on the theory that an empty result might be a scan miss — but the +build view answered the identical condition with zero families for the same path, so the rig +ran every board to validate a file that nothing compiled. In the build view that theory costs +74 legs, and the +evidence does not support it: of the 28 `src/portable/*/*` directories, **26 resolve to at +least one family**. The two that do not are both real orphans as far as CI is concerned: +`microchip/pic` (only `dcd_pic.c` and a README, with no `hw/bsp/pic` family at all) and +`microchip/pic32mz` (`hw/bsp/pic32mz` has only a `family.mk`, and `pic32mz` is in neither +provider's family list, so no CI job builds it today). A file no CI job compiles cannot be +validated by building anything. + +The safety this gives up is recovered structurally: a unit test asserts every +`src/portable/*/*` and every tracked `hw/mcu/` resolves to ≥1 family, with an explicit +allowlist of known orphans (`microchip/pic`, `microchip/pic32mz`). Adding a port without +wiring a family then fails +pre-commit instead of silently building nothing on every later PR. Same enforcement style as +the existing `test_hil_util.BottomLayer` structural tests. + +Fail-open survives where it belongs: an *unclassified* path or any exception widens to `ALL` on +every axis. + +### Why `hw/mcu/**` is rule 7 and not "full" + +`hw/mcu` is overwhelmingly dependency territory — `tools/get_deps.py` has 87 entries under it, +and those paths are gitignored, so they can never appear in a diff. Only 51 files survive +in-tree, touched 4 times in 3 years, and they resolve through the same scan the ports use: + +| Tracked directory | In `get_deps`? | Resolves to | +| ----------------------------- | ------------------------------------------- | ----------- | +| `hw/mcu/dialog/` (`da1469x`) | **no** — real in-repo MCU support, 21 files | `da1469x` | +| `hw/mcu/nordic/` (`nrf5x`) | beside the `nrfx` dep | `nrf` | +| `hw/mcu/sony/` (`cxd56`) | beside the `spresense-exported-sdk` dep | `cxd56` | +| `hw/mcu/bridgetek/` (`ft9xx`) | beside the `ft90x-sdk` dep | `ft9xx` | + +Rule 7 is therefore not a mechanism of its own — it is rules 3–5's scan pointed at a second +tree, because `src/portable/` and `hw/mcu/` ask the same question. + +Unlike the port rule, an `hw/mcu` path that resolves to no family contributes *nothing* on +either axis (maintainer ruling): if no family's build references it, no build compiles it. The +table above is kept honest by `test_tracked_mcu_vendors_resolve`, which fails pre-commit if a +tracked vendor directory stops resolving. + +### Why `lib/**` is rule 16a and scanned per example + +Its tracked contents (`SEGGER_RTT`, `networking`, `rt-thread`, `embedded-cli`, 22 commits in +3 years) are wired in at `examples/build_system` and per-example `CMakeLists.txt`, not per +family — so the family scan the ports use is the wrong instrument here: it would *wrongly* +narrow `SEGGER_RTT` to the three families that name the path in their `family.cmake`, while +the path is not compiled by any of them by default (it is reached only through `LOGGER=rtt`). +That scan stays applied to `src/portable/` and `hw/mcu/` only. + +Rule 16a asks the per-example question instead (maintainer ruling: only the examples that use +the lib need building): `lib_examples()` reads each example's own `CMakeLists.txt` and +`Makefile` and keeps the ones naming `lib/` at a directory boundary. Every family stays +in play — any of them can build those examples — while the example list collapses: + +| Tracked lib | Examples that build it | HIL tests among them | +| -------------- | ----------------------------------------------------------- | -------------------- | +| `embedded-cli` | `host/msc_file_explorer`, `host/msc_file_explorer_freertos` | both | +| `networking` | `device/net_lwip_webserver` | none (test disabled) | +| `SEGGER_RTT` | — | — | +| `rt-thread` | — | — | + +`SEGGER_RTT` and `rt-thread` resolve to nothing, and "empty means empty" applies: no CI build +compiles them, so there is nothing to validate by building. + +### Why `tools/get_deps.py` is rule 16b + +`deps_mandatory` / `deps_optional` are data: `path -> [url, commit, 'fam1 fam2 ...']`. A commit +bump therefore affects exactly the families listed in that entry, and building the other 70+ is +pure waste. `get_deps_changed_families()` parses both sides of the file with `ast` (never +`exec` — this is PR content), diffs the two dict literals **separately**, and unions the family +tokens of every added, removed or edited entry, from **both** sides (a removed entry has only a +base side; an edited family list must cover the families that lose the dep as well as the ones +that gain it). Separately, because merging the dicts before diffing hides a *move* between +`deps_mandatory` and `deps_optional` — the value is untouched, but mandatory deps are fetched +for every family, so demoting one stops families fetching it. + +It falls open to the full matrix whenever the entries are not the whole answer: + +* anything outside the two dict assignments differs — a logic change to `get_deps` can change + what every family fetches (compared as `ast.dump(..., annotate_fields=False)` of the module + with those two assignments removed, so comments and reformatting alone are not a logic + change); +* an `'all'` entry (every mandatory dep) changed; +* the file will not parse; +* there is no base content: `--diff-file` mode has no git, so no merge-base blob; +* a changed entry carries a family token that names no `hw/bsp/` and is not one of the + eight known aliases. "Changed but unmappable" is not "nothing changed": reading it as the + latter empties the whole build matrix for a dep bump. + +The eight known aliases (`sam3x`, `samd21`, `samd51`, `same5x`, `fc100s`, `spresense`, +`stm32l1`, `stm32l5`) are pinned in `_DEPS_ALIAS_TOKENS` and select nothing. `get_deps` matches +a token against a requested family name verbatim (`f in entry[2].split()`), so these tokens +match nothing there either — four are pre-rename spellings listed beside the current name in +the same entry, two point at a differently-named family dir (`fc100s`→`f1c100s`, +`spresense`→`cxd56`, both unreachable in `get_deps` itself), and two name no family in the tree. +A ninth appearing fails `TestOrphanInvariant`. + +## Component: `tools/ci_select.py` + +`git mv test/hil/helper/hil_select.py tools/ci_select.py` (history preserved). The HIL +classifier is unchanged apart from rule 7; a second, independent build classifier is added +beside it. One diff read, two classifiers, one unit suite. + +``` +python3 tools/ci_select.py --base [--diff-file ] [CONFIG.json ...] +``` + +`configs` becomes `nargs='*'`. With rosters it emits everything it emits today plus the new +keys; with none it emits only the build view, so CircleCI never needs to know HIL exists. + +```json +{ + "full": false, + "boards": {"raspberry_pi_pico": "all"}, + "families": ["rp2040"], + "args": {"tinyusb.json": "-b raspberry_pi_pico"}, + "args_flasher": {"tinyusb.json": {"openocd": "-b raspberry_pi_pico"}}, + "hil_examples": {"raspberry_pi_pico": ["device/cdc_msc", "device/board_test"]}, + "build": { + "full": false, + "families": ["rp2040"], + "family_examples": { + "rp2040": ["device/cdc_msc", "device/hid_composite", "dual/dynamic_switch"] + } + }, + "reasons": ["src/portable/raspberrypi/rp2040/dcd_rp2040.c: port rp2040 -> ..."] +} +``` + +`build.families` is the build family axis. `build.family_examples` maps a family to its example +list; **a family absent from the map builds all its examples**, so the common "narrow families, +all examples" case carries no payload. `build.full` true means no build narrowing at all. + +`hil_examples` is the new HIL build axis: per roster board, the examples `hil-build` must +produce. It is `board_tests(board)` — which the selector already computes — **plus +`device/board_test`**, which `hil_test.py` flashes to park every board at each variant boundary +and at end-of-board teardown (`hil_test.py:1798`, `:1866`). It is emitted even when +`full: true`, because the HIL example universe is 21 of 46 examples regardless of any diff. + +All pre-existing keys keep their exact meaning, so `.github/scripts/hil_ci_set_matrix.py`, the +HIL legs in `build.yml` and `.claude/skills/pre-pr/SKILL.md` need only a path update. + +### Shared helper change + +`port_families(port_dir, repo_root)` generalizes to a path-to-families reference scan over a +second tree (`hw/mcu/`). Its existing **CMake-only** behaviour is kept unchanged and is now the +rule for every axis: it scans `hw/bsp/*/family.cmake` plus the espressif component +`CMakeLists.txt`, and never `family.mk`. + +CMake is the first-class build system; Make follows whatever CMake decides. A family that +CMake does not wire up to a port is not a consumer of that port, and the Make legs on CircleCI +build the same families CMake does. Scanning `family.mk` as well would only ever *widen* the +selection to families CMake never builds, which is coverage nobody asked for — and it would +resolve `microchip/pic32mz` to a family that appears in no CI family list. + +One consequence to keep in view: because HIL and build now share one scan, there is no +per-caller flag, no second cache key, and no way for the two axes to disagree about which +families own a port. + +**Boundary matching.** A directory reference must match at a directory boundary — a trailing +`/` *or* end-of-token — not as a bare substring. Both traps are live: `hw/bsp/nrf/family.cmake` +writes `${TOP}/hw/mcu/nordic/nrf5x` with no trailing slash, while the existing port scan +requires a trailing `/` precisely to stop `microchip/pic` matching `microchip/pic32mz`. + +### Move fallout + +All mechanical, all one-line: + +| File | Change | +| ---------------------------------- | ----------------------------------------------------------------- | +| `tools/ci_select.py` | `sys.path` walk 4 levels → 2 | +| `test/hil/hil_ci.sh` | drop from the scp list (nothing on the rig imports it) | +| `test/hil/test/test_hil_select.py` | rename to `test_ci_select.py`, import path | +| `test/hil/test/test_hil_util.py` | `BottomLayer` stdlib-closure allowlist + module list | +| `.pre-commit-config.yaml` | both hooks (`hil-select-test` → `ci-select-test`, `files:` globs) | +| `.github/workflows/build.yml` | selector path, step name | +| `.claude/skills/pre-pr/SKILL.md` | selector path | + +The test file stays in `test/hil/test/` — it still consumes the rig rosters and `hil_util`. + +The selector gains one non-stdlib-but-local import: `tools/build_utils.skip_example` for the +buildability post-filter. `build_utils` imports only `subprocess`, `pathlib` and `re`, so the +stdlib closure the bare GitHub runner depends on is preserved; `BottomLayer` must be extended +to cover it. + +Because a wrong parents-count already broke this module once (there is a comment in the source +recording it), the moved module gets a guard test asserting its derived repo root contains +`src/` and `hw/bsp/`. + +## Component: `tools/build.py --example` + +`build.py` has no example filter today. `-T/--target` exists and maps to +`cmake --build --target `, but it hard-fails on a target that does not exist, and absent +targets are routine (40 `skip.txt`, 13 `only.txt`). + +New repeatable `-e/--example /`: + +- Default (none given) keeps today's behaviour exactly: `--target all`. +- Given, each board's list is intersected with `build_utils.skip_example(example, board)`, then + passed as one `--target ` per example. Example target names are the directory names and + are unique across all four roles (verified: 46 examples, zero collisions). +- A board whose intersection is empty is reported **skipped**, not failed. +- `--target tinyusb_metrics` must stay last so metrics run after the examples that feed them. +- The espressif path already builds per example via `get_examples` + `skip_example`; it takes + the same filter. + +Both the family matrix and `hil-build` use this one flag. + +## CI wiring + +### `.github/scripts/ci_set_matrix.py` + +Two mutually exclusive optional flags. **Output shape is unchanged** — `{toolchain: [family]}`, +just fewer families. With no flags the output is byte-for-byte today's, so push, release and +`workflow_dispatch` are untouched. + +| Flag | Caller | Behaviour | +| --------------- | -------- | -------------------------------------------------------- | +| `--select JSON` | GHA | consumes the selector JSON the workflow already computes | +| `--base REF` | CircleCI | runs `tools/ci_select.py` itself | + +`build.full` true, or any exception, prints the full matrix with a warning on stderr. + +### `.github/scripts/hil_ci_set_matrix.py` + +Already takes `--select` and already scopes boards. It additionally appends `-e ` per +board from `hil_examples`, so each `hil-build` entry builds only what its board will run plus +`board_test`. When `hil_examples` is absent (hand-runs), it falls back to today's `--target all`. + +### The example map is a side channel, not a matrix entry + +The build example list deliberately does **not** ride inside the family matrix entry string. On +CircleCI the `family` parameter is also passed to `python tools/get_deps.py +<< parameters.family >>` and tested with `if [ << parameters.family >> == "rp2040" ]` — a value +carrying `-e` flags breaks both — and CircleCI matrix parameters form a cartesian product, so a +parallel `example-args` parameter would multiply the jobs rather than zip with them. + +So `build.family_examples` travels as one JSON blob and each build job resolves its own entry: + +- **GHA:** `set-matrix` exposes it as an output; `build_util.yml` gains an optional + `example-map` input (default `''`); a step resolves `-e` flags for `matrix.arg` with `jq`. +- **CircleCI:** `set-matrix` writes `example_map.json` and `persist_to_workspace`s it; the + `build` job gains `attach_workspace` and resolves the same way. + +Consequences of keeping the matrix shape: the metrics artifact name stays `metrics-`, +and CircleCI's generated `config2.yml` does not inflate to one entry per family. `hil-build` +needs none of this — its matrix entries are already compound per-board strings from +`hil_ci_set_matrix.py`, so `-e` flags go straight in. + +### `.github/workflows/build.yml` + +The existing `HIL selection (PR only)` step in `set-matrix` is already gated on +`pull_request` — exactly the gate wanted. It is renamed, repointed at `tools/ci_select.py`, and +its `select` output is threaded into `ci_set_matrix.py --select`, so the filter costs zero extra +selector invocations. + +`build_util.yml`'s `if: inputs.build-args != '[]'` already skips a toolchain leg whose list is +empty, and a partially-skipped matrix aggregating to success is the documented pattern +`hil-build` already relies on. When every leg is empty (a `test/hil`-only PR), the `cmake` job +has nothing to build. Accepted: GitHub treats a skipped job as satisfying a required status +check, and HIL is unaffected because `hil-build` is a separate matrix. `code-metrics` still +runs (`!cancelled()` plus `cmake` success-or-skipped) and posts a "built no families on this +push" marker, so the sticky size comment never shows a stale table from an earlier push. + +### `.circleci/config.yml` + +The `set-matrix` job passes `--base origin/master` when `CIRCLE_PULL_REQUEST` is set, after +`git fetch --no-tags origin master || true`; unfiltered otherwise. CircleCI does not expose the +PR base branch, so `master` is assumed — true for essentially every tinyusb PR, and any ref or +clone problem falls back to the full matrix. + +Two fixes the GHA side does not need: + +- `gen_build_entry` must **skip** a toolchain whose family list is `[]`. An empty matrix + parameter is a hard CircleCI config error, not a skipped job. +- `BUILD_ALIASES` must collect only aliases that were actually generated, or `code-metrics`' + `requires:` names a job that does not exist. + +## Code metrics + +`tools/metrics.py` averages per-file sizes across every build, and the per-family +`metrics-` artifact stores only that average — over whichever examples were built. Both +build axes therefore break the comparison: a 3-family PR against master's 64-family average, +and an 11-example average against master's 46-example one. + +The fix is to make the artifact carry per-example detail and compare the intersection. + +1. **`metrics.py combine --by-example`** additionally writes `metrics_by_example.json`, + `{example: {files: [...]}}`. The example name is the map.json's parent directory + (`///*.map.json`). +2. `examples/CMakeLists.txt`'s `tinyusb_metrics` target emits both files; `build_util.yml` + uploads both under the existing `metrics-` artifact name. +3. `combine` learns to expand a by-example JSON into one data entry per example and an + `--only-examples` filter, so a subset can be averaged on demand. +4. `code-metrics` computes the **intersection of `(family, example)` pairs present on both + sides**, averages each side over exactly those pairs, and compares. Dropped pairs are named + in the PR comment. An empty intersection skips the compare with an explicit note. + +`search_artifacts: true` is required on the base-side download: a docs-only master push +produces no per-family artifacts — which is why `metrics-carry-forward` exists for the +aggregate — so per-family baselines may come from different master runs. That is still a valid +per-family baseline. + +Today's `metrics-tinyusb` aggregate keeps being produced for the unfiltered path, releases and +`metrics-carry-forward`. The filtered path never falls back to it — that is precisely the +mismatched compare this section exists to prevent. `hil-build` uploads no metrics, so its +narrowing does not touch any of this. + +For narrow PRs this is sharper than today: a `dcd_rp2040` PR's size delta stops being diluted +by a 64-family, 46-example average. + +**Size check the plan must run first:** the by-example JSON is ~46× the entries of today's +average. If it proves too large as an artifact, drop per-symbol detail from the by-example file +(sizes only) — symbols are only needed in the aggregate. The plan must also verify that +`dawidd6/action-download-artifact@v11` supports `name_is_regexp`; the fallback is a +`gh run download` loop. + +## Testing + +Extended in `test/hil/test/test_ci_select.py` (stdlib-only, ~0.1 s, already a pre-commit hook +and already gating CI's selector step): + +- One case per rule 1–17, asserting all three outputs. +- Per-family union: a mixed diff (`dcd_rp2040.c` + `cdc_device.c`) gives `rp2040` the device + list and every other family the CDC list — not the cross product of both. +- Include edges: an `audio.h` change selects the MIDI examples; a `cdc.h` change the net one. +- `hil_examples` always contains `device/board_test` for every selected board, including when + `full: true`, and is otherwise exactly `board_tests(board)`. +- `hil_examples` never exceeds the 21-example HIL universe. +- The scan is CMake-only: a port referenced solely from a `family.mk` (`microchip/pic32mz`) + resolves to no family, and no `family.mk` is ever read. +- Boundary matching: `microchip/pic` does not inherit `microchip/pic32mz`'s families, and + `hw/mcu/nordic/nrf5x` resolves despite having no trailing slash at its reference site. +- Buildability post-filter: `typec/power_delivery` prunes to 2 families, `host/bare_api` to 23. +- Structural invariant: every `src/portable/*/*` and every tracked `hw/mcu/` resolves + to ≥1 family, allowlist `{microchip/pic, microchip/pic32mz}`. +- Every name in `build.families` is a real `hw/bsp/`; every example name on either axis is + a real `examples//` directory. +- Repo-root guard for the moved module. +- `ci_set_matrix.py`: no flags → byte-identical to today; `--select` with `build.full` → + identical; `--select` narrow → a subset; malformed `--select` → full plus a warning. +- `hil_ci_set_matrix.py`: no `hil_examples` → today's args byte-for-byte; with it → `-e` flags + appended per board, `board_test` always present. +- `build.py`: `-e` with an example the board skips builds nothing and reports skipped, not + failed; no `-e` still passes `--target all`. + +## Known gaps + +- **CircleCI size comparison.** CircleCI stores only the combined `metrics.json`, so the + intersection compare is unavailable there; when filtered it prints a note and copies + `metrics.md`. Its `metrics_compare.md` is a stored artifact that nothing reads in review — the + PR comment comes from GHA. Making CircleCI store per-example metrics is a follow-up. +- **HIL re-run attempts.** A re-run spec is a subset of the original selection, so the + firmware `hil-build` produced already covers it. This holds only while re-run specs stay + subsets; a future "re-run with extra tests" feature would need `hil-build` re-run too. +- **Membrowse** receives rows for fewer families and fewer examples on filtered PRs. If that + service misbehaves, the escape hatch is keeping the membrowse upload leg unfiltered. +- **`typec/power_delivery`** is reached only through rules 5 and 13 (`src/portable/st/typec` + has neither a `dcd_` nor an `hcd_` prefix, so it selects `ALL` examples on its 5 families, + which the post-filter then prunes to 2). A dedicated typec rule is possible later; the + post-filter already makes it cheap. +- **A `test/hil`-only PR reports `cmake` as skipped** rather than passing. Accepted; + revertible with a one-family floor if branch protection turns out to disagree. + `code-metrics` still runs in that case: with no `cmake-build/*/metrics.json` to + aggregate it writes `_Code-size comparison skipped: PR selection built no families + on this push._` and posts that as the sticky comment, so the size section reflects + THIS push instead of keeping the previous one's table. +- **`microchip/pic32mz` builds nothing.** The scan is CMake-only and `hw/bsp/pic32mz` ships + only a `family.mk`, so a change there selects no family. That matches reality — `pic32mz` is + in neither provider's family list — but it means the port is unbuilt by CI whether or not + this design lands. Giving it a `family.cmake` is the fix, and is out of scope here. +- **Seven bsp families are in no CI toolchain today** (`espressif`, `efm32`, `same7x`, + `cxd56`, `f1c100s`, `pic32mz`, `py32f0`); the intersection drops them, matching current + behaviour. This change does not alter that. -- cgit v1.3.1 From e13eff8d4e757ebe7709a58fce44017b8be5a84d Mon Sep 17 00:00:00 2001 From: hathach Date: Fri, 21 Aug 2026 12:41:47 +0700 Subject: ci: fix nine ways the selection under-selected or mismatched Every one of these dropped coverage silently - the worst failure mode here, because the PR still goes green. Found by review, each reproduced first. Selection rules: * class_macros derived the config macro from the class DIRECTORY, so a change to src/class/midi/midi2_device.c selected the midi_test examples (which do not compile it) and never examples/device/midi2_device (the only one that enables CFG_TUD_MIDI2, and the only one that does). The file's own macro is unioned in where it differs - union, never replace: over-selecting costs a build, under-selecting merges a break. * the ${FAMILY_MCUS} fallback added for espressif fired on any family whose _family_mcus came back empty, and _cmake_sets is if()-blind and keeps the FIRST definition - so mcx/frdm_mcxn947 answered MCXA15, a token six examples' skip.txt names, dropping 12 firmware images CMake builds. Limited now to families that never spell set(FAMILY_MCUS ...) at all. * lib_examples read only an example's top-level CMakeLists.txt/Makefile; host/msc_file_explorer_freertos names lib/embedded-cli in src/CMakeLists.txt and survived by luck. The whole example tree is scanned. (SEGGER_RTT and rt-thread still resolve to nothing: all three references sit inside a LOGGER=rtt guard no CI build sets - the documented ruling, not a miss.) * get_family_boards applied ci_skip_boards/ci_preferred_boards only under GITHUB_ACTIONS/CIRCLECI, so the selector answered differently on a laptop than on a runner; _prune_buildable forces CI semantics. Its one-board pick also abandoned the whole preferred list when entry one could not build the -e set, and asked skip_example without the build's -D tokens. * _config_enables and lib_examples still read with the locale encoding - under LC_ALL=C the selector tracebacked on three tracked tusb_config.h files. The whole selector and its suite run clean there now. Workflows: * the Membrowse Upload step omitted $EX_ARGS, but --one-first now picks the board from the -e set, so it configured a different, empty build dir and uploaded --identical for a board never compiled. It takes $EX_ARGS for the BOARD; the target stays the aggregate, which has no DEPENDS and still records every example. * blanking FAM_REGEX reset only build_filtered, leaving the build scoped while code-metrics took the UNSCOPED branch and diffed a 1-family run against the full averaged baseline. All three drop together now, as CircleCI's fall-open does. * CircleCI's EX_ARGS had no character screen and is used unquoted, and its code-metrics job still exit 1'd on an empty metrics set - which a scoped build makes a legitimate outcome. * a `ci-full` PR label now turns the scoping off for one PR. A selector bug under-selects silently, and without a label the only ways back to a full matrix are accidental. Performance, since the selector gates every other job: family.cmake texts are read once rather than per changed directory (a 6,000-file dep bump re-read 84 files 99,892 times) and _scrape_mcu is cached: 2.2s -> 0.29s there, 0.8s -> 0.33s on a class diff. Tests: a drift guard for hw/bsp families absent from ci_set_matrix.family_list (they select zero legs now, where they used to ride the full matrix); the rule-4 port test asserted a SUBSET, which set() satisfies, so it could not fail on the empty selection it exists to catch; the GITHUB_ENV guard test counted a SUM of two guards. Drops metrics.py's --only-examples, which nothing called, and applies the TOTAL scrub to the by-example branch that skipped it. --- .circleci/config2.yml | 18 +++++- .github/workflows/build.yml | 24 +++++-- .github/workflows/build_util.yml | 16 ++--- .../2026-08-19-ci-build-family-filter-design.md | 16 ++--- test/hil/test/test_ci_metrics.py | 35 +++++++---- test/hil/test/test_ci_select.py | 24 +++++++ tools/build.py | 31 +++++++-- tools/build_utils.py | 14 ++++- tools/ci_select.py | 73 ++++++++++++++++------ tools/metrics.py | 15 +++-- 10 files changed, 199 insertions(+), 67 deletions(-) (limited to 'docs/superpowers/specs') diff --git a/.circleci/config2.yml b/.circleci/config2.yml index 899cbe24a..2e69588ae 100644 --- a/.circleci/config2.yml +++ b/.circleci/config2.yml @@ -125,6 +125,15 @@ commands: # shell-text interpolation (unsafe characters); family is a job # parameter with charset [a-z0-9_], safe to interpolate directly. EX_ARGS=$(printf '%s' "$EXAMPLE_MAP" | jq -r --arg fam "<< parameters.family >>" '(.[$fam] // []) | map("-e " + .) | join(" ")' 2>/dev/null) || EX_ARGS='' + # same screen as build_util.yml's: the values are example dir names from the + # PR checkout and $EX_ARGS is used unquoted below, so a glob metacharacter + # would pathname-expand against the build cwd. Dropping the filter builds + # everything - the safe direction, and what GHA does for the same input. + case "$EX_ARGS" in + *[!-A-Za-z0-9_/\ ]*) + echo "warning: unexpected characters in the example filter - building all examples" + EX_ARGS='' ;; + esac if [ << parameters.toolchain >> == esp-idf ]; then docker run --rm -v $PWD:/project -w /project espressif/idf:v5.5.3 python tools/build.py << parameters.build-args >> --target all $EX_ARGS << parameters.family >> @@ -253,8 +262,13 @@ jobs: if ls /tmp/metrics/*/*.json 1> /dev/null 2>&1; then python tools/metrics.py combine -j -m -f tinyusb/src /tmp/metrics/*/*.json else - echo "No metrics files found" - exit 1 + # A scoped PR can legitimately build no metrics leg at all (every selected + # family empty, or none of them on a metrics toolchain), so this is not an + # error any more - it was, when the matrix was always the full 64 families. + # An empty file keeps store_artifacts and the compare step below honest: + # both would otherwise act on a missing path. + echo "No metrics files found - PR selection built no metrics leg" + echo '{"files": []}' > metrics.json fi - store_artifacts: diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 2ee124cb3..39a4e7afd 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -68,9 +68,14 @@ jobs: with: fetch-depth: 0 + # The `ci-full` PR label turns the scoping off for one PR: no selection file is + # written, so both matrices and every rig job fall back to the unscoped behaviour. + # An escape hatch is the point - a selector bug under-selects SILENTLY, and without + # a label the only routes back to a full matrix are accidental (touch an + # unclassified path, or break the selector badly enough that it falls open). - name: CI selection (PR only) id: hil-select - if: github.event_name == 'pull_request' + if: github.event_name == 'pull_request' && !contains(github.event.pull_request.labels.*.name, 'ci-full') env: BASE_REF: ${{ github.base_ref }} run: | @@ -166,8 +171,6 @@ jobs: fi fi [ -z "$MATRIX_JSON" ] && MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py) - echo "matrix=$MATRIX_JSON" - echo "matrix=$MATRIX_JSON" >> $GITHUB_OUTPUT # Build-axis extras: the per-family example map rides as a side channel # (a value inside matrix entries would break CircleCI's family parameter @@ -188,12 +191,23 @@ jobs: # silently match another family's baseline case "$FAM_REGEX" in *[!-A-Za-z0-9_\|]*) - echo "::warning::unexpected characters in the family list - unscoped metrics" + echo "::warning::unexpected characters in the family list - dropping the scoping" FAM_REGEX='' ;; esac - [ -z "$FAM_REGEX" ] && BUILD_FILTERED='false' + if [ -z "$FAM_REGEX" ]; then + # all three drop together, as CircleCI's fall-open does. Resetting only + # build_filtered leaves the build scoped while code-metrics takes the + # UNSCOPED branch, diffing a 1-family run against the full averaged + # baseline and publishing that as the PR's code-size impact. + BUILD_FILTERED='false' + EXAMPLE_MAP='{}' + MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py) + fi fi fi + # emitted once, after every path that can still change it + echo "matrix=$MATRIX_JSON" + echo "matrix=$MATRIX_JSON" >> $GITHUB_OUTPUT echo "example_map=$EXAMPLE_MAP" >> $GITHUB_OUTPUT echo "build_filtered=$BUILD_FILTERED" >> $GITHUB_OUTPUT echo "build_families_regex=$FAM_REGEX" >> $GITHUB_OUTPUT diff --git a/.github/workflows/build_util.yml b/.github/workflows/build_util.yml index dfbd83ee2..52999616d 100644 --- a/.github/workflows/build_util.yml +++ b/.github/workflows/build_util.yml @@ -126,14 +126,16 @@ jobs: MEMBROWSE_API_KEY: ${{ secrets.MEMBROWSE_API_KEY }} run: | # if code-changed is false --> there is no elf -> membrowse target upload with --identical flag - # Deliberately NOT scoped by $EX_ARGS: -membrowse-upload has no - # DEPENDS (hw/bsp/family_support.cmake), so the aggregate rebuilds nothing - - # it just records every example, reporting the ones with an elf and - # --identical for the rest. Filtering it here would drop the excluded - # examples from the dataset membrowse-comment.yml reports against, instead - # of recording them as unchanged. + # $EX_ARGS is passed for the BOARD it picks, not to scope the targets: + # --one-first now chooses a board that can build the -e set (tools/build.py), + # so omitting it here would configure a DIFFERENT, empty build dir and upload + # --identical for a board that was never compiled. The target list is not + # scoped by it - `examples-membrowse-upload` is not `all`, so it passes + # through as the aggregate, which has no DEPENDS (hw/bsp/family_support.cmake): + # it rebuilds nothing and still records every example, --identical for the + # ones without an elf. BUILD_PY_ARGS="-s ${{ inputs.build-system }} ${{ steps.setup-toolchain.outputs.build_option }} ${{ inputs.build-options }}" - python tools/build.py $BUILD_PY_ARGS --target examples-membrowse-upload -j 1 ${{ matrix.arg }} + python tools/build.py $BUILD_PY_ARGS --target examples-membrowse-upload -j 1 ${{ matrix.arg }} $EX_ARGS shell: bash - name: Upload Artifacts for Metrics diff --git a/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md b/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md index 9fa358bee..8f77dc50a 100644 --- a/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md +++ b/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md @@ -211,16 +211,16 @@ It falls open to the full matrix whenever the entries are not the whole answer: * the file will not parse; * there is no base content: `--diff-file` mode has no git, so no merge-base blob; * a changed entry carries a family token that names no `hw/bsp/` and is not one of the - eight known aliases. "Changed but unmappable" is not "nothing changed": reading it as the + known aliases. "Changed but unmappable" is not "nothing changed": reading it as the latter empties the whole build matrix for a dep bump. -The eight known aliases (`sam3x`, `samd21`, `samd51`, `same5x`, `fc100s`, `spresense`, -`stm32l1`, `stm32l5`) are pinned in `_DEPS_ALIAS_TOKENS` and select nothing. `get_deps` matches -a token against a requested family name verbatim (`f in entry[2].split()`), so these tokens -match nothing there either — four are pre-rename spellings listed beside the current name in -the same entry, two point at a differently-named family dir (`fc100s`→`f1c100s`, -`spresense`→`cxd56`, both unreachable in `get_deps` itself), and two name no family in the tree. -A ninth appearing fails `TestOrphanInvariant`. +The six known aliases (`sam3x`, `samd21`, `samd51`, `same5x`, `stm32l1`, `stm32l5`) are +pinned in `_DEPS_ALIAS_TOKENS` and select nothing. `get_deps` matches a token against a +requested family name verbatim (`f in entry[2].split()`), so these tokens match nothing +there either — four are pre-rename spellings listed beside the current name in the same +entry, and two name no family in the tree. (`fc100s` and `spresense` were on this list +until they were corrected in `get_deps.py`; those two were the only ones that left a +real dep unreachable for its own family.) A seventh appearing fails `TestOrphanInvariant`. ## Component: `tools/ci_select.py` diff --git a/test/hil/test/test_ci_metrics.py b/test/hil/test/test_ci_metrics.py index 89d03aaae..6c236e827 100644 --- a/test/hil/test/test_ci_metrics.py +++ b/test/hil/test/test_ci_metrics.py @@ -58,13 +58,16 @@ class TestByExample(unittest.TestCase): '-o', out, os.path.join(td, '*', '*', '*.map.json')], check=True) out2 = os.path.join(td, 'sub') r = subprocess.run([sys.executable, METRICS, 'combine', '-q', '-j', - '--only-examples', 'device/cdc_msc', '-o', out2, out + '_by_example.json'], capture_output=True, text=True) self.assertEqual(r.returncode, 0, r.stderr) sub = json.load(open(out2 + '.json')) names = {f['file'] for f in sub['files']} - self.assertEqual(names, {'usbd.c', 'cdc_device.c'}) # bare_api filtered out + # one data entry per example, not one blob: reading it as an ordinary + # metrics.json would double-count every file + self.assertIn('usbd.c', names) + self.assertIn('cdc_device.c', names) + self.assertNotIn('TOTAL', {n.upper() for n in names}) def test_by_example_expansion_is_keyed_on_the_filename(self): # the '_by_example.json' suffix IS the contract (write_by_example, the CMake @@ -339,9 +342,15 @@ class TestWorkflowSelectionHandOff(unittest.TestCase): # with secrets - and for run_*, flips which rig jobs execute for name in ('EX_ARGS', 'ARTIFACT_TAG'): self.assertIn(f'echo "{name}=', self.util) - self.assertEqual(self.util.count('case "$EX_ARGS" in') + - self.util.count('case "$TAG" in'), 2, - 'both GITHUB_ENV writes must screen their value first') + # per guard, not a sum: `count(a) + count(b) == 2` stays green when one guard is + # deleted and the other duplicated + for guard in ('case "$EX_ARGS" in', 'case "$TAG" in'): + self.assertEqual(self.util.count(guard), 1, + f'{guard}: each GITHUB_ENV write screens its value exactly once') + # CircleCI builds from the same PR-derived map and uses $EX_ARGS unquoted + cci = open(os.path.join(CIRCLECI, 'config2.yml')).read() + self.assertIn('case "$EX_ARGS" in', cci, + 'the CircleCI copy of the example filter needs the same screen') self.assertIn('case "$BUILD_ARGS" in', self.build) self.assertIn('unexpected characters in the " + key', self.build, 'the args_*/run_* emitter must screen each board filter') @@ -429,12 +438,16 @@ class TestWorkflowSelectionHandOff(unittest.TestCase): self.assertEqual(matrix.count('ci_set_matrix: UNSCOPED'), 2, 'every fall-open path must print the marker build.yml greps for') - def test_membrowse_upload_is_not_scoped(self): - # -membrowse-upload has no DEPENDS, so the aggregate rebuilds nothing - - # it records every example, --identical for the ones without an elf. Scoping it - # drops the excluded examples from the dataset instead of marking them unchanged. - upload = self.util[self.util.index('--target examples-membrowse-upload'):] - self.assertNotIn('$EX_ARGS', upload.split('\n')[0]) + def test_membrowse_upload_sees_the_same_board_as_the_build(self): + # $EX_ARGS is passed for the BOARD it selects: --one-first picks a board that can + # build the -e set, so without it membrowse configures a different, empty build + # dir and uploads --identical for a board that was never compiled. It does NOT + # scope the targets - `examples-membrowse-upload` is not `all`, so it passes + # through as the aggregate, which has no DEPENDS and still records every example. + line = [l for l in self.util.splitlines() + if '--target examples-membrowse-upload' in l][0] + self.assertIn('$EX_ARGS', line) + self.assertNotIn('-e ', line.replace('$EX_ARGS', '')) if __name__ == '__main__': diff --git a/test/hil/test/test_ci_select.py b/test/hil/test/test_ci_select.py index 74e5f48e6..031e8e287 100644 --- a/test/hil/test/test_ci_select.py +++ b/test/hil/test/test_ci_select.py @@ -777,6 +777,24 @@ class TestOrphanInvariant(unittest.TestCase): for v in vendors: self.assertTrue(ci_select.mcu_families(v + '/x.c', REPO), f'{v}: resolves to no family') + # hw/bsp families ci_set_matrix's family_list does not map to any toolchain. Before + # scoping these were harmless - the matrix was always every family in family_list, + # so a PR touching one of them still compiled the other 64. Now the selection + # intersects to nothing and every leg skips, so a family landing here by accident is + # a silent hole. espressif is deliberate: its boards are built by hil-build-esp, + # keyed on board name rather than family. + UNBUILT_FAMILIES = {'cxd56', 'efm32', 'espressif', 'f1c100s', 'pic32mz', 'py32f0', + 'same7x'} + + def test_every_bsp_family_is_in_the_ci_matrix(self): + sys.path.insert(0, os.path.join(REPO, '.github/scripts')) + import ci_set_matrix + fams = set(ci_select.all_bsp_families(REPO)) + self.assertEqual(fams - set(ci_set_matrix.family_list), self.UNBUILT_FAMILIES, + 'a hw/bsp family that no toolchain in ci_set_matrix.family_list ' + 'builds: a PR touching only it now selects zero build legs. Wire ' + 'it into family_list, or add it here with a reason.') + def test_every_get_deps_family_token_resolves_or_is_a_known_alias(self): """Same drift guard, dep side. A token naming no hw/bsp dir makes the entry unreachable for its family in get_deps.py itself (`f in entry[2].split()`), and @@ -1132,8 +1150,14 @@ class TestBuildClassifier(unittest.TestCase): # real feather_rp2040_max3421 board) and espressif's component CMakeLists also # references it — so the raw (unpruned) scan legitimately finds both; Task 4's # buildability post-filter is what may later prune either away + # non-empty FIRST: a subset assertion is satisfied by set(), and since ports are + # now empty-means-empty (fail-closed) an unnoticed regression to zero families + # would select no build leg at all and merge an uncompiled HCD + self.assertTrue(s['families'], 'a host-port change must select some family') self.assertLessEqual(set(s['families']), {'espressif', 'rp2040'}) + self.assertTrue(s['family_examples'], 'and must name the examples for them') for exs in s['family_examples'].values(): + self.assertTrue(exs) self.assertFalse(any(e.startswith(('device/', 'typec/')) for e in exs)) def test_port_shared_file_selects_all_examples(self): # rule 5 diff --git a/tools/build.py b/tools/build.py index e7ca1c839..eeefca22d 100755 --- a/tools/build.py +++ b/tools/build.py @@ -299,7 +299,8 @@ def build_boards_list(boards, build_defines, build_system, build_name, build_cfl return ret -def get_family_boards(family, one_random, one_first, examples=None, build_system='cmake'): +def get_family_boards(family, one_random, one_first, examples=None, build_system='cmake', + extra_defines=(), ci=None): """Get list of boards for a family. Args: @@ -314,13 +315,23 @@ def get_family_boards(family, one_random, one_first, examples=None, build_system which every one of those examples skips - and the leg runs to green having compiled nothing and uploaded no metrics. build_system: which skip answer to ask for; the two differ (build_utils) + extra_defines: this build's -D tokens, so a board whose only.txt match comes + from -DMAX3421_HOST=1 is not judged unbuildable here and buildable in + cmake_board + ci: force the ci_skip_boards / ci_preferred_boards lists on or off. Default + None reads the environment, which is right for a build but NOT for a caller + asking what CI would do: ci_select must answer the same on a laptop as on a + runner, or /pre-pr and the code-size skill report a family list CI will not + reproduce. Returns: List of board names """ + if ci is None: + ci = bool(os.getenv('GITHUB_ACTIONS') or os.getenv('CIRCLECI')) skip_list = [] preferred_list = [] - if os.getenv('GITHUB_ACTIONS') or os.getenv('CIRCLECI'): + if ci: skip_list = ci_skip_boards.get(family, []) preferred_list = ci_preferred_boards.get(family, []) @@ -339,9 +350,16 @@ def get_family_boards(family, one_random, one_first, examples=None, build_system # no filter, or nothing in the filter is buildable anywhere: keep today's # answer rather than inventing a different board return examples is None or any( - not build_utils.skip_example(e, board, (), build_system) for e in examples) - - if preferred_list and buildable(preferred_list[0]): + not build_utils.skip_example(e, board, extra_defines, build_system) + for e in examples) + + # the WHOLE preferred list, in order - stopping at entry one would abandon a + # curated list for the raw alphabetical order the moment its first board cannot + # build the filter, which also moves the board the metrics baseline is keyed on + for b in preferred_list: + if buildable(b): + return [b] + if preferred_list and examples is None: return [preferred_list[0]] candidates = [b for b in all_boards if buildable(b)] or all_boards if one_first: @@ -434,7 +452,8 @@ def main(): # get boards from families and append to boards list all_boards = list(boards) for f in all_families: - all_boards.extend(get_family_boards(f, one_random, one_first, examples, build_system)) + all_boards.extend(get_family_boards(f, one_random, one_first, examples, + build_system, tuple(build_defines))) # build all boards result = build_boards_list(all_boards, build_defines, build_system, build_name, build_cflags, build_targets, diff --git a/tools/build_utils.py b/tools/build_utils.py index 2af8fd624..1eeef0269 100755 --- a/tools/build_utils.py +++ b/tools/build_utils.py @@ -141,9 +141,12 @@ def _family_mcus(family_dir, board_dir): board_cmake = pathlib.Path(board_dir) / "board.cmake" out = set() depth = 0 + any_set = False for line in text.splitlines(): line = line.strip() m = _FAMILY_MCUS_RE.match(line) + if m: + any_set = True if m and depth == 0: files = (str(board_cmake), str(fam_cmake)) for tok in m.group(1).split(): @@ -156,16 +159,23 @@ def _family_mcus(family_dir, board_dir): depth += 1 elif re.match(r'endif\s*\(', line): depth = max(0, depth - 1) - if not out: + if not out and not any_set: # FAMILY_MCUS can also be produced rather than set: hw/bsp/espressif derives it # with `string(TOUPPER ${IDF_TARGET} FAMILY_MCUS)`, which _FAMILY_MCUS_RE cannot - # see, leaving espressif's whole cmake answer resting on the IDF_TARGET scrape + # see, leaving espressif's whole cmake answer resting on the IDF_TARGET scrape. + # + # `not any_set` is load-bearing: _cmake_sets is if()-blind and keeps the FIRST + # definition, so on a family that sets FAMILY_MCUS only inside conditionals + # (mcx, nrf) this would leak branch one's value onto every board - mcx/frdm_mcxn947 + # answered MCXA15, which six examples' skip.txt names, dropping 12 firmware + # images CMake actually builds. Those families keep the CFG_TUSB_MCU scrape. val = _cmake_expand('${FAMILY_MCUS}', (str(board_cmake), str(fam_cmake))) if val: out.add(val) return frozenset(out) +@functools.lru_cache(maxsize=None) def _scrape_mcu(family_dir, board_dir, family): """(CFG_TUSB_MCU token of this board, the text it was read from), master's algorithm verbatim: family.mk (family.cmake when there is none) first, falling diff --git a/tools/ci_select.py b/tools/ci_select.py index d253f8c01..cd63899c1 100755 --- a/tools/ci_select.py +++ b/tools/ci_select.py @@ -42,6 +42,7 @@ ALL_TESTS = {'device': device_tests, 'dual': dual_tests, 'host': host_test} # class dir -> config macro suffix exceptions (rule 3); dfu is per-file, handled inline NET_MACROS = ('ECM_RNDIS', 'NCM') + def _read(path: str) -> str: """Read a source file with a fixed encoding. The locale's is not it: several tracked sources carry non-ASCII bytes, and under LC_ALL=C the decode raises UnicodeDecodeError @@ -211,17 +212,25 @@ def path_families(rel_dir: str, repo_root: str) -> set: CI job and resolves to nothing. Boundary = '/', whitespace, quote, paren, brace or end: `${TOP}/hw/mcu/nordic/nrfx` has no trailing slash, while bare 'microchip/pic' must not match '.../microchip/pic32mz/...'.""" - fams = set() - bsp_root = os.path.join(repo_root, 'hw/bsp') pat = re.compile(re.escape(rel_dir) + r'(?=[/\s"\')}]|$)', re.M) - for f in glob.glob(os.path.join(bsp_root, '*/family.cmake')) + \ - glob.glob(os.path.join(bsp_root, '*/components/*/CMakeLists.txt')): + return {fam for fam, text in _family_file_texts(repo_root) if pat.search(text)} + + +@functools.lru_cache(maxsize=None) +def _family_file_texts(repo_root: str) -> tuple: + """((family, text), ...) for every family.cmake and espressif component + CMakeLists.txt, read once. path_families is called per distinct directory in the + diff and its own cache only helps repeats: a 6,000-file hw/mcu dep bump re-read + these 84 files 99,892 times (2.2 s) before this.""" + bsp_root = os.path.join(repo_root, 'hw/bsp') + out = [] + for f in sorted(glob.glob(os.path.join(bsp_root, '*/family.cmake')) + + glob.glob(os.path.join(bsp_root, '*/components/*/CMakeLists.txt'))): try: - if pat.search(_read(f)): - fams.add(os.path.relpath(f, bsp_root).split(os.sep, 1)[0]) + out.append((os.path.relpath(f, bsp_root).split(os.sep, 1)[0], _read(f))) except OSError: pass - return fams + return tuple(out) def port_families(port_dir: str, repo_root: str) -> set: @@ -348,10 +357,14 @@ def class_include_edges(repo_root: str) -> dict: return edges +_CLS_STEM_RE = re.compile(r'(.*?)(?:_(?:device|host))?\.[ch]$') + + def class_macros(cls: str, base: str, prefix: str) -> list: """Config macros that compile a class dir's code, for role prefix TUD/TUH. - `base` refines dfu only (it splits DFU from DFU_RUNTIME per file); pass '' for - a class reached through an include edge, where the widest set is correct.""" + `base` refines dfu (it splits DFU from DFU_RUNTIME per file) and adds the file's + own macro where that differs from the directory's; pass '' for a class reached + through an include edge, where the widest set is correct.""" if cls == 'net': return [f'CFG_{prefix}_{m}' for m in NET_MACROS] if cls == 'dfu': @@ -360,7 +373,18 @@ def class_macros(cls: str, base: str, prefix: str) -> list: if base.startswith('dfu_device') or base.startswith('dfu_host'): return [f'CFG_{prefix}_DFU'] return [f'CFG_{prefix}_DFU', f'CFG_{prefix}_DFU_RUNTIME'] - return [f'CFG_{prefix}_{cls.upper()}'] + out = [f'CFG_{prefix}_{cls.upper()}'] + # A class directory can hold more than one class. src/class/midi ships MIDI 1.0 + # AND MIDI 2.0: midi2_device.c is `#if CFG_TUD_ENABLED && CFG_TUD_MIDI2`, and + # examples/device/midi2_device is the only example that enables it - so the + # directory macro alone selected the midi_test examples, which do not compile the + # changed file, and none of the ones that do. Union, never replace: the file may + # still be pulled in by the directory's own macro, and over-selecting costs a build + # while under-selecting merges a break. + m = _CLS_STEM_RE.match(base) + if m and m.group(1) and m.group(1) != cls: + out.append(f'CFG_{prefix}_{m.group(1).upper()}') + return out # A define is OFF only when its value is a literal zero (0, 00, (0)), optionally @@ -407,7 +431,7 @@ def _class_roles(base: str) -> set: def _config_enables(cfg_path: str, macros) -> bool: try: - with open(cfg_path) as f: + with open(cfg_path, encoding='utf-8', errors='replace') as f: text = f.read() except OSError: return False @@ -435,15 +459,23 @@ def lib_examples(lib_name: str, repo_root: str) -> set: 'lib/net' cannot inherit lib/networking's example). Per-example on purpose: lib/SEGGER_RTT is named by family_support.cmake's - LOGGER=rtt plumbing, which no CI example build turns on, so a family-file scan - would wrongly narrow it to three families instead of answering 'nobody'.""" + LOGGER=rtt plumbing, which no CI example build turns on (all three references - + family_support.cmake, family_support.mk, rp2040/family.cmake - sit inside a + LOGGER=rtt guard), so a family-file scan would wrongly narrow it to three families + instead of answering 'nobody'. + + The whole example TREE is scanned, not just its top-level files: examples/host/ + msc_file_explorer_freertos/src/CMakeLists.txt names lib/embedded-cli, and that + example survived only because its top-level file happens to name it too.""" pat = re.compile(re.escape('lib/' + lib_name) + r'(?=[/\s"\')}]|$)', re.M) out = set() for ex in all_examples(repo_root): - for f in ('CMakeLists.txt', 'Makefile'): + for f in sorted(glob.glob(os.path.join(repo_root, 'examples', ex, '**', '*'), + recursive=True)): + if os.path.basename(f) not in ('CMakeLists.txt', 'Makefile'): + continue try: - with open(os.path.join(repo_root, 'examples', ex, f)) as fh: - text = fh.read() + text = _read(f) except OSError: continue if pat.search(text): @@ -804,7 +836,7 @@ def main(): repo_root = _REPO_ROOT rosters = [] for c in a.configs: - with open(c) as f: + with open(c, encoding='utf-8', errors='replace') as f: rosters.append((c, json.load(f)['boards'])) files = (_read(a.diff_file).splitlines() if a.diff_file @@ -1029,7 +1061,12 @@ def _prune_buildable(fams, fam_ex, repo_root): reasons.append(f'{fam}: family dir gone from tree, dropped') continue try: - boards = build_py.get_family_boards(fam, False, False) + # ci=True unconditionally: this answers "what will CI build", so it must + # not change with GITHUB_ACTIONS/CIRCLECI being set. Locally the lists + # are off by default, and rp2040 would keep feather_rp2040_max3421 - + # the only board satisfying the max3421 only.txt files - giving a + # developer a family list the runner will not reproduce. + boards = build_py.get_family_boards(fam, False, False, ci=True) except OSError as e: # belt and braces: never traceback here reasons.append(f'{fam}: boards unreadable ({e}), dropped') continue diff --git a/tools/metrics.py b/tools/metrics.py index b97b2b206..27c995954 100644 --- a/tools/metrics.py +++ b/tools/metrics.py @@ -83,7 +83,7 @@ def parse_bloaty_csv(csv_text, filters=None): return {"files": files, "TOTAL": total_all} -def combine_files(input_files, filters=None, only_examples=None): +def combine_files(input_files, filters=None): """Combine multiple metrics inputs (bloaty CSV or metrics JSON) into a single data set.""" filters = filters or [] @@ -105,9 +105,11 @@ def combine_files(input_files, filters=None, only_examples=None): # rule and metrics_pair_compare all spell that suffix) - a shape # sniff would silently reroute any coincidentally-shaped JSON. for ex in sorted(json_data): - if only_examples and ex not in only_examples: - continue - sub = {'files': list(json_data[ex]['files'])} + # same TOTAL scrub the shared path below applies: this branch + # `continue`s past it, so do it here or a by-example input keeps + # the fake TOTAL rows an ordinary input has stripped + sub = {'files': [f for f in json_data[ex]['files'] + if str(f.get('file', '')).upper() != 'TOTAL']} if filters: sub['files'] = [f for f in sub['files'] if f.get('path') and any(x in f['path'] for x in filters)] @@ -614,8 +616,7 @@ def render_compare_table(rows, include_sum): def cmd_combine(args): """Handle combine subcommand.""" input_files = expand_files(args.files) - only_examples = set(args.only_examples.split(',')) if args.only_examples else None - all_json_data = combine_files(input_files, args.filters, only_examples=only_examples) + all_json_data = combine_files(input_files, args.filters) json_average = compute_avg(all_json_data) if json_average is None: @@ -673,8 +674,6 @@ def main(argv=None): help='Sort order: size/size- (descending), size+ (ascending), name/name+ (ascending), name- (descending). Default: size-') combine_parser.add_argument('--by-example', dest='by_example', action='store_true', help='Also write _by_example.json: per-example file lists keyed by role/example') - combine_parser.add_argument('--only-examples', dest='only_examples', default='', - help='Comma-separated role/example ids to keep when reading by-example JSON inputs') # Compare subcommand compare_parser = subparsers.add_parser('compare', help='Compare two metrics inputs (bloaty CSV or metrics JSON)') -- cgit v1.3.1 From 03a329eeda09e073d7de9be84df55d65392e4013 Mon Sep 17 00:00:00 2001 From: hathach Date: Fri, 21 Aug 2026 22:17:51 +0700 Subject: ci_select: classify the 254 files that were reaching rule 17 Rule 17 (unclassified -> full on both axes) is the fail-open net for paths nobody anticipated, and it must stay that way: a wrong `full` costs runner minutes and is visible in the run, a wrong `empty` costs a merged regression and is invisible. But nothing in the tree should REACH it, and 254 tracked files did. The cost was real. PR #3842 changed a skill, a README and .gitignore; .gitignore matched no rule, so both axes went full and 74 cmake legs span up runners to do checkout + toolchain + get_deps before skipping the build, plus the whole 30-board rig. Three changes, none of which touch rule 17 itself: 1. _META_RE - repo metadata and tooling no Build step reads: .gitignore, .gitattributes, .clang-format, .codespellrc, .pre-commit-config.yaml, .readthedocs.yaml, .PVS-Studio/, .idea/, sonar-project.properties, the packaging manifests, CMakePresets, udev rules, test/{fuzz,unit-test} (their own jobs build those), the non-build .github/ files, and the tools/*.py scripts no build invokes. Deliberately NOT included, and still full: .circleci/**, .github/workflows/build*.yml, .github/actions/**, .github/scripts/**. The line is "does a Build step read this", not "is it source". 2. Rules 15 and 16 now match what they already claimed. Row 15 names examples//CMakeLists.txt and the regex never had it; row 16 says tools/build*.py but anchored tools/build\.py$. Both got the right answer only because rule 17 caught them on the way past. Also names their siblings - family_support.mk, family_rules.mk, src/CMakeLists.txt, src/tinyusb.mk - and .circleci/**, which generates the whole CircleCI matrix and was in no row at all. 3. src/typec/** gets row 12b. It is listed unconditionally by both build systems but its body is `#if CFG_TUC_ENABLED`, which only examples/typec/power_delivery sets - the same shape as the class rule, so the same answer: the examples that enable it (stm32g4 and stm32u5 after the buildability prune), and nothing on the rig, which runs no typec test. It was force-fulling 82 families and all 30 boards. TestNoTrackedFileIsUnclassified walks every tracked file and asserts none reaches rule 17, on both axes - 254 -> 0. Verified it fails when a new unclassified path appears. That turns 17 into what it should be: unreachable for anything in the tree, so it fires only for genuinely new shapes, and the author is told to write the row rather than letting the fall-through pick an answer for them. test_full_paths used sonar-project.properties as its stand-in for "unclassified"; that is now metadata, so the case moved to the new test_repo_metadata_is_not_a_build_input, with test_the_build_machinery_is_still_full pinning the other side of the line. --- .../2026-08-19-ci-build-family-filter-design.md | 48 ++++++------- test/hil/test/test_ci_select.py | 62 ++++++++++++++++- tools/ci_select.py | 79 ++++++++++++++++++++-- 3 files changed, 160 insertions(+), 29 deletions(-) (limited to 'docs/superpowers/specs') diff --git a/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md b/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md index 8f77dc50a..b10f5b4ae 100644 --- a/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md +++ b/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md @@ -46,29 +46,31 @@ never inflates one axis with another's breadth. `FAM` = the families whose `family.cmake` references the changed path (CMake only — see below). "roster boards" = boards on `test/hil/{tinyusb,hfp}.json`. -| # | Changed path | Build families | Build examples | HIL boards → tests | -| --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | -| 1 | `docs/`, `.claude/`, `*.md`, `*.rst`, `LICENSE` | — | — | — | -| 2 | `test/hil/**` | — | — | all boards → all tests | -| 2b | `tools/metrics.py`, `.github/scripts/metrics_*.py` | `ALL` (unchanged — `tinyusb_metrics` runs `metrics.py` as a build target) | `ALL` | — (nothing on the rig runs it) | -| 3 | `src/portable//dcd_*`, `*_device.[ch]` | `FAM` | `DEV`+`DUAL` | `FAM`'s device-role boards → device+dual tests | -| 4 | `src/portable//hcd_*`, `*_host.[ch]` | `FAM` | `HOST`+`DUAL` | `FAM`'s host-role boards → host+dual tests | -| 5 | `src/portable//**` (anything else) | `FAM` | `ALL` | `FAM`'s boards → all their tests | -| 5b | `src/portable//**` where `FAM` is empty | — | — | — (empty resolves to nothing on BOTH axes) | -| 6 | `hw/bsp//**` | that family | `ALL` | that family's boards → all tests (a `boards//` path narrows to that board) | -| 7 | `hw/mcu//**` | `FAM` — empty resolves to nothing (maintainer ruling) | `ALL` | `FAM`'s boards → all tests; empty resolves to nothing (maintainer ruling) ⚠ *see below* | -| 8 | `src/class//*_device.[ch]` | `ALL` | examples enabling `CFG_TUD_` | device-role boards → HIL tests enabling `CFG_TUD_` | -| 9 | `src/class//*_host.[ch]` | `ALL` | examples enabling `CFG_TUH_` | host-role boards → HIL tests enabling `CFG_TUH_` | -| 10 | `src/class//**` (shared header) | `ALL` | either, **plus include-edge classes** | both roles → same, plus include-edge classes | -| 11 | `src/device/**` | `ALL` | `DEV`+`DUAL` | device-role boards → device+dual tests | -| 12 | `src/host/**` | `ALL` | `HOST`+`DUAL` | host-role boards → host+dual tests | -| 13 | `examples///**` | `ALL` | just `` | if `` is a HIL test: all boards → that test; else nothing | -| 14 | `examples/device/board_test/**` | `ALL` | just `board_test` | all boards → all tests (HIL parking firmware) | -| 15 | `examples/build_system/**`, `examples/CMakeLists.txt`, `examples//CMakeLists.txt` | `ALL` | `ALL` | all boards → all tests | -| 16 | `src/common/`, `src/osal/`, `src/tusb.[ch]`, `src/tusb_option.h`, `tools/build*.py`, `tools/cmake/**`, `hw/bsp/{family_support.cmake,board.c,board_api.h,ansi_escape.h}`, `.github/**` | `ALL` | `ALL` | all boards → all tests | -| 16a | `lib//**` | `ALL` | examples whose own `CMakeLists.txt`/`Makefile` names `lib/` | those examples that are HIL tests, on all boards; empty resolves to nothing | -| 16b | `tools/get_deps.py` | families whose `deps_mandatory`/`deps_optional` entries changed | `ALL` | those families' boards → all tests; a logic change, an `'all'` entry, no base content or a changed token naming no family → full | -| 17 | anything unclassified | `ALL` | `ALL` | all boards → all tests (fail-open) | +| # | Changed path | Build families | Build examples | HIL boards → tests | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| 1 | `docs/`, `.claude/`, `*.md`, `*.rst`, `LICENSE` | — | — | — | +| 1b | `.gitignore`, `.clang-format`, `.idea/**`, `test/{fuzz,unit-test}/**`, non-build `.github/**`, packaging manifests | — | — | — | +| 2 | `test/hil/**` | — | — | all boards → all tests | +| 2b | `tools/metrics.py`, `.github/scripts/metrics_*.py` | `ALL` (unchanged — `tinyusb_metrics` runs `metrics.py` as a build target) | `ALL` | — (nothing on the rig runs it) | +| 3 | `src/portable//dcd_*`, `*_device.[ch]` | `FAM` | `DEV`+`DUAL` | `FAM`'s device-role boards → device+dual tests | +| 4 | `src/portable//hcd_*`, `*_host.[ch]` | `FAM` | `HOST`+`DUAL` | `FAM`'s host-role boards → host+dual tests | +| 5 | `src/portable//**` (anything else) | `FAM` | `ALL` | `FAM`'s boards → all their tests | +| 5b | `src/portable//**` where `FAM` is empty | — | — | — (empty resolves to nothing on BOTH axes) | +| 6 | `hw/bsp//**` | that family | `ALL` | that family's boards → all tests (a `boards//` path narrows to that board) | +| 7 | `hw/mcu//**` | `FAM` — empty resolves to nothing (maintainer ruling) | `ALL` | `FAM`'s boards → all tests; empty resolves to nothing (maintainer ruling) ⚠ *see below* | +| 8 | `src/class//*_device.[ch]` | `ALL` | examples enabling `CFG_TUD_` | device-role boards → HIL tests enabling `CFG_TUD_` | +| 9 | `src/class//*_host.[ch]` | `ALL` | examples enabling `CFG_TUH_` | host-role boards → HIL tests enabling `CFG_TUH_` | +| 10 | `src/class//**` (shared header) | `ALL` | either, **plus include-edge classes** | both roles → same, plus include-edge classes | +| 11 | `src/device/**` | `ALL` | `DEV`+`DUAL` | device-role boards → device+dual tests | +| 12 | `src/host/**` | `ALL` | `HOST`+`DUAL` | host-role boards → host+dual tests | +| 12b | `src/typec/**` | `ALL` | examples enabling `CFG_TUC_ENABLED` | — (no rig board runs a typec test) | +| 13 | `examples///**` | `ALL` | just `` | if `` is a HIL test: all boards → that test; else nothing | +| 14 | `examples/device/board_test/**` | `ALL` | just `board_test` | all boards → all tests (HIL parking firmware) | +| 15 | `examples/build_system/**`, `examples/CMakeLists.txt`, `examples//CMakeLists.txt` | `ALL` | `ALL` | all boards → all tests | +| 16 | `src/common/`, `src/osal/`, `src/tusb.[ch]`, `src/tusb_option.h`, `tools/{build,build_utils,ci_select}.py`, `tools/cmake/**`, `src/CMakeLists.txt`, `src/tinyusb.mk`, `hw/bsp/{family_support.{cmake,mk},family_rules.mk,zephyr_board_aliases.cmake,board.c,board_api.h,ansi_escape.h}`, `.github/**`, `.circleci/**` | `ALL` | `ALL` | all boards → all tests | +| 16a | `lib//**` | `ALL` | examples whose own `CMakeLists.txt`/`Makefile` names `lib/` | those examples that are HIL tests, on all boards; empty resolves to nothing | +| 16b | `tools/get_deps.py` | families whose `deps_mandatory`/`deps_optional` entries changed | `ALL` | those families' boards → all tests; a logic change, an `'all'` entry, no base content or a changed token naming no family → full | +| 17 | anything unclassified (no tracked file reaches this — TestNoTrackedFileIsUnclassified) | `ALL` | `ALL` | all boards → all tests (fail-open) | **Rule 2 is deliberately asymmetric.** A `test/hil/**` change is invisible to the family matrix but is exactly what the rig exercises, so it builds nothing and runs everything. diff --git a/test/hil/test/test_ci_select.py b/test/hil/test/test_ci_select.py index 8f1841531..66b20b2e4 100644 --- a/test/hil/test/test_ci_select.py +++ b/test/hil/test/test_ci_select.py @@ -840,6 +840,45 @@ class TestRostersDoNotOverlap(unittest.TestCase): seen[b['name']] = b.get('tests') +class TestNoTrackedFileIsUnclassified(unittest.TestCase): + """Rule 17 (unclassified -> full on both axes) is the fail-open net for paths nobody + anticipated. It must stay that way - a wrong `full` costs runner minutes and is + visible in the run, a wrong `empty` costs a merged regression and is invisible - but + nothing in the tree should REACH it. Every tracked file is classified by a rule, so + 17 fires only for genuinely new shapes, and this test is what tells the author to + write the row instead of letting the fall-through pick an answer for them. + + Before this guard, 254 tracked files reached 17: .gitignore took a docs-only PR to + 74 cmake legs and the whole rig, while examples//CMakeLists.txt got the RIGHT + answer from the wrong rule - row 15 names it, the regex never matched it.""" + + def _unclassified(self, axis): + import subprocess as sp + r = sp.run(['git', 'ls-files'], cwd=REPO, capture_output=True, text=True) + if r.returncode != 0: + self.skipTest('not a git checkout') + files = r.stdout.split() + self.assertGreater(len(files), 1000, 'suspiciously few tracked files') + out = [] + for f in files: + s = (ci_select.classify_build([f], REPO) if axis == 'build' + else ci_select.classify([f], REPO, real_rosters())) + if any('unclassified' in why for why in s['reasons']): + out.append(f) + return out + + def test_build_axis(self): + left = self._unclassified('build') + self.assertEqual(left, [], f'{len(left)} tracked files fall through to rule 17 on ' + f'the build axis, e.g. {left[:5]} - classify them, or ' + f'add the pattern to _META_RE if no build reads them') + + def test_hil_axis(self): + left = self._unclassified('hil') + self.assertEqual(left, [], f'{len(left)} tracked files fall through to rule 17 on ' + f'the HIL axis, e.g. {left[:5]}') + + class TestLibRule(unittest.TestCase): """lib/** is not a full-matrix path: only the examples that build the lib need it.""" @@ -1241,7 +1280,28 @@ class TestBuildClassifier(unittest.TestCase): 'tools/build.py', 'tools/cmake/cpu/cortex-m4.cmake', 'examples/CMakeLists.txt', 'examples/device/CMakeLists.txt', 'examples/build_system/cmake/cpu.cmake', '.github/workflows/build.yml', - 'sonar-project.properties', 'some/unknown/path.c'): + '.circleci/config.yml', 'src/CMakeLists.txt', 'src/tinyusb.mk', + 'hw/bsp/family_support.mk', 'tools/build_utils.py', + 'some/unknown/path.c'): + self.assertTrue(self.b([p])['full'], p) + + def test_repo_metadata_is_not_a_build_input(self): + # these used to reach `full` through rule 17: a PR touching only .gitignore and a + # README created 74 cmake legs and booked the whole rig. No Build step reads them. + for p in ('sonar-project.properties', '.gitignore', '.gitattributes', + '.clang-format', '.idea/misc.xml', 'version.yml', 'library.json', + 'examples/CMakePresets.json', 'test/fuzz/fuzz.cc', + 'test/unit-test/project.yml', '.github/workflows/pr_comment.yml', + 'tools/gen_doc.py'): + s = self.b([p]) + self.assertFalse(s['full'], p) + self.assertEqual(s['families'], [], p) + + def test_the_build_machinery_is_still_full(self): + # the other side of the same line: these DECIDE what gets built + for p in ('.circleci/config.yml', '.github/workflows/build.yml', + '.github/scripts/ci_set_matrix.py', 'tools/ci_select.py', + 'tools/build_utils.py', 'tools/metrics.py'): self.assertTrue(self.b([p])['full'], p) def test_mixed_diff_unions_per_family(self): diff --git a/tools/ci_select.py b/tools/ci_select.py index ced3bbbc0..cf5a0da5b 100755 --- a/tools/ci_select.py +++ b/tools/ci_select.py @@ -54,6 +54,34 @@ def _read(path: str) -> str: _NONCODE_RE = re.compile( r'^(docs/|\.claude/|.*\.(md|rst)$|LICENSE)') +# Repo metadata and tooling that no CI build reads. Enumerated rather than left to +# rule 17, which widens BOTH axes: a PR touching only .gitignore and a README was +# creating 74 cmake legs (each a runner doing checkout + toolchain + get_deps before +# skipping the build) and booking the whole 30-board rig. +# +# Deliberately NOT here, and still full: .circleci/**, .github/workflows/build*.yml, +# .github/actions/**, .github/scripts/** - those decide what gets built. The line is +# "does any Build step read this file", not "is it source". +# +# test/{fuzz,unit-test} have their own jobs (cifuzz.yml, the unit-test pre-commit hook +# and workflow); the Build matrix never compiles them, and test/hil is rule 2. +_META_RE = re.compile( + r'^(' + r'\.(gitignore|gitattributes|clang-format|codespellrc|readthedocs\.yaml)$|' + r'\.pre-commit-config\.yaml$|\.PVS-Studio/|\.idea/|\.vscode/|' + r'sonar-project\.properties$|library\.json$|pkg\.yml$|repository\.yml$|' + r'version\.yml$|SConscript$|' + r'.*CMakePresets\.json$|hw/bsp/BoardPresets\.json$|examples/west\.yml$|' + r'.*/[0-9]+-tinyusb[^/]*\.rules$|tools/usb_drivers/|tools/codespell/|' + r'test/(fuzz|unit-test)/|' + # .github, minus the build machinery named in _FULL_RE + r'\.github/(FUNDING\.yml|labeler\.yml|membrowse_pr_message\.j2|ISSUE_TEMPLATE/|' + r'workflows/(cifuzz|claude|claude-code-review|labeler|membrowse-comment|' + r'membrowse-onboard|pr_comment|pre-commit|static_analysis|trigger)\.yml$)|' + # tools/ scripts no build invokes (tools/build*.py and metrics are handled above) + r'tools/(build_doc|check_example_pids|file2carray|gen_doc|gen_presets|iar_gen|' + r'make_release|mksunxi|pcapng_to_corpus)\.py$|tools/iar_template\.ipcf$' + r')') # Build-size metrics tooling. HIL axis ONLY: nothing on the rig runs any of it, and # without this rule these paths are unclassified, so a metrics-only PR booked an # exclusive full 30-board sweep to validate a script no board executes. @@ -66,9 +94,20 @@ _METRICS_RE = re.compile( _FULL_RE = re.compile( r'^(src/common/|src/osal/|src/tusb\.c$|src/tusb\.h$|src/tusb_option\.h$|' r'test/hil/|\.github/workflows/build.*\.yml$|\.github/actions/|\.github/scripts/|' - r'tools/build\.py$|tools/cmake/|' - r'hw/bsp/(family_support\.cmake|board_api\.h|board\.c|ansi_escape\.h)$|' + # generates the whole CircleCI matrix, same authority as .github/** + r'\.circleci/|' + # rule 16 says `tools/build*.py`; name the two siblings the glob implies. Both + # decide what gets built, so neither can be trusted to narrow its own change. + r'tools/(build|build_utils|ci_select)\.py$|tools/cmake/|' + # the make twins of family_support.cmake are the same authority for the make legs + r'hw/bsp/(family_support\.(cmake|mk)|family_rules\.mk|zephyr_board_aliases\.cmake|' + r'board_api\.h|board\.c|ansi_escape\.h)$|' + # rule 15 lists examples//CMakeLists.txt - it registers every target in that + # role, so it was only ever reaching `full` through rule 17's fall-through r'examples/build_system/|examples/CMakeLists\.txt$|' + r'examples/[^/]+/CMakeLists\.txt$|' + # every firmware compiles these unconditionally (src/CMakeLists.txt, src/tinyusb.mk) + r'src/CMakeLists\.txt$|src/tinyusb\.mk$|' # board_test is HIL infrastructure, not a test: hil_test.py flashes it to park # every board (variant boundary + end-of-board teardown), so every board depends on it r'examples/device/board_test/)') @@ -538,7 +577,7 @@ class _Sel: def _classify_one(path, repo_root, roster_boards, extras: set, s: _Sel, get_deps_families=None): base = os.path.basename(path) - if _NONCODE_RE.match(path): + if _NONCODE_RE.match(path) or _META_RE.match(path): s.reasons.append(f'{path}: non-code, no contribution') return if _METRICS_RE.match(path): @@ -673,6 +712,11 @@ def _classify_one(path, repo_root, roster_boards, extras: set, s: _Sel, s.add(boards, sorted(tests), f'{path}: lib {lib} -> {sorted(tests)} on all boards') return + if re.match(r'src/typec/', path): + # only examples/typec enables CFG_TUC_ENABLED, and no rig board runs a typec + # test (see _HIL_EX_ROLES) - so the build axis covers it and the rig cannot + s.reasons.append(f'{path}: typec, no HIL contribution') + return m = _BUILD_EX_RE.match(path) if m: if m.group(1) not in _HIL_EX_ROLES: @@ -935,7 +979,8 @@ class _BSel: def _classify_build_one(path, repo_root, s: _BSel, get_deps_families=None): base = os.path.basename(path) - if _NONCODE_RE.match(path): # rule 1 + if _NONCODE_RE.match(path) or _META_RE.match(path): # rule 1 + s.reasons.append(f'{path}: non-code, no build contribution') return if re.match(r'test/hil/', path): # rule 2 s.reasons.append(f'{path}: HIL harness, no build contribution') @@ -1006,6 +1051,18 @@ def _classify_build_one(path, repo_root, s: _BSel, get_deps_families=None): # CMakeLists (rule 15) is what forces the full matrix s.reasons.append(f'{path}: not an example dir, no build contribution') return + if re.match(r'src/typec/', path): # rule 12b + # listed unconditionally by src/CMakeLists.txt and src/tinyusb.mk, but the whole + # body is `#if CFG_TUC_ENABLED` - so it is PARSED by every build and COMPILED + # only for examples that enable it. Same shape as the class rule, same answer: + # the examples whose tusb_config.h turns it on, and empty means empty. + exs = examples_enabling(role_examples(repo_root, ('typec',)), + ('CFG_TUC_ENABLED',), repo_root) + if not exs: + s.reasons.append(f'{path}: typec enabled by no example config, no contribution') + return + s.add(all_bsp_families(repo_root), exs, f'{path}: typec -> {sorted(exs)}') + return m = re.match(r'lib/([^/]+)/', path) if m: # lib rule lib = m.group(1) @@ -1018,7 +1075,19 @@ def _classify_build_one(path, repo_root, s: _BSel, get_deps_families=None): return s.add(all_bsp_families(repo_root), exs, f'{path}: lib {lib} -> {sorted(exs)}') return - s.force_full(f'{path}: unclassified -> full build matrix') # rules 15-17 + if _METRICS_RE.match(path): + # HIL-suppressed above; on this axis they stay full - tools/metrics.py runs as + # the `tinyusb_metrics` build target, so a break in it fails the build + s.force_full(f'{path}: metrics tooling runs in the build -> full build matrix') + return + if _FULL_RE.match(path): # rules 15-16 + # attribution, not behaviour: these already reached `full` through the + # fall-through below. Naming them means a future narrowing of rule 17 cannot + # silently change what they do. Deliberately last, so every earlier rule keeps + # priority - examples/device/board_test is rule 14 (just board_test), not ALL. + s.force_full(f'{path}: core/infra -> full build matrix') + return + s.force_full(f'{path}: unclassified -> full build matrix') # rule 17 @contextlib.contextmanager -- cgit v1.3.1 From da255b1d2db10b8f31332a779b2a526f579acee1 Mon Sep 17 00:00:00 2001 From: Ha Thach Date: Tue, 25 Aug 2026 09:46:42 +0700 Subject: ci: an empty selection must build nothing, plus selector follow-ups (#3845) ci: an empty selection must build nothing, plus selector follow-ups A PR whose build axis legitimately selected nothing rebuilt everything. build.yml reads .build.families twice - as a |-joined regex, and implicitly as "is anything selected" - but tested only -z "$FAMILY_REGEX", which an empty list and a charset-rejected one both satisfy while meaning opposite things. ci_set_matrix had already returned the correct all-empty matrix; the fall-open branch discarded it. #3842 and #3840 each spent 74 cmake legs on it. Branch on the two cases instead, rename FAM_* to FAMILY_*, and cover the block with a test that extracts it from build.yml and executes it - it had no test at all, which is how this shipped through two merges. Follow-ups to the same machinery: glob.escape the repo root at five sites, so a checkout path containing [ or * stops failing closed; drop the ci-full label, read after the matrix was already computed and so never functional; delete 13 mcu:MKL25ZXX / mcu:SAME5X skip tokens matching no board; carry the rule table in the module docstring, guarded against drift; and pin six selection behaviours a mutation pass proved untested. Cut the selector's cost 1.8x (26.0s -> 14.6s) with 0 divergences over 260 paths, and stop scoping the membrowse upload by the PR example filter. --- .github/scripts/ci_set_matrix.py | 8 +- .github/workflows/build.yml | 35 ++-- .github/workflows/build_util.yml | 13 +- docs/reference/hil_boards.md | 2 +- .../2026-08-19-ci-build-family-filter-design.md | 13 ++ examples/device/audio_4_channel_mic/skip.txt | 1 - .../device/audio_4_channel_mic_freertos/skip.txt | 1 - examples/device/audio_test/skip.txt | 1 - examples/device/audio_test_freertos/skip.txt | 1 - examples/device/audio_test_multi_rate/skip.txt | 1 - examples/device/cdc_msc_freertos/skip.txt | 1 - examples/device/cdc_uac2/skip.txt | 1 - examples/device/hid_composite_freertos/skip.txt | 1 - examples/device/midi_test_freertos/skip.txt | 1 - examples/device/msc_dual_lun/skip.txt | 1 - examples/device/uac2_headset/skip.txt | 1 - examples/device/uac2_speaker_fb/skip.txt | 1 - test/hil/test/test_ci_metrics.py | 126 ++++++++++++- test/hil/test/test_ci_select.py | 204 ++++++++++++++++++++- tools/build.py | 4 +- tools/ci_select.py | 130 ++++++++++--- 21 files changed, 466 insertions(+), 81 deletions(-) (limited to 'docs/superpowers/specs') diff --git a/.github/scripts/ci_set_matrix.py b/.github/scripts/ci_set_matrix.py index 79f466893..409e6dbc1 100755 --- a/.github/scripts/ci_set_matrix.py +++ b/.github/scripts/ci_set_matrix.py @@ -131,7 +131,13 @@ def set_matrix_json(select=None): # a family this file does not list builds on no toolchain, so it contributes no # leg. hw/bsp holds several CI has never built (efm32, py32f0, same7x, ...) plus # espressif, whose boards hil-build-esp builds by name. - unbuilt = sorted(f for f in sel_fams if f not in family_list) + # espressif is not a gap: its examples need the ESP-IDF environment + # (CLAUDE.md: `. "$IDF_PATH/export.sh"` before any build), which the cmake legs + # do not have - that is why it is commented out of family_list above. Its + # coverage comes from hil-build-esp, which builds those boards BY NAME in an IDF + # container, so an espressif-only PR is already validated and falling open to the + # full matrix would add 74 legs, none of which can compile espressif. + unbuilt = sorted(f for f in sel_fams if f not in family_list and f != 'espressif') if unbuilt and not any(matrix.values()): # NONE of the selected families is buildable here, so every leg would skip # and the PR would go green from a build job that ran no compiler. That is diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 39a4e7afd..c26fe5cf8 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -68,14 +68,9 @@ jobs: with: fetch-depth: 0 - # The `ci-full` PR label turns the scoping off for one PR: no selection file is - # written, so both matrices and every rig job fall back to the unscoped behaviour. - # An escape hatch is the point - a selector bug under-selects SILENTLY, and without - # a label the only routes back to a full matrix are accidental (touch an - # unclassified path, or break the selector badly enough that it falls open). - name: CI selection (PR only) id: hil-select - if: github.event_name == 'pull_request' && !contains(github.event.pull_request.labels.*.name, 'ci-full') + if: github.event_name == 'pull_request' env: BASE_REF: ${{ github.base_ref }} run: | @@ -179,29 +174,43 @@ jobs: # treats false like null, so .build.full is compared explicitly. EXAMPLE_MAP='{}' BUILD_FILTERED='false' - FAM_REGEX='' + FAMILY_REGEX='' if [ -n "$BUILD_SELECT_FILE" ]; then EXAMPLE_MAP=$(jq -c '.build.family_examples // {}' "$BUILD_SELECT_FILE") || EXAMPLE_MAP='{}' BUILD_FILTERED=$(jq -r 'if (.build? | type) == "object" and .build.full == false then "true" else "false" end' "$BUILD_SELECT_FILE") || BUILD_FILTERED='false' if [ "$BUILD_FILTERED" = "true" ]; then - FAM_REGEX=$(jq -r '.build.families | join("|")' "$BUILD_SELECT_FILE") || FAM_REGEX='' + FAMILY_COUNT=$(jq -r '.build.families | length' "$BUILD_SELECT_FILE") || FAMILY_COUNT=0 + FAMILY_REGEX=$(jq -r '.build.families | join("|")' "$BUILD_SELECT_FILE") || FAMILY_REGEX='' # family names come from hw/bsp dir names, which rule 6 reads straight out # of the PR's diff path - and this is interpolated raw into a # `name_is_regexp` artifact pattern, so a regex metacharacter there would # silently match another family's baseline - case "$FAM_REGEX" in + FAMILY_REJECTED=0 + case "$FAMILY_REGEX" in *[!-A-Za-z0-9_\|]*) echo "::warning::unexpected characters in the family list - dropping the scoping" - FAM_REGEX='' ;; + FAMILY_REGEX=''; FAMILY_REJECTED=1 ;; esac - if [ -z "$FAM_REGEX" ]; then - # all three drop together, as CircleCI's fall-open does. Resetting only + # An EMPTY families list and a REJECTED one both leave FAMILY_REGEX empty and + # mean opposite things, so branch on which happened. Testing `-z` alone sent + # every nothing-selected PR down the fall-open path: a docs/.gitignore diff + # (#3842) and a test/hil-only diff (#3840) each rebuilt all 74 cmake legs + # after the selector had correctly chosen none. + if [ "$FAMILY_REJECTED" = "1" ]; then + # unusable: fall open, and all three drop together. Resetting only # build_filtered leaves the build scoped while code-metrics takes the # UNSCOPED branch, diffing a 1-family run against the full averaged # baseline and publishing that as the PR's code-size impact. BUILD_FILTERED='false' EXAMPLE_MAP='{}' MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py) + elif [ "$FAMILY_COUNT" = "0" ]; then + # legitimate nothing-selected. MATRIX_JSON already holds the all-empty + # matrix ci_set_matrix produced from this selection - keep it, so every + # leg skips. Nothing is built, so there is nothing to compare a baseline + # against: build_filtered goes false to keep code-metrics off the scoped + # path, and EXAMPLE_MAP stays '{}' (family_examples is empty anyway). + BUILD_FILTERED='false' fi fi fi @@ -210,7 +219,7 @@ jobs: echo "matrix=$MATRIX_JSON" >> $GITHUB_OUTPUT echo "example_map=$EXAMPLE_MAP" >> $GITHUB_OUTPUT echo "build_filtered=$BUILD_FILTERED" >> $GITHUB_OUTPUT - echo "build_families_regex=$FAM_REGEX" >> $GITHUB_OUTPUT + echo "build_families_regex=$FAMILY_REGEX" >> $GITHUB_OUTPUT # HIL matrix (merged from tinyusb + hifiphile configs), scoped on PRs. # Scoping is best-effort too: fall back to the unscoped (full) matrix. diff --git a/.github/workflows/build_util.yml b/.github/workflows/build_util.yml index 52999616d..407ed1e71 100644 --- a/.github/workflows/build_util.yml +++ b/.github/workflows/build_util.yml @@ -126,16 +126,11 @@ jobs: MEMBROWSE_API_KEY: ${{ secrets.MEMBROWSE_API_KEY }} run: | # if code-changed is false --> there is no elf -> membrowse target upload with --identical flag - # $EX_ARGS is passed for the BOARD it picks, not to scope the targets: - # --one-first now chooses a board that can build the -e set (tools/build.py), - # so omitting it here would configure a DIFFERENT, empty build dir and upload - # --identical for a board that was never compiled. The target list is not - # scoped by it - `examples-membrowse-upload` is not `all`, so it passes - # through as the aggregate, which has no DEPENDS (hw/bsp/family_support.cmake): - # it rebuilds nothing and still records every example, --identical for the - # ones without an elf. + # deliberately unscoped by $EX_ARGS: keeps the size history on a stable board + # per family, at the cost of an --identical-only upload where that board is not + # the one the Build step picked (test_ci_metrics pins which families those are) BUILD_PY_ARGS="-s ${{ inputs.build-system }} ${{ steps.setup-toolchain.outputs.build_option }} ${{ inputs.build-options }}" - python tools/build.py $BUILD_PY_ARGS --target examples-membrowse-upload -j 1 ${{ matrix.arg }} $EX_ARGS + python tools/build.py $BUILD_PY_ARGS --target examples-membrowse-upload -j 1 ${{ matrix.arg }} shell: bash - name: Upload Artifacts for Metrics diff --git a/docs/reference/hil_boards.md b/docs/reference/hil_boards.md index e8f364646..678f7f0ed 100644 --- a/docs/reference/hil_boards.md +++ b/docs/reference/hil_boards.md @@ -12,7 +12,7 @@ | espressif_s3_devkitm | device, host | esptool | espressif_s3_devkitm, espressif_s3_devkitm-DMA | Use TS3USB30 mux to test both device and host | | feather_nrf52840_express | device | jlink | | | | max32666fthr | device | openocd | | | -| metro_m4_express | device, dual | jlink | | pl23x; audio_test_freertos skipped: samd51 iso-IN capture fails (arecord EIO) | +| metro_m4_express | device, dual | jlink | metro_m4_express | pl23x; audio_test_freertos skipped: samd51 iso-IN capture fails (arecord EIO) | | lpcxpresso11u37 | device | jlink | | | | lpcxpresso55s28 | device | jlink | | | | ra4m1_ek | device | jlink | | | diff --git a/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md b/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md index b10f5b4ae..524568aeb 100644 --- a/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md +++ b/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md @@ -146,6 +146,19 @@ the existing `test_hil_util.BottomLayer` structural tests. Fail-open survives where it belongs: an *unclassified* path or any exception widens to `ALL` on every axis. +### A class no example enables selects nothing + +`src/class/bth` is the live instance: no example's `tusb_config.h` sets `CFG_TUD_BTH`, so +rules 8-10 resolve to no examples and a bth-only PR builds nothing and runs nothing. That is +the empty-means-empty ruling applied to classes, and it is deliberate — nothing compiles the +file, so nothing can validate it, and the master-push build is the net. + +Worth stating plainly because the exposure changed: GHA used to rebuild everything for such +a PR by accident, through the empty-`families` bug in `build.yml`. With that fixed, both +providers now correctly build nothing, so `tud_bt_*` can be broken by a green PR. +`TestClassesWithNoEnablingExample` pins the set to `{bth}` so a second class cannot enter +this state unnoticed. + ### Why `hw/mcu/**` is rule 7 and not "full" `hw/mcu` is overwhelmingly dependency territory — `tools/get_deps.py` has 87 entries under it, diff --git a/examples/device/audio_4_channel_mic/skip.txt b/examples/device/audio_4_channel_mic/skip.txt index 3ca433c08..e5e74cd60 100644 --- a/examples/device/audio_4_channel_mic/skip.txt +++ b/examples/device/audio_4_channel_mic/skip.txt @@ -1,5 +1,4 @@ mcu:SAMD11 -mcu:SAME5X mcu:SAMG family:broadcom_64bit family:espressif diff --git a/examples/device/audio_4_channel_mic_freertos/skip.txt b/examples/device/audio_4_channel_mic_freertos/skip.txt index 1fd6b4b8a..cfde51051 100644 --- a/examples/device/audio_4_channel_mic_freertos/skip.txt +++ b/examples/device/audio_4_channel_mic_freertos/skip.txt @@ -7,7 +7,6 @@ mcu:CXD56 mcu:F1C100S mcu:GD32VF103 mcu:MCXA15 -mcu:MKL25ZXX mcu:MSP430x5xx mcu:FT90X mcu:SAMD11 diff --git a/examples/device/audio_test/skip.txt b/examples/device/audio_test/skip.txt index 42394bb11..862c91c6f 100644 --- a/examples/device/audio_test/skip.txt +++ b/examples/device/audio_test/skip.txt @@ -1,5 +1,4 @@ mcu:SAMD11 -mcu:SAME5X mcu:SAMG family:espressif mcu:CH583 diff --git a/examples/device/audio_test_freertos/skip.txt b/examples/device/audio_test_freertos/skip.txt index 660bacd25..3d8d43286 100644 --- a/examples/device/audio_test_freertos/skip.txt +++ b/examples/device/audio_test_freertos/skip.txt @@ -7,7 +7,6 @@ mcu:CXD56 mcu:F1C100S mcu:GD32VF103 mcu:MCXA15 -mcu:MKL25ZXX mcu:MSP430x5xx mcu:FT90X mcu:SAMD11 diff --git a/examples/device/audio_test_multi_rate/skip.txt b/examples/device/audio_test_multi_rate/skip.txt index 42394bb11..862c91c6f 100644 --- a/examples/device/audio_test_multi_rate/skip.txt +++ b/examples/device/audio_test_multi_rate/skip.txt @@ -1,5 +1,4 @@ mcu:SAMD11 -mcu:SAME5X mcu:SAMG family:espressif mcu:CH583 diff --git a/examples/device/cdc_msc_freertos/skip.txt b/examples/device/cdc_msc_freertos/skip.txt index 48781de84..095e350c9 100644 --- a/examples/device/cdc_msc_freertos/skip.txt +++ b/examples/device/cdc_msc_freertos/skip.txt @@ -7,7 +7,6 @@ mcu:CXD56 mcu:F1C100S mcu:GD32VF103 mcu:MCXA15 -mcu:MKL25ZXX mcu:MSP430x5xx mcu:FT90X mcu:SAMD11 diff --git a/examples/device/cdc_uac2/skip.txt b/examples/device/cdc_uac2/skip.txt index db1d5b80b..3159cb176 100644 --- a/examples/device/cdc_uac2/skip.txt +++ b/examples/device/cdc_uac2/skip.txt @@ -2,7 +2,6 @@ mcu:LPC11UXX mcu:LPC13XX mcu:NUC121 mcu:SAMD11 -mcu:SAME5X mcu:SAMG board:stm32l052dap52 family:espressif diff --git a/examples/device/hid_composite_freertos/skip.txt b/examples/device/hid_composite_freertos/skip.txt index 97d8e168b..0e8415d3b 100644 --- a/examples/device/hid_composite_freertos/skip.txt +++ b/examples/device/hid_composite_freertos/skip.txt @@ -7,7 +7,6 @@ mcu:CXD56 mcu:F1C100S mcu:GD32VF103 mcu:MCXA15 -mcu:MKL25ZXX mcu:MSP430x5xx mcu:FT90X mcu:SAMD11 diff --git a/examples/device/midi_test_freertos/skip.txt b/examples/device/midi_test_freertos/skip.txt index 97d8e168b..0e8415d3b 100644 --- a/examples/device/midi_test_freertos/skip.txt +++ b/examples/device/midi_test_freertos/skip.txt @@ -7,7 +7,6 @@ mcu:CXD56 mcu:F1C100S mcu:GD32VF103 mcu:MCXA15 -mcu:MKL25ZXX mcu:MSP430x5xx mcu:FT90X mcu:SAMD11 diff --git a/examples/device/msc_dual_lun/skip.txt b/examples/device/msc_dual_lun/skip.txt index a9e3a99b1..833fd072c 100644 --- a/examples/device/msc_dual_lun/skip.txt +++ b/examples/device/msc_dual_lun/skip.txt @@ -1,3 +1,2 @@ mcu:SAMD11 -mcu:MKL25ZXX family:espressif diff --git a/examples/device/uac2_headset/skip.txt b/examples/device/uac2_headset/skip.txt index db1d5b80b..3159cb176 100644 --- a/examples/device/uac2_headset/skip.txt +++ b/examples/device/uac2_headset/skip.txt @@ -2,7 +2,6 @@ mcu:LPC11UXX mcu:LPC13XX mcu:NUC121 mcu:SAMD11 -mcu:SAME5X mcu:SAMG board:stm32l052dap52 family:espressif diff --git a/examples/device/uac2_speaker_fb/skip.txt b/examples/device/uac2_speaker_fb/skip.txt index 0c7339c65..88df3e549 100644 --- a/examples/device/uac2_speaker_fb/skip.txt +++ b/examples/device/uac2_speaker_fb/skip.txt @@ -2,7 +2,6 @@ mcu:LPC11UXX mcu:LPC13XX mcu:NUC121 mcu:SAMD11 -mcu:SAME5X mcu:SAMG board:stm32l052dap52 family:broadcom_64bit diff --git a/test/hil/test/test_ci_metrics.py b/test/hil/test/test_ci_metrics.py index a76b6e3a0..aac251824 100644 --- a/test/hil/test/test_ci_metrics.py +++ b/test/hil/test/test_ci_metrics.py @@ -447,16 +447,126 @@ class TestWorkflowSelectionHandOff(unittest.TestCase): self.assertIn('UNSCOPED', flat[max(0, i - 200):i], 'a fall-open path without the marker build.yml greps for') - def test_membrowse_upload_sees_the_same_board_as_the_build(self): - # $EX_ARGS is passed for the BOARD it selects: --one-first picks a board that can - # build the -e set, so without it membrowse configures a different, empty build - # dir and uploads --identical for a board that was never compiled. It does NOT - # scope the targets - `examples-membrowse-upload` is not `all`, so it passes - # through as the aggregate, which has no DEPENDS and still records every example. + def _run_extras_block(self, sel): + """Extract the build-extras shell block from build.yml and run it for real. + Nothing else exercises it, which is why the empty/rejected conflation shipped.""" + import re as _re, shlex, subprocess, tempfile, json as _json + repo = os.path.dirname(CIRCLECI) + i = self.build.index("EXAMPLE_MAP='{}'\n BUILD_FILTERED='false'") + i = self.build.rindex('\n', 0, i) + 1 + j = self.build.index(' echo "matrix=$MATRIX_JSON"', i) + block = _re.sub(r'^ {10}', '', self.build[i:j], flags=_re.M) + with tempfile.TemporaryDirectory() as d: + selp = os.path.join(d, 'sel.json') + with open(selp, 'w') as fh: + _json.dump(sel, fh) + matrix = subprocess.run( + [sys.executable, os.path.join(repo, '.github/scripts/ci_set_matrix.py'), + '--select-file', selp], capture_output=True, text=True, cwd=repo).stdout.strip() + self.assertTrue(matrix, 'ci_set_matrix produced nothing') + sh = os.path.join(d, 'probe.sh') + with open(sh, 'w') as fh: + # shlex.quote, not hand-rolled quoting: a TMPDIR with a space in it + # made this fail for a reason that had nothing to do with the block + fh.write('BUILD_SELECT_FILE=' + shlex.quote(selp) + '\n') + fh.write('MATRIX_JSON=' + shlex.quote(matrix) + '\n') + fh.write(block) + # sentinel + newline separated: the block itself writes ::warning:: to + # stdout, and '|' would collide with the regex's own separator + fh.write('\nprintf "@@R@@\\n%s\\n%s\\n%s" "$MATRIX_JSON" "$BUILD_FILTERED" "$FAMILY_REGEX"\n') + r = subprocess.run(['bash', sh], capture_output=True, text=True, cwd=repo) + self.assertEqual(r.returncode, 0, r.stderr) + mj, filtered, regex = r.stdout.split('@@R@@\n', 1)[1].split('\n', 2) + return sum(len(v) for v in _json.loads(mj).values()), filtered, regex + + def test_an_empty_family_list_is_not_treated_as_unusable(self): + """.build.families is read twice - as a count and as a `|`-joined regex. An EMPTY + list and one REJECTED by the charset guard both leave the regex empty and mean + opposite things, so the block has to branch on which happened. + + Testing `-z "$FAMILY_REGEX"` alone sent every nothing-selected PR down the + fall-open path and discarded the correct all-empty matrix: #3842 (docs + + .gitignore) and #3840 (test/hil only) each rebuilt all 74 cmake legs after the + selector had correctly chosen none.""" + legs, filtered, regex = self._run_extras_block( + {'build': {'full': False, 'families': [], 'family_examples': {}}}) + self.assertEqual(legs, 0, 'an empty families list must keep the all-empty matrix') + self.assertEqual(filtered, 'false', 'nothing was built, so nothing to compare') + self.assertEqual(regex, '') + + def test_a_real_family_list_stays_scoped(self): + legs, filtered, regex = self._run_extras_block( + {'build': {'full': False, 'families': ['stm32f4', 'rp2040'], + 'family_examples': {}}}) + self.assertGreater(legs, 0) + self.assertEqual(filtered, 'true') + self.assertEqual(regex, 'stm32f4|rp2040') + + def test_a_regex_metacharacter_in_a_family_name_falls_open(self): + # the name is interpolated raw into a name_is_regexp artifact pattern, so a + # metacharacter would match another family's baseline - reject and widen + legs, filtered, regex = self._run_extras_block( + {'build': {'full': False, 'families': ['stm32f4.*'], 'family_examples': {}}}) + self.assertGreater(legs, 100, 'a rejected family list must fall open to full') + self.assertEqual(filtered, 'false') + self.assertEqual(regex, '') + + def test_membrowse_upload_is_not_scoped_by_the_pr_filter(self): + # by decision, the upload runs unfiltered so the size history stays keyed on the + # family's preferred board whatever the PR touched. $EX_ARGS would not have + # scoped the targets either way - `examples-membrowse-upload` is not `all`, so + # resolve_example_target_groups passes it through as the aggregate - but it DID + # move the board, because --one-first picks one that can build the -e set. + # + # The accepted cost: on a family whose preferred board cannot build that set, + # the upload lands on a board the Build step never compiled and every example + # goes up --identical. test_the_upload_board_can_diverge_from_the_built_board + # keeps that consequence measured rather than assumed. line = [l for l in self.util.splitlines() if '--target examples-membrowse-upload' in l][0] - self.assertIn('$EX_ARGS', line) - self.assertNotIn('-e ', line.replace('$EX_ARGS', '')) + self.assertNotIn('$EX_ARGS', line) + self.assertNotIn('-e ', line) + + def test_the_upload_board_can_diverge_from_the_built_board(self): + """Pins the SIZE of what the removal gave up, so it cannot grow unnoticed. + + --one-first with no -e returns preferred_list[0]; with one it returns the first + preferred board that can build it. Where those differ, the Membrowse Upload step + configures a build dir the Build step never wrote.""" + sys.path.insert(0, os.path.join(REPO, 'tools')) + import build as build_py + roles = ('device', 'host', 'dual') + exs = sorted(f'{r}/{n}' for r in roles + for n in os.listdir(os.path.join(REPO, 'examples', r)) + if os.path.isdir(os.path.join(REPO, 'examples', r, n))) + fams = sorted(d for d in os.listdir(os.path.join(REPO, 'hw/bsp')) + if os.path.isdir(os.path.join(REPO, 'hw/bsp', d, 'boards'))) + cwd = os.getcwd() + os.chdir(REPO) + try: + diverging = set() + for fam in fams: + try: + base = build_py.get_family_boards(fam, False, True, None, 'cmake', ()) + except Exception: + continue + if not base: + continue + for e in exs: + try: + one = build_py.get_family_boards(fam, False, True, [e], 'cmake', ()) + except Exception: + continue + if one and one[0] != base[0]: + diverging.add(fam) + break + finally: + os.chdir(cwd) + self.assertEqual(diverging, {'imxrt', 'lpc11', 'lpc18', 'lpc54', 'mcx', 'rp2040', + 'rx', 'samd11', 'stm32l0', 'stm32l4', 'tm4c'}, + 'the set of families whose membrowse upload can land on an ' + 'uncompiled board changed; re-check whether dropping $EX_ARGS ' + 'from the upload step is still the right trade') if __name__ == '__main__': diff --git a/test/hil/test/test_ci_select.py b/test/hil/test/test_ci_select.py index a19392bde..dc10f769a 100644 --- a/test/hil/test/test_ci_select.py +++ b/test/hil/test/test_ci_select.py @@ -302,7 +302,11 @@ class TestArgsEmission(unittest.TestCase): out = j.loads(r.stdout) self.assertFalse(out['full']) self.assertIn('tinyusb.json', out['args']) - self.assertTrue(any('cdc_device' in line for line in out['reasons'])) + # reasons are a stderr diagnostic, deliberately NOT in the payload: they were + # 97% of a 9.8 MB JSON on a dep bump, and every consumer re-parses that file + self.assertNotIn('reasons', out, 'reasons must not ride in the machine-read JSON') + self.assertNotIn('reasons', out['build']) + self.assertIn('cdc_device', r.stderr) # A core-class diff must select boards THROUGH THE CLI: the in-process tests # inject their own repo root, so only this subprocess path catches a broken # repo_root derivation -- which once made every repo-relative glob match @@ -938,6 +942,181 @@ class TestClassesWithNoEnablingExample(unittest.TestCase): 'both axes, so nothing compiles it until the next master push') +class TestExampleMapOmitsFullFamilies(unittest.TestCase): + """A family whose selection is ALREADY everything it can build carries no -e list. + + Sixth of the same shape as the class below, found the same way: a perf rewrite of + _prune_buildable dropped the `set(kept) != set(buildable)` test and all 216 tests + stayed green. The build outcome is identical either way -- build.py applies the same + skip_example the pruner just did -- so nothing compiled differently and only the + payload grew (22 families x 33 examples on one dcd_dwc2.c diff). That is exactly the + kind of drift no build failure ever reports.""" + + def test_a_device_only_port_diff_still_omits_families_it_cannot_narrow(self): + # dcd_dwc2.c selects device+dual examples only, but a family whose host examples + # are all unbuildable anyway ends up wanting its entire buildable set + b = ci_select.classify_build(['src/portable/synopsys/dwc2/dcd_dwc2.c'], REPO) + self.assertFalse(b['full']) + self.assertTrue(b['families']) + omitted = [f for f in b['families'] if f not in b['family_examples']] + self.assertTrue(omitted, 'no family omitted its -e list; the "already everything ' + 'this family builds" case stopped being detected') + for fam in omitted: + self.assertNotIn(fam, b['family_examples']) + + def test_a_family_that_can_build_more_than_the_diff_wants_keeps_its_list(self): + # the other direction: one example selects itself and nothing else, so every + # family it lands on must carry an explicit -e or CI builds all 46 + b = ci_select.classify_build(['examples/device/cdc_msc/src/main.c'], REPO) + self.assertFalse(b['full']) + for fam in b['families']: + self.assertEqual(b['family_examples'].get(fam), ['device/cdc_msc'], fam) + + +class TestSelectionBehavioursThatHadNoTest(unittest.TestCase): + """Five behaviours a reviewer's mutation pass proved were unpinned: break each one + and the whole suite stayed green. Each test here fails against its mutant. + + They are grouped because they share a shape - every one is a small expression whose + removal silently NARROWS the selection, which is the failure direction that merges a + regression rather than wasting a runner.""" + + def test_build_defines_reach_the_prefilter(self): + # mutant: `defines = ()` in build.py's build_boards_list. metro_m4_express gets + # MAX3421_HOST=1 from its roster variant, never from its BSP, so without the + # defines the -e prefilter drops the rig's only MAX3421 firmware and hil-tinyusb + # has nothing to flash. + import build as build_py, build_utils, inspect + src = inspect.getsource(build_py.build_boards_list) + self.assertIn('defines = tuple(sorted(build_defines))', src, + 'the -D tokens must reach cmake_board/skip_example') + old = os.getcwd() + os.chdir(REPO) + try: + ex, board = 'dual/host_info_to_device_cdc', 'metro_m4_express' + self.assertTrue(build_utils.skip_example(ex, board), + 'without the define this example is correctly skipped') + self.assertFalse(build_utils.skip_example(ex, board, ('MAX3421_HOST=1',)), + 'with it, it must build - that is what the roster passes') + finally: + os.chdir(old) + + def test_one_first_prefers_a_board_that_can_build_the_filter(self): + # mutant: buildable() -> True, i.e. back to all_boards[0]. lpc54's first board + # skips every msc_file_explorer example, so the leg would compile nothing. + import build as build_py + old_env, old = os.environ.get('GITHUB_ACTIONS'), os.getcwd() + os.environ['GITHUB_ACTIONS'] = 'true' + os.chdir(REPO) + try: + unfiltered = build_py.get_family_boards('lpc54', False, True) + filtered = build_py.get_family_boards('lpc54', False, True, + ['host/msc_file_explorer']) + self.assertEqual(unfiltered, ['lpcxpresso54114'], 'unfiltered pick must not move') + self.assertNotEqual(filtered, unfiltered, + 'the -e pick must avoid a board that skips the whole filter') + import build_utils + self.assertFalse(build_utils.skip_example('host/msc_file_explorer', filtered[0]), + f'{filtered[0]} must actually build the filtered example') + finally: + os.chdir(old) + if old_env is None: + os.environ.pop('GITHUB_ACTIONS', None) + else: + os.environ['GITHUB_ACTIONS'] = old_env + + def test_a_class_file_selects_its_own_macro_not_just_the_directory(self): + # mutant: delete the _CLS_STEM_RE block. src/class/midi holds MIDI 1.0 AND 2.0; + # examples/device/midi2_device is the only example enabling CFG_TUD_MIDI2 and the + # only one that compiles midi2_device.c, but the directory macro alone misses it. + got = ci_select._build_class_examples('midi', 'midi2_device.c', {'device'}, REPO) + self.assertIn('device/midi2_device', got, + 'a midi2 change must select the example that compiles it') + host = ci_select._build_class_examples('midi', 'midi2_host.c', {'host'}, REPO) + self.assertIn('host/midi2_host', host) + # and the plain midi files must NOT drag midi2 in + plain = ci_select._build_class_examples('midi', 'midi_device.c', {'device'}, REPO) + self.assertNotIn('device/midi2_device', plain) + + def test_a_port_change_selects_the_dual_examples(self): + # mutant: drop `+ ('dual',)`. A dcd/hcd change must build the dual examples - + # they exercise both stacks on one board, so a dwc2 break lands there first. + s = ci_select.classify_build(['src/portable/synopsys/dwc2/dcd_dwc2.c'], REPO) + duals = {e for exs in s['family_examples'].values() for e in exs + if e.startswith('dual/')} + self.assertTrue(duals, 'a dcd change selected no dual example') + + def test_the_selector_answers_the_same_with_and_without_ci_env(self): + # mutant: drop ci=True from _prune_buildable. ci_skip_boards/ci_preferred_boards + # only apply when GITHUB_ACTIONS/CIRCLECI is set, so without the pin a laptop and + # a runner disagree - and /pre-pr would report a family list CI will not build. + files = ['examples/host/cdc_msc_hid_freertos/src/main.c'] + old = os.environ.get('GITHUB_ACTIONS') + os.environ.pop('GITHUB_ACTIONS', None) + try: + local = ci_select.classify_build(files, REPO)['families'] + os.environ['GITHUB_ACTIONS'] = 'true' + import importlib + importlib.reload(ci_select) + runner = ci_select.classify_build(files, REPO)['families'] + finally: + if old is None: + os.environ.pop('GITHUB_ACTIONS', None) + else: + os.environ['GITHUB_ACTIONS'] = old + import importlib + importlib.reload(ci_select) + self.assertEqual(local, runner, 'the selector must not depend on the CI env vars') + + +class TestRuleTableIsCarbonOfTheSpec(unittest.TestCase): + """ci_select's module docstring carries the rule table so a reader landing in the + code does not have to open the spec to learn what rule 6 is. Both are maintained by + hand, so this pins them cell-for-cell: edit one without the other and this fails. + + It also pins the table against the CODE - every rule id the docstring claims must + appear as a `# rule N` marker on a branch of _classify_build_one, so a row cannot be + documented without a branch, or a branch renumbered without the table.""" + + @staticmethod + def _rows(text): + import re as _re + out = [] + for l in text.splitlines(): + if not l.startswith('| '): + continue + c = [x.strip() for x in l.strip().strip('|').split('|')] + if len(c) == 5 and _re.fullmatch(r'\d+[a-z]?', c[0]): + out.append(c) + return out + + def test_docstring_table_matches_the_spec(self): + spec = open(os.path.join( + REPO, 'docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md')).read() + doc, spec_rows = self._rows(ci_select.__doc__), self._rows(spec) + self.assertTrue(spec_rows, 'no rule table found in the spec') + self.assertEqual([r[0] for r in doc], [r[0] for r in spec_rows], + 'rule ids differ between ci_select.__doc__ and the spec') + for d, s in zip(doc, spec_rows): + self.assertEqual(d, s, f'rule {d[0]} differs between the docstring and the spec') + + def test_every_documented_rule_has_a_branch(self): + import re as _re + src = open(os.path.join(REPO, 'tools/ci_select.py')).read() + marked = set() + # handles `# rule 6`, `# rules 1, 1b` and `# rules 8-10` + for m in _re.finditer(r'#\s*rules?\s+([0-9a-z, -]+)', src): + for tok in _re.split(r',\s*', m.group(1).strip()): + rng = _re.fullmatch(r'(\d+)\s*-\s*(\d+)', tok.strip()) + if rng: + marked.update(str(n) for n in range(int(rng.group(1)), int(rng.group(2)) + 1)) + elif _re.fullmatch(r'\d+[a-z]?', tok.strip()): + marked.add(tok.strip()) + documented = {r[0] for r in self._rows(ci_select.__doc__)} + missing = sorted(documented - marked, key=lambda s: (int(_re.match(r'\d+', s).group()), s)) + self.assertEqual(missing, [], f'documented rules with no `# rule N` branch marker: {missing}') + + class TestNoTrackedFileIsUnclassified(unittest.TestCase): """Rule 17 (unclassified -> full on both axes) is the fail-open net for paths nobody anticipated. It must stay that way - a wrong `full` costs runner minutes and is @@ -1477,10 +1656,17 @@ class TestBuildPostFilter(unittest.TestCase): # the accepted net for a break outside its #if guard). src/class/bth is the # live instance of this state today; TestClassesWithNoEnablingExample pins the # whole set, so a new one cannot appear unnoticed. - s = ci_select.classify_build(['src/class/vendor/vendor_host.c'], REPO) + # src/class/bth/bth_device.c, a file that EXISTS: the old assertion named + # src/class/vendor/vendor_host.c, deleted by the same branch, so any made-up + # path reached the same branch and the test passed vacuously. + real = os.path.join(REPO, 'src/class/bth/bth_device.c') + self.assertTrue(os.path.isfile(real), 'the case needs a file that exists') + s = ci_select.classify_build(['src/class/bth/bth_device.c'], REPO) self.assertFalse(s['full']) self.assertEqual(s['families'], []) self.assertTrue(any('no contribution' in r for r in s['reasons']), s['reasons']) + # and the reason must name the class, not just any empty answer + self.assertTrue(any('bth' in r for r in s['reasons']), s['reasons']) def test_class_source_with_examples_still_scopes(self): s = ci_select.classify_build(['src/class/cdc/cdc_device.c'], REPO) @@ -2040,14 +2226,14 @@ class TestMcuTokensResolve(unittest.TestCase): # produce, or a rename nobody followed through. `family:samd21` was one of these # until the nine examples/host/*/only.txt files were corrected to samd2x_l2x. # - # The `mcu:` entries are NOT all harmless. MIMXRT10XX/MIMXRT11XX and LPC177X_8X sit - # beside a live token in the same file, so they gate nothing either way. MKL25ZXX - # (device/msc_dual_lun) and SAME5X (device/audio_test) do not: those skips are dead, - # and both examples are built today on the boards their skip file meant to exclude - - # successfully, which is why nobody noticed. Correcting them REMOVES working build - # coverage, so it is a maintainer call, not a drive-by fix. + # The remaining `mcu:` entries sit beside a live token in the same file, so they gate + # nothing either way. MKL25ZXX (7 files) and SAME5X (1) were dead too, but unlike + # these they were the ONLY token for their board - the examples were already being + # built on the very boards those lines meant to exclude. Dropping them is a no-op for + # the build (verified per example) and was chosen over re-pointing, which would have + # removed working coverage. UNREACHABLE_TOKENS = { - 'mcu': {'LPC177X_8X', 'MIMXRT10XX', 'MIMXRT11XX', 'MKL25ZXX', 'SAME5X', 'STM32U3'}, + 'mcu': {'LPC177X_8X', 'MIMXRT10XX', 'MIMXRT11XX', 'STM32U3'}, 'family': set(), 'board': set(), } diff --git a/tools/build.py b/tools/build.py index eeefca22d..0bb366e3d 100755 --- a/tools/build.py +++ b/tools/build.py @@ -356,11 +356,11 @@ def get_family_boards(family, one_random, one_first, examples=None, build_system # the WHOLE preferred list, in order - stopping at entry one would abandon a # curated list for the raw alphabetical order the moment its first board cannot # build the filter, which also moves the board the metrics baseline is keyed on + # the whole preferred list, in order. Unreachable-when-unfiltered: with + # examples is None, buildable() is True and the loop returns on entry one. for b in preferred_list: if buildable(b): return [b] - if preferred_list and examples is None: - return [preferred_list[0]] candidates = [b for b in all_boards if buildable(b)] or all_boards if one_first: return [candidates[0]] diff --git a/tools/ci_select.py b/tools/ci_select.py index 89a0d214c..53fbcd3a1 100755 --- a/tools/ci_select.py +++ b/tools/ci_select.py @@ -13,6 +13,38 @@ JSON: full, boards (name -> 'all' | [tests]), families (bsp families the diff touches, including ones with no rig board - build-only consumers such as /pre-pr sample from these), args (hil_test.py args per config) and args_flasher (the same args split by each board's flasher, for CI legs that split one rig by flasher). + +THE RULE TABLE. First match wins; answers union per family (build) and per board +(HIL). A CARBON COPY of the table in the design spec above - edit both, or +TestRuleTableIsCarbonOfTheSpec fails. `FAM` = the families whose family.cmake +references the changed path (CMake only; make follows it). `DEV`/`HOST`/`DUAL`/ +`TYPEC`/`ALL` are the example role sets. The Build families column is PRE-PRUNE: +_prune_buildable then intersects each family with what it can actually build. + +| # | Changed path | Build families | Build examples | HIL boards → tests | +| 1 | `docs/`, `.claude/`, `*.md`, `*.rst`, `LICENSE` | — | — | — | +| 1b | `.gitignore`, `.clang-format`, `.idea/**`, `test/{fuzz,unit-test}/**`, non-build `.github/**`, packaging manifests | — | — | — | +| 2 | `test/hil/**` | — | — | all boards → all tests | +| 2b | `tools/metrics.py`, `.github/scripts/metrics_*.py` | `ALL` (unchanged — `tinyusb_metrics` runs `metrics.py` as a build target) | `ALL` | — (nothing on the rig runs it) | +| 3 | `src/portable//dcd_*`, `*_device.[ch]` | `FAM` | `DEV`+`DUAL` | `FAM`'s device-role boards → device+dual tests | +| 4 | `src/portable//hcd_*`, `*_host.[ch]` | `FAM` | `HOST`+`DUAL` | `FAM`'s host-role boards → host+dual tests | +| 5 | `src/portable//**` (anything else) | `FAM` | `ALL` | `FAM`'s boards → all their tests | +| 5b | `src/portable//**` where `FAM` is empty | — | — | — (empty resolves to nothing on BOTH axes) | +| 6 | `hw/bsp//**` | that family | `ALL` | that family's boards → all tests (a `boards//` path narrows to that board) | +| 7 | `hw/mcu//**` | `FAM` — empty resolves to nothing (maintainer ruling) | `ALL` | `FAM`'s boards → all tests; empty resolves to nothing (maintainer ruling) ⚠ *see below* | +| 8 | `src/class//*_device.[ch]` | `ALL` | examples enabling `CFG_TUD_` | device-role boards → HIL tests enabling `CFG_TUD_` | +| 9 | `src/class//*_host.[ch]` | `ALL` | examples enabling `CFG_TUH_` | host-role boards → HIL tests enabling `CFG_TUH_` | +| 10 | `src/class//**` (shared header) | `ALL` | either, **plus include-edge classes** | both roles → same, plus include-edge classes | +| 11 | `src/device/**` | `ALL` | `DEV`+`DUAL` | device-role boards → device+dual tests | +| 12 | `src/host/**` | `ALL` | `HOST`+`DUAL` | host-role boards → host+dual tests | +| 12b | `src/typec/**` | `ALL` | examples enabling `CFG_TUC_ENABLED` | — (no rig board runs a typec test) | +| 13 | `examples///**` | `ALL` | just `` | if `` is a HIL test: all boards → that test; else nothing | +| 14 | `examples/device/board_test/**` | `ALL` | just `board_test` | all boards → all tests (HIL parking firmware) | +| 15 | `examples/build_system/**`, `examples/CMakeLists.txt`, `examples//CMakeLists.txt` | `ALL` | `ALL` | all boards → all tests | +| 16 | `src/common/`, `src/osal/`, `src/tusb.[ch]`, `src/tusb_option.h`, `tools/{build,build_utils,ci_select}.py`, `tools/cmake/**`, `src/CMakeLists.txt`, `src/tinyusb.mk`, `hw/bsp/{family_support.{cmake,mk},family_rules.mk,zephyr_board_aliases.cmake,board.c,board_api.h,ansi_escape.h}`, `.github/**`, `.circleci/**` | `ALL` | `ALL` | all boards → all tests | +| 16a | `lib//**` | `ALL` | examples whose own `CMakeLists.txt`/`Makefile` names `lib/` | those examples that are HIL tests, on all boards; empty resolves to nothing | +| 16b | `tools/get_deps.py` | families whose `deps_mandatory`/`deps_optional` entries changed | `ALL` | those families' boards → all tests; a logic change, an `'all'` entry, no base content or a changed token naming no family → full | +| 17 | anything unclassified (no tracked file reaches this — TestNoTrackedFileIsUnclassified) | `ALL` | `ALL` | all boards → all tests (fail-open) | """ import argparse import ast @@ -53,7 +85,10 @@ def _read(path: str) -> str: _NONCODE_RE = re.compile( - r'^(docs/|\.claude/|.*\.(md|rst)$|LICENSE)') + # LICENSE is anchored and LICENSES/ named separately: a bare `LICENSE` alternative + # also swallowed anything merely STARTING with it (a future LICENSE_extra.c), + # which is the silent-under-selection direction + r'^(docs/|\.claude/|.*\.(md|rst)$|LICENSE$|LICENSES/)') # Repo metadata and tooling that no CI build reads. Enumerated rather than left to # rule 17, which widens BOTH axes: a PR touching only .gitignore and a README was # creating 74 cmake legs (each a runner doing checkout + toolchain + get_deps before @@ -152,10 +187,20 @@ def board_tests(board: dict) -> list: return [x for x in run if x not in t.get('skip', [])] + +def _rg(repo_root: str, *parts: str) -> str: + """A glob pattern rooted at repo_root, with the ROOT escaped and the parts left as + patterns. The root is a filesystem path, not a pattern: a checkout at + /w/pr[1]/tinyusb (a worktree named after a PR, a CI workspace with brackets) makes + an unescaped '[1]' a character class that matches nothing, and every lookup below + then resolves to zero - families=0 instead of 30, i.e. the selector fails CLOSED + and the whole matrix compiles nothing while reporting green.""" + return os.path.join(glob.escape(repo_root), *parts) + # cached: called per changed file x roster board, and the tree doesn't change mid-run @functools.lru_cache(maxsize=None) def board_family(board_name: str, repo_root: str): - hits = glob.glob(os.path.join(repo_root, 'hw/bsp/*/boards', board_name)) + hits = glob.glob(_rg(repo_root, 'hw/bsp/*/boards', board_name)) return os.path.basename(os.path.dirname(os.path.dirname(hits[0]))) if hits else None @@ -263,10 +308,10 @@ def _family_file_texts(repo_root: str) -> tuple: CMakeLists.txt, read once. path_families is called per distinct directory in the diff and its own cache only helps repeats: a 6,000-file hw/mcu dep bump re-read these 84 files 99,892 times (2.2 s) before this.""" - bsp_root = os.path.join(repo_root, 'hw/bsp') + bsp_root = os.path.join(repo_root, 'hw/bsp') # escaped by _rg below out = [] - for f in sorted(glob.glob(os.path.join(bsp_root, '*/family.cmake')) + - glob.glob(os.path.join(bsp_root, '*/components/*/CMakeLists.txt'))): + for f in sorted(glob.glob(_rg(bsp_root, '*/family.cmake')) + + glob.glob(_rg(bsp_root, '*/components/*/CMakeLists.txt'))): try: out.append((os.path.relpath(f, bsp_root).split(os.sep, 1)[0], _read(f))) except OSError: @@ -386,7 +431,7 @@ def class_include_edges(repo_root: str) -> dict: Derived from the actual #include lines rather than a hand-written table so it cannot rot when a class picks up or drops a cross-class include.""" edges = {} - for f in sorted(glob.glob(os.path.join(repo_root, 'src/class/*/*.[ch]'))): + for f in sorted(glob.glob(_rg(repo_root, 'src/class/*/*.[ch]'))): cls = os.path.basename(os.path.dirname(f)) try: text = _read(f) @@ -470,11 +515,23 @@ def _class_roles(base: str) -> set: return {'device', 'host'} -def _config_enables(cfg_path: str, macros) -> bool: +@functools.lru_cache(maxsize=None) +def _config_text(cfg_path: str) -> str: + """An example's tusb_config.h, read once. Every class path re-asks the same 46 + configs on both axes, so the reads go up with the diff: 4,240 of the same 46 files + for a diff touching all of src/class (0.48s -> 0.13s), and they cannot change + mid-run. Cached here rather than on _config_enables so the macros argument stays an + ordinary list at every call site.""" try: with open(cfg_path, encoding='utf-8', errors='replace') as f: - text = f.read() + return f.read() except OSError: + return '' + + +def _config_enables(cfg_path: str, macros) -> bool: + text = _config_text(cfg_path) + if not text: return False for m in macros: for value in re.findall(_DEF_VALUE.format(m), text, re.M): @@ -511,10 +568,13 @@ def lib_examples(lib_name: str, repo_root: str) -> set: pat = re.compile(re.escape('lib/' + lib_name) + r'(?=[/\s"\')}]|$)', re.M) out = set() for ex in all_examples(repo_root): - for f in sorted(glob.glob(os.path.join(repo_root, 'examples', ex, '**', '*'), + # the two filenames directly: '**/*' enumerated 489 entries per lib against a + # clean tree to use 107, and grows without bound once `make BOARD=... all` has + # written examples///_build/ - which is where /pre-pr runs + for f in sorted(glob.glob(_rg(repo_root, 'examples', ex, '**', 'CMakeLists.txt'), + recursive=True) + + glob.glob(_rg(repo_root, 'examples', ex, '**', 'Makefile'), recursive=True)): - if os.path.basename(f) not in ('CMakeLists.txt', 'Makefile'): - continue try: text = _read(f) except OSError: @@ -580,7 +640,7 @@ def _classify_one(path, repo_root, roster_boards, extras: set, s: _Sel, if _NONCODE_RE.match(path) or _META_RE.match(path): s.reasons.append(f'{path}: non-code, no contribution') return - if _METRICS_RE.match(path): + if _METRICS_RE.match(path): # rule 2b s.reasons.append(f'{path}: build-size metrics tooling, no HIL contribution') return if _FULL_RE.match(path): @@ -903,7 +963,13 @@ def main(): print(f'ci_select[build]: {r}', file=sys.stderr) for r in s['reasons']: print(f'ci_select: {r}', file=sys.stderr) - print(json.dumps(s)) + # reasons go to stderr ONLY - they are a human diagnostic and no consumer reads them + # back. They are also ~97% of the payload (a whole-tree diff: 453 KB -> 12 KB), which + # build.yml re-parses with ci_set_matrix, hil_ci_set_matrix, an inline python and + # three jq calls. The in-process dicts still carry them, for the log and the tests. + out = {k: v for k, v in s.items() if k != 'reasons'} + out['build'] = {k: v for k, v in s['build'].items() if k != 'reasons'} + print(json.dumps(out)) # ------------------------------------------------------------- @@ -925,7 +991,7 @@ def all_examples(repo_root: str) -> tuple: """Every examples// with a CMakeLists.txt, as 'role/name'.""" out = [] for role in _EX_ROLES: - for d in sorted(glob.glob(os.path.join(repo_root, 'examples', role, '*/'))): + for d in sorted(glob.glob(_rg(repo_root, 'examples', role, '*/'))): if os.path.isfile(os.path.join(d, 'CMakeLists.txt')): out.append(f'{role}/{os.path.basename(d.rstrip(os.sep))}') return tuple(out) @@ -979,13 +1045,13 @@ class _BSel: def _classify_build_one(path, repo_root, s: _BSel, get_deps_families=None): base = os.path.basename(path) - if _NONCODE_RE.match(path) or _META_RE.match(path): # rule 1 + if _NONCODE_RE.match(path) or _META_RE.match(path): # rules 1, 1b s.reasons.append(f'{path}: non-code, no build contribution') return if re.match(r'test/hil/', path): # rule 2 s.reasons.append(f'{path}: HIL harness, no build contribution') return - if path == GET_DEPS_PATH: # get_deps rule + if path == GET_DEPS_PATH: # rule 16b if get_deps_families is None: s.force_full(f'{path}: dep changes not resolvable -> full build matrix') return @@ -1002,6 +1068,7 @@ def _classify_build_one(path, repo_root, s: _BSel, get_deps_families=None): roles = _port_roles(base) exs = 'all' if roles == {'device', 'host'} else \ role_examples(repo_root, tuple(roles) + ('dual',)) + # rule 5b: fams empty -> s.add iterates nothing -> no contribution s.add(fams, exs, f'{path}: port {port} -> families {sorted(fams)}') return if re.match(r'hw/bsp/[^/]+/', path): # rule 6 @@ -1064,7 +1131,7 @@ def _classify_build_one(path, repo_root, s: _BSel, get_deps_families=None): s.add(all_bsp_families(repo_root), exs, f'{path}: typec -> {sorted(exs)}') return m = re.match(r'lib/([^/]+)/', path) - if m: # lib rule + if m: # rule 16a lib = m.group(1) exs = lib_examples(lib, repo_root) if not exs: @@ -1150,11 +1217,25 @@ def _prune_buildable(fams, fam_ex, repo_root): # for anything else spins up CI's most expensive leg to skip every example # it was given. Identical to the unfiltered list on all 81 other families. pool = set(build_py.get_examples(fam)) + + # asked per example instead of materialising the family's whole buildable + # list: skip_example is by far the hottest call in the selector, and every + # question below short-circuits (one cdc_device.c diff: 6,883 calls -> 1,889) + def can_build(ex): + # EITHER build system: this one list gates CircleCI's make legs too, and + # the two answer differently (build_utils.skip_example) + return ex in pool and any( + not build_utils.skip_example(ex, b) or + not build_utils.skip_example(ex, b, (), 'make') for b in boards) + + want = fam_ex.get(fam) try: - buildable = [e for e in allex if e in pool and - any(not build_utils.skip_example(e, b) or - not build_utils.skip_example(e, b, (), 'make') - for b in boards)] + if want is None: + kept = None if any(can_build(e) for e in allex) else [] + else: + kept = [e for e in want if can_build(e)] + if kept and not any(can_build(e) for e in allex if e not in want): + kept = None # already everything the family can build except OSError as e: # a family mid-bring-up (boards/ but no family.cmake/family.mk yet) # reads as unbuildable to the scrape; keep it rather than tracebacking @@ -1162,13 +1243,10 @@ def _prune_buildable(fams, fam_ex, repo_root): reasons.append(f'{fam}: mcu scrape unreadable ({e}), kept unfiltered') out_fams.append(fam) continue - want = fam_ex.get(fam) - have = set(buildable) - kept = buildable if want is None else [e for e in want if e in have] - if not kept: + if kept == []: continue # this diff builds nothing for this family out_fams.append(fam) - if set(kept) != set(buildable): + if kept is not None: out_ex[fam] = kept return out_fams, out_ex, reasons -- cgit v1.3.1 From b610ff039bafa1040c19d6a11cb04adcb22936e5 Mon Sep 17 00:00:00 2001 From: Ha Thach Date: Tue, 25 Aug 2026 10:35:45 +0700 Subject: ci_select: fix the membrowse test's env dependence, and stop HIL unit tests taking the rig (#3846) test_the_upload_board_can_diverge_from_the_built_board called get_family_boards without ci=True, so it pinned the developer's set, not the runner's: the CI skip lists move the one-first pick on three families. It held locally and went red on its first CI run. Pass ci=True, as _prune_buildable already does, and pin the runner's twelve. Rule 2 is a bare test/hil/ prefix, so the harness's own unit tests booked the full 27-board rig for diffs that cannot reach it. Carve test/hil/test/** out to rule 1b, beside test/{fuzz,unit-test}/**; the harness itself is untouched. A test pins that directory's file list, so anything added there that the rig does read fails rather than silently skipping hardware. Rule table updated in the spec and its carbon in the docstring. --- .../2026-08-19-ci-build-family-filter-design.md | 10 ++++- test/hil/test/test_ci_metrics.py | 18 ++++++--- test/hil/test/test_ci_select.py | 47 ++++++++++++++++++++++ tools/ci_select.py | 10 +++-- 4 files changed, 75 insertions(+), 10 deletions(-) (limited to 'docs/superpowers/specs') diff --git a/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md b/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md index 524568aeb..799c83c23 100644 --- a/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md +++ b/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md @@ -49,8 +49,8 @@ never inflates one axis with another's breadth. | # | Changed path | Build families | Build examples | HIL boards → tests | | --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | 1 | `docs/`, `.claude/`, `*.md`, `*.rst`, `LICENSE` | — | — | — | -| 1b | `.gitignore`, `.clang-format`, `.idea/**`, `test/{fuzz,unit-test}/**`, non-build `.github/**`, packaging manifests | — | — | — | -| 2 | `test/hil/**` | — | — | all boards → all tests | +| 1b | `.gitignore`, `.clang-format`, `.idea/**`, `test/{fuzz,unit-test}/**`, `test/hil/test/**`, non-build `.github/**`, packaging manifests | — | — | — | +| 2 | `test/hil/**` (not `test/hil/test/**`) | — | — | all boards → all tests | | 2b | `tools/metrics.py`, `.github/scripts/metrics_*.py` | `ALL` (unchanged — `tinyusb_metrics` runs `metrics.py` as a build target) | `ALL` | — (nothing on the rig runs it) | | 3 | `src/portable//dcd_*`, `*_device.[ch]` | `FAM` | `DEV`+`DUAL` | `FAM`'s device-role boards → device+dual tests | | 4 | `src/portable//hcd_*`, `*_host.[ch]` | `FAM` | `HOST`+`DUAL` | `FAM`'s host-role boards → host+dual tests | @@ -75,6 +75,12 @@ never inflates one axis with another's breadth. **Rule 2 is deliberately asymmetric.** A `test/hil/**` change is invisible to the family matrix but is exactly what the rig exercises, so it builds nothing and runs everything. +`test/hil/test/**` is carved out to rule 1b: it holds the harness's own unit tests, which +nothing on the rig runs (pre-commit does, and `build.yml` runs `test_ci_select.py` as the +gate before trusting a selection). A bare `test/hil/` prefix was booking the full 27-board +rig for diffs that cannot reach it. The carve-out is a claim about that directory's +contents, so a test pins its file list: add anything the rig reads and it fails. + **Rule 7 is the one HIL-side behaviour change in this design.** Today `hw/mcu/` sits in `hil_select`'s `_FULL_RE` and forces the full HIL matrix. Since the build axis now resolves those paths to a family through the same scan, forcing full on the rig is inconsistent. The diff --git a/test/hil/test/test_ci_metrics.py b/test/hil/test/test_ci_metrics.py index aac251824..6f1511913 100644 --- a/test/hil/test/test_ci_metrics.py +++ b/test/hil/test/test_ci_metrics.py @@ -532,7 +532,12 @@ class TestWorkflowSelectionHandOff(unittest.TestCase): --one-first with no -e returns preferred_list[0]; with one it returns the first preferred board that can build it. Where those differ, the Membrowse Upload step - configures a build dir the Build step never wrote.""" + configures a build dir the Build step never wrote. + + ci=True unconditionally, as _prune_buildable does and for the same reason: the + answer must be the runner's, not the developer's. The CI skip lists are off by + default locally, which moves the pick on three families - this test asserted the + local set and went red on its first CI run.""" sys.path.insert(0, os.path.join(REPO, 'tools')) import build as build_py roles = ('device', 'host', 'dual') @@ -547,14 +552,16 @@ class TestWorkflowSelectionHandOff(unittest.TestCase): diverging = set() for fam in fams: try: - base = build_py.get_family_boards(fam, False, True, None, 'cmake', ()) + base = build_py.get_family_boards(fam, False, True, None, 'cmake', + (), ci=True) except Exception: continue if not base: continue for e in exs: try: - one = build_py.get_family_boards(fam, False, True, [e], 'cmake', ()) + one = build_py.get_family_boards(fam, False, True, [e], 'cmake', + (), ci=True) except Exception: continue if one and one[0] != base[0]: @@ -562,8 +569,9 @@ class TestWorkflowSelectionHandOff(unittest.TestCase): break finally: os.chdir(cwd) - self.assertEqual(diverging, {'imxrt', 'lpc11', 'lpc18', 'lpc54', 'mcx', 'rp2040', - 'rx', 'samd11', 'stm32l0', 'stm32l4', 'tm4c'}, + self.assertEqual(diverging, {'imxrt', 'lpc11', 'lpc18', 'lpc54', 'mcx', 'rx', + 'samd11', 'samd2x_l2x', 'samd5x_e5x', 'stm32l0', + 'stm32l4', 'tm4c'}, 'the set of families whose membrowse upload can land on an ' 'uncompiled board changed; re-check whether dropping $EX_ARGS ' 'from the upload step is still the right trade') diff --git a/test/hil/test/test_ci_select.py b/test/hil/test/test_ci_select.py index dc10f769a..c34bccd1f 100644 --- a/test/hil/test/test_ci_select.py +++ b/test/hil/test/test_ci_select.py @@ -942,6 +942,53 @@ class TestClassesWithNoEnablingExample(unittest.TestCase): 'both axes, so nothing compiles it until the next master push') +class TestTheHarnessTestsAreNotTheHarness(unittest.TestCase): + """test/hil/test/ selects nothing; test/hil/ itself still selects everything. + + Rule 2 is a bare `test/hil/` prefix, so the harness's own unit tests were booking + the full 27-board rig - ~11 minutes of exclusive hardware for a diff that cannot + reach it. Nothing on the rig runs them: pre-commit does, and build.yml runs + test_ci_select.py as the gate before trusting a selection at all. + + The carve-out is only safe while that directory holds nothing rig-affecting, which + is what the second test pins.""" + + def test_the_harness_own_tests_select_nothing_on_either_axis(self): + for p in ('test/hil/test/test_ci_select.py', 'test/hil/test/test_ci_metrics.py', + 'test/hil/test/test_hil_bounded.py', 'test/hil/test/stubs/pymtp.py'): + s = ci_select.classify([p], REPO, ROSTERS) + self.assertFalse(s['full'], p) + self.assertFalse(s['boards'], p) + b = ci_select.classify_build([p], REPO) + self.assertFalse(b['full'], p) + self.assertFalse(b['families'], p) + + def test_the_harness_itself_still_takes_the_whole_rig(self): + # the thing rule 2 exists for: these decide what the rig does, so they cannot be + # trusted to narrow their own blast radius + for p in ('test/hil/hil_test.py', 'test/hil/tinyusb.json', + 'test/hil/helper/hil_ci_set_matrix.py'): + s = ci_select.classify([p], REPO, ROSTERS) + self.assertTrue(s['full'], f'{p} must still force the full rig') + + def test_nothing_rig_affecting_has_moved_into_the_carve_out(self): + """The carve-out is a claim about that directory's contents; pin them. + + A new file there that the rig DOES read would silently stop selecting the rig. + Listing them costs one line per file and makes that a failing test instead.""" + out = subprocess.run(['git', 'ls-files', 'test/hil/test'], cwd=REPO, + capture_output=True, text=True, check=True) + self.assertEqual(sorted(out.stdout.split()), [ + 'test/hil/test/stubs/pymtp.py', + 'test/hil/test/test_ci_metrics.py', + 'test/hil/test/test_ci_select.py', + 'test/hil/test/test_hil_bounded.py', + 'test/hil/test/test_hil_health.py', + 'test/hil/test/test_hil_util.py', + ], 'test/hil/test/ gained or lost a file; it is carved out of rule 2, so confirm ' + 'the rig still does not read anything in there before updating this list') + + class TestExampleMapOmitsFullFamilies(unittest.TestCase): """A family whose selection is ALREADY everything it can build carries no -e list. diff --git a/tools/ci_select.py b/tools/ci_select.py index 53fbcd3a1..ca9d54c27 100755 --- a/tools/ci_select.py +++ b/tools/ci_select.py @@ -23,8 +23,8 @@ _prune_buildable then intersects each family with what it can actually build. | # | Changed path | Build families | Build examples | HIL boards → tests | | 1 | `docs/`, `.claude/`, `*.md`, `*.rst`, `LICENSE` | — | — | — | -| 1b | `.gitignore`, `.clang-format`, `.idea/**`, `test/{fuzz,unit-test}/**`, non-build `.github/**`, packaging manifests | — | — | — | -| 2 | `test/hil/**` | — | — | all boards → all tests | +| 1b | `.gitignore`, `.clang-format`, `.idea/**`, `test/{fuzz,unit-test}/**`, `test/hil/test/**`, non-build `.github/**`, packaging manifests | — | — | — | +| 2 | `test/hil/**` (not `test/hil/test/**`) | — | — | all boards → all tests | | 2b | `tools/metrics.py`, `.github/scripts/metrics_*.py` | `ALL` (unchanged — `tinyusb_metrics` runs `metrics.py` as a build target) | `ALL` | — (nothing on the rig runs it) | | 3 | `src/portable//dcd_*`, `*_device.[ch]` | `FAM` | `DEV`+`DUAL` | `FAM`'s device-role boards → device+dual tests | | 4 | `src/portable//hcd_*`, `*_host.[ch]` | `FAM` | `HOST`+`DUAL` | `FAM`'s host-role boards → host+dual tests | @@ -108,7 +108,11 @@ _META_RE = re.compile( r'version\.yml$|SConscript$|' r'.*CMakePresets\.json$|hw/bsp/BoardPresets\.json$|examples/west\.yml$|' r'.*/[0-9]+-tinyusb[^/]*\.rules$|tools/usb_drivers/|tools/codespell/|' - r'test/(fuzz|unit-test)/|' + # test/hil/test/ holds the harness's own unit tests, not the harness: nothing on + # the rig runs them (pre-commit does, and build.yml runs test_ci_select.py as the + # gate before trusting a selection), so they cannot change what the rig does. + # The harness itself stays under _FULL_RE's test/hil/ prefix. + r'test/(fuzz|unit-test)/|test/hil/test/|' # .github, minus the build machinery named in _FULL_RE r'\.github/(FUNDING\.yml$|labeler\.yml$|membrowse_pr_message\.j2$|ISSUE_TEMPLATE/|' r'workflows/(cifuzz|claude|claude-code-review|labeler|membrowse-comment|' -- cgit v1.3.1 From 375e20090a0a60fd2d47ba6553d189e05482dfea Mon Sep 17 00:00:00 2001 From: Ha Thach Date: Tue, 25 Aug 2026 15:04:42 +0700 Subject: hil: make hil_report.md a rendering of hil_report.json (#3840) hil_report.json and hil_report.md were written independently. Four writers produced the markdown and three wrote no JSON at all -- and those three are the paths where a run died, so a JSON consumer saw nothing exactly when it mattered: the per-board verdicts an agent hands back reported the whole fleet as "no report row" while a human read the real story from the markdown. Every writer now goes through render_report(), so a table can never contain something the JSON does not. The document gains `scope` (a three-board PR run and a full run that lost 24 boards were indistinguishable) and `caveat` (how the run ended). `banner` carries rig health across an --accumulate retry; `caveat` records how a run ended and must not -- conflating them made a clean retry publish an abandonment that never happened. helper/hil_report.py owns the document end to end, dissolving the import cycle that forced write_timeout_report to compose its own markdown and removing a duplicate cell classifier kept in sync by hand. hil_summary.py is deleted; its CLI moves there. hil_ci.sh uploads the sidecar so a remote --accumulate has a merge bas --- .claude/agents/hil-operator.md | 6 +- .claude/workflows/hil-validate.js | 37 +- .claude/workflows/test-hil-validate.mjs | 29 +- .../followup/pr3803-hil-blindness-reporting.md | 3 +- .../followup/pr3836-report-single-source.md | 470 -------- .../followup/pr3840-mret-board-result.md | 90 ++ .../followup/pr3840-skill-md-no-boards-drift.md | 38 + .../followup/pr3840-write-report-atomicity.md | 30 + .../plans/2026-08-21-hil-report-module.md | 658 +++++++++++ .../specs/2026-08-21-hil-report-module-design.md | 144 +++ test/hil/helper/hil_health.py | 39 +- test/hil/helper/hil_report.py | 544 +++++++++ test/hil/helper/hil_summary.py | 115 -- test/hil/hil_ci.sh | 113 +- test/hil/hil_test.py | 222 +--- test/hil/test/test_ci_select.py | 1 + test/hil/test/test_hil_bounded.py | 149 +-- test/hil/test/test_hil_health.py | 43 - test/hil/test/test_hil_report.py | 1162 ++++++++++++++++++++ 19 files changed, 2880 insertions(+), 1013 deletions(-) delete mode 100644 docs/superpowers/followup/pr3836-report-single-source.md create mode 100644 docs/superpowers/followup/pr3840-mret-board-result.md create mode 100644 docs/superpowers/followup/pr3840-skill-md-no-boards-drift.md create mode 100644 docs/superpowers/followup/pr3840-write-report-atomicity.md create mode 100644 docs/superpowers/plans/2026-08-21-hil-report-module.md create mode 100644 docs/superpowers/specs/2026-08-21-hil-report-module-design.md create mode 100644 test/hil/helper/hil_report.py delete mode 100644 test/hil/helper/hil_summary.py create mode 100644 test/hil/test/test_hil_report.py (limited to 'docs/superpowers/specs') diff --git a/.claude/agents/hil-operator.md b/.claude/agents/hil-operator.md index a37501211..81fdc08e9 100644 --- a/.claude/agents/hil-operator.md +++ b/.claude/agents/hil-operator.md @@ -68,12 +68,12 @@ For a board run, do NOT transcribe the report table. Run the tests, then hand ba output verbatim: ```bash -python3 test/hil/helper/hil_summary.py -b BOARD [-b BOARD...] # from the report dir +python3 test/hil/helper/hil_report.py -b BOARD [-b BOARD...] # from the report dir ``` -`{"results": , "banner": , "wedged": ["board", ...]}` +`{"results": , "banner": , "caveat": , "wedged": ["board", ...]}` -`results` and `banner` are copied, never retyped, reworded or re-ordered: report rows are named +`results`, `banner` and `caveat` are copied, never retyped, reworded or re-ordered (`caveat` is the run-level notice — abandoned, aborted, no-boards — and it can say the run failed while every row says pass): report rows are named per variant, a variant name need not start with the board name, and lock contention is a cell rather than a phrase, so re-deriving any of it by hand is how this contract broke before. `wedged` is yours — the boards your run left unresponsive, usually none — and the only field you diff --git a/.claude/workflows/hil-validate.js b/.claude/workflows/hil-validate.js index bc0bda8b9..741ba481d 100644 --- a/.claude/workflows/hil-validate.js +++ b/.claude/workflows/hil-validate.js @@ -11,10 +11,10 @@ if (!args || !Array.isArray(args.boards) || args.boards.length === 0) { throw new Error('args must be { boards: string[], force? } with the boards already built') } -// The operator returns hil_summary.py's JSON verbatim plus its own observations. It does NOT +// The operator returns hil_report.py's JSON verbatim plus its own observations. It does NOT // retype the report table: rows are named per variant, a variant need not start with the board // name, and lock contention is a cell rather than a phrase — rebuilding board identity from -// prose produced a defect in each of four review rounds. hil_summary.py does that join against +// prose produced a defect in each of four review rounds. hil_report.py does that join against // the roster, so `locked` and `ran` arrive as fields and nothing here parses a detail string. const BOARD = { type: 'object', additionalProperties: false, @@ -26,12 +26,16 @@ const BOARD = { } const HIL = { type: 'object', additionalProperties: false, - required: ['results', 'wedged'], + required: ['results', 'wedged', 'caveat'], properties: { results: { type: 'array', items: BOARD }, // the operator's own observation — not derivable from the report wedged: { type: 'array', items: { type: 'string' } }, banner: { type: 'string' }, + // the run-level caveat: abandoned / aborted / selected-no-boards. `banner` carries rig + // HEALTH across an --accumulate retry; `caveat` carries how THIS run ended, and every + // row can still say pass while it failed — so it gates `pass` in summarize() below. + caveat: { type: 'string' }, }, } @@ -51,12 +55,12 @@ const runBoards = (boards, isRetry = false) => agent( (args.force ? 'THE USER HAS EXPLICITLY AUTHORIZED FORCING: run hil_test.py with HIL_NO_BOARD_LOCK=1 in the environment (bypasses the board lock check; do NOT release or kill the existing holder). ' : 'A board whose lock is held (a dev session or concurrent CI job) fails fast inside the run without blocking the others — never force the lock. ') + - 'If hil_test.py refuses the run with "board(s) not in ", re-run it WITHOUT the unknown names but keep the FULL board list on the hil_summary call below — it emits a ran:false entry for every board you name, so the unknown ones surface as "no report row" instead of costing the whole batch. ' + + 'If hil_test.py refuses the run with "board(s) not in ", re-run it WITHOUT the unknown names but keep the FULL board list on the hil_report call below — it emits a ran:false entry for every board you name, so the unknown ones surface as "no report row" instead of costing the whole batch. ' + 'Use the config for this host (hostname first). Run hil_test.py as a BACKGROUND Bash task and wait for it (a stuck fleet runs to its pool guard, 60 min by default — beyond any foreground timeout); never cancel it early. ' + 'On non-lock failures retry ONCE from the re-run spec hil_test.py just wrote — `.failed`, which already begins with --accumulate — adding -v. A usbtest battery that produced per-case verdicts is NOT auto-retried, so its result already stands. ' + 'THEN, from the directory the run wrote its report to, produce the results with:\n' + - ` python3 test/hil/helper/hil_summary.py ${boards.map((b) => `-b ${b}`).join(' ')}\n` + - 'Return its `results` array and `banner` EXACTLY as printed — do not retype, reword, re-order or "correct" them, and never transcribe the markdown table instead. ' + + ` python3 test/hil/helper/hil_report.py ${boards.map((b) => `-b ${b}`).join(' ')}\n` + + 'Return its `results` array, `banner` and `caveat` EXACTLY as printed — do not retype, reword, re-order or "correct" them, and never transcribe the markdown table instead. ' + 'Add `wedged`: the board names whose board or fixture your run left unresponsive (usually none). That is your own observation and the one field you author; put `dmesg | tail -50` in your reply text for any board you list.', { label: boards.length === 1 ? `hil:${boards[0]}` : `hil:${boards.length} boards`, @@ -64,7 +68,7 @@ const runBoards = (boards, isRetry = false) => agent( }, ) -// A lookup, not a reconciliation: hil_summary.py emits exactly one entry per requested board, +// A lookup, not a reconciliation: hil_report.py emits exactly one entry per requested board, // so a missing entry means the operator dropped it rather than that the names disagree. const byBoard = (out) => new Map((out?.results || []) .filter((r) => r && typeof r.board === 'string') @@ -91,7 +95,9 @@ const results = args.boards.map((b) => { }) for (const r of results) log(`${r.board}: ${r.pass ? 'PASS' : r.locked ? 'LOCKED' : 'FAIL'}`) if (first?.banner) log(`report banner: ${first.banner.trim().split('\n')[0]}`) +if (first?.caveat) log(`report caveat: ${first.caveat.trim().split('\n')[0]}`) +let runCaveat = first?.caveat || '' // A concurrent CI job may have held some boards (its hil_test.py flock). // CI finishes a board in minutes — retry locked boards once, at the end. if (!args.force) { @@ -116,19 +122,26 @@ if (!args.force) { } log(`${b}: retry ${results[i].pass ? 'PASS' : 'FAIL'}`) } + // the retry's own run-level verdict, not the first attempt's: a retry that abandoned + // or aborted must sink the run even though its rows may all say pass. + if (again?.caveat) runCaveat = again.caveat } } -// pass/wedged/locked in one place so it can be exercised without running an agent -const summarize = (rs, force) => ({ - pass: rs.every((r) => r.pass), +// pass/wedged/locked in one place so it can be exercised without running an agent. +// `caveat` is a RUN-level verdict and must gate `pass`: on the abandon and no-boards +// paths every row can legitimately say pass while the run itself failed (hil_test.py +// os._exit(1) -- a red job), so per-row agreement alone published those runs as green. +const summarize = (rs, force, caveat) => ({ + pass: rs.every((r) => r.pass) && !/^\*\*HIL run (abandoned|aborted|selected no boards)/m + .test(caveat || ''), wedged: rs.filter((r) => r.wedged).map((r) => r.board), locked: force ? [] : rs.filter((r) => !r.pass && r.locked).map((r) => r.board), }) -const { pass, wedged, locked } = summarize(results, args.force) +const { pass, wedged, locked } = summarize(results, args.force, runCaveat) if (wedged.length) log(`WEDGED boards needing usb-kernel-recover: ${wedged.join(', ')}`) // Workers cannot prompt the user — surface still-locked boards for the main // session to ask: force (re-invoke with force: true), wait, or accept. if (locked.length) log(`still locked after retry: ${locked.join(', ')} — ask the user: force / keep waiting / accept`) -return { pass, results, wedged, locked } +return { pass, results, wedged, locked, caveat: runCaveat } diff --git a/.claude/workflows/test-hil-validate.mjs b/.claude/workflows/test-hil-validate.mjs index db73095f6..e8be57cfc 100644 --- a/.claude/workflows/test-hil-validate.mjs +++ b/.claude/workflows/test-hil-validate.mjs @@ -4,7 +4,7 @@ // `board locked` out of a prose detail, folding rows, keeping a wedged flag alive -- produced // a defect in each of four review rounds, including a test that asserted an invariant using // the one input shape that could not break it. That logic now lives in -// test/hil/helper/hil_summary.py, where the roster is, and arrives here as fields. What is +// test/hil/helper/hil_report.py, where the roster is, and arrives here as fields. What is // left is a lookup and a verdict, and this pins both. // // Run: node .claude/workflows/test-hil-validate.mjs @@ -23,6 +23,11 @@ const cut = (start, end) => { } const body = cut('const byBoard =', 'const first = await runBoards') + cut('const summarize =', 'const { pass, wedged, locked } =') +// more than one schema declares `required:`; pick the HIL one by its contents +const HIL_REQUIRED = (src.match(/required: \[[^\]]*\]/g) || []) + .map((m) => m.replace('required: ', '').replace(/'/g, '"')) + .map((m) => JSON.parse(m)) + .find((a) => a.includes('wedged')) || [] const { byBoard, summarize, wedgedFor } = new Function(`${body}; return { byBoard, summarize, wedgedFor }`)() let failed = 0 @@ -65,5 +70,27 @@ check('wedged surfaces', summarize([R('a', false, false, true)], false).wedged, check('a wedged board that passed still surfaces', summarize([R('a', true, false, true)], false).wedged, ['a']) +// A run-level caveat outranks per-row agreement: on the abandon and no-boards paths every +// row can legitimately pass while hil_test.py exits non-zero. Row agreement alone published +// those runs green. +check('all rows pass and no caveat is a pass', + summarize([R('a', true), R('b', true)], false, '').pass, true) +check('an abandoned run is not a pass', + summarize([R('a', true)], false, + '**HIL run abandoned: the worker pool would not shut down.** x').pass, false) +check('an aborted run is not a pass', + summarize([R('a', true)], false, '**HIL run aborted: a worker raised RuntimeError**').pass, + false) +check('a no-boards run is not a pass', + summarize([R('a', true)], false, '**HIL run selected no boards.** filters emptied').pass, + false) +check('a rig-health note is NOT a caveat and does not fail the run', + summarize([R('a', true)], false, '> **Rig note.** 2 process(es) in D state').pass, true) +check('a retry that abandoned sinks the run even with all rows passing', + summarize([R('a', true)], false, + '**HIL run abandoned: the worker pool would not shut down.** retry').pass, false) +check('an omitted caveat cannot silently disable the gate (schema requires it)', + HIL_REQUIRED.includes('caveat'), true) + console.log(failed ? `\n${failed} FAILED` : '\nall checks passed') process.exit(failed ? 1 : 0) diff --git a/docs/superpowers/followup/pr3803-hil-blindness-reporting.md b/docs/superpowers/followup/pr3803-hil-blindness-reporting.md index 69ff939b0..374ee62c7 100644 --- a/docs/superpowers/followup/pr3803-hil-blindness-reporting.md +++ b/docs/superpowers/followup/pr3803-hil-blindness-reporting.md @@ -158,7 +158,8 @@ def _board_result_on_error(name, exc): """A row for a board that died by exception. err_count 1, no per-test detail, but the blindness and stray fields are still accurate -- they explain the failure more often than the exception text does.""" - rows = [(name, {BOUNDARY_CELL: f'{REPORT_CELL["fail"]} {type(exc).__name__}'}, None)] + rows = [(name, {hil_report.BOUNDARY_CELL: + f'{hil_report.REPORT_CELL["fail"]} {type(exc).__name__}'}, None)] return _board_result(name, 1, [], rows, 0.0, True) ``` diff --git a/docs/superpowers/followup/pr3836-report-single-source.md b/docs/superpowers/followup/pr3836-report-single-source.md deleted file mode 100644 index f4ccc77c2..000000000 --- a/docs/superpowers/followup/pr3836-report-single-source.md +++ /dev/null @@ -1,470 +0,0 @@ -# One Source of Truth for the HIL Report Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make `hil_report.md` a rendering of `hil_report.json` rather than a second, independently written artifact, so no run can produce a table whose contents are not in the JSON. - -**Architecture:** `hil_report.json` gains the two fields the markdown carries but the JSON does not (`scope`, and a `caveat` for text prepended after the fact). A single `render_report(doc) -> str` turns that document into the markdown, and every writer — the normal path, the pool-guard fallback, the no-boards exit, and `_abandon_exit` — goes through `write_report(report_dir, doc)`, which writes both files from the same dict. `_abandon_exit` stops doing a text-prepend on a file it did not write and instead sets `doc['caveat']`. - -**Tech Stack:** Python 3.13 stdlib only (`json`, `pathlib`); existing unit suites under `test/hil/test/` run with plain `unittest`. - -**Spec:** none — this is a follow-up split out of the `claude/hil-doc-audit` branch. The evidence it argues from is inline below. - -**Origin:** split out of PR #3836 (the HIL one-run rework + `.claude` instruction audit). Delete this file when its own PR lands. - -## Global Constraints - -- **No behaviour change to the containment paths' ordering or exit codes.** `_abandon_exit` runs while the interpreter is being torn down; its own comments record that anything raising between the pool's `finally` and `os._exit` hangs the process in multiprocessing's unbounded `join()` (reproduced at rc=124/25s with SIGTERM-ignoring workers). Serialisation added there must stay inside the existing `try`/`except` and must never raise past it. -- **The markdown stays the human artifact.** `.github/workflows/build.yml:487` uploads `hil_report.md`, `test/hil/hil_ci.sh:293` copies only it back, and `.claude/skills/hil/SKILL.md` tells the operator to paste that table verbatim. It becomes generated output, not a dropped file. -- **Banner outranks the scope note outranks the table.** Preserve the existing order (`hil_test.py:2153-2161`): the caveat is outermost because that is where `hil/SKILL.md` tells an agent to look. -- **`--accumulate` merges from the JSON** (`hil_test.py:2102-2115`), including carrying the prior banner forward. Adding fields must not break that merge for a sidecar written by an older version. -- Run `python3 -m unittest discover -s test/hil/test` (115 tests, ~78 s) before each commit; `pre-commit run --files ` before pushing. - -## Why this is worth doing - -Four writers produce `hil_report.md`, and three of them write no JSON at all: - -| Writer | JSON? | Line | -|---|---|---| -| `accumulate_report` — the normal path | yes | `hil_test.py:2149`, `:2162` | -| `**HIL run selected no boards.**` | **no** | `hil_test.py:2317` | -| pool-guard fallback → `hil_health.write_timeout_report(...)` | **no** | `hil_test.py:2469`, `hil_health.py:346` | -| `_abandon_exit` — prepends to whatever `.md` exists | **no** | `hil_test.py:2603` | - -Those three are exactly the paths where the run died, so they are the cases where the artifact matters most and where a JSON consumer sees nothing. `test/hil/helper/hil_summary.py` (added on the origin branch) reads the JSON to build the per-board verdicts an agent hands back — on any of those three paths it finds no file and reports "no report row for this board" for the whole fleet, while a human reading the markdown sees the real story. - -Separately, `scope` exists only in the markdown (`hil_test.py:2154`, from `accumulate_report`'s `scope: str = ''` parameter at `:2092`). A PR-scoped three-board table and a full-fleet run that lost 24 boards are indistinguishable in the JSON. - ---- - -### Task 1: Put `scope` in the JSON - -**Files:** -- Modify: `test/hil/hil_test.py:2092-2163` (`accumulate_report`) -- Test: `test/hil/test/test_hil_bounded.py` (new class beside `CaveatSurvivesAccumulate`) - -**Interfaces:** -- Produces: `hil_report.json` gains a top-level `"scope": str` (empty string when unscoped). Existing keys `rows` and `banner` are unchanged. - -- [ ] **Step 1: Write the failing test** - -```python -class ScopeSurvivesInTheJson(unittest.TestCase): - """A scoped run's small table is indistinguishable from a full run that lost boards. - The markdown says so; the JSON did not, so any JSON consumer could not tell.""" - - def _rows(self, board, cell): - return [(board, 0, 0, [(board, {cell: 'OK'}, '1s')], 0)] - - def test_scope_is_recorded_in_the_sidecar(self): - import json - td = TemporaryDirectory() - self.addCleanup(td.cleanup) - rd = Path(td.name) - hil_test.accumulate_report(self._rows('boardA', 'cdc_msc'), rd, True, - '-b boardA', '') - doc = json.loads((rd / 'hil_report.json').read_text()) - self.assertEqual(doc['scope'], '-b boardA') - - def test_an_unscoped_run_records_an_empty_scope(self): - import json - td = TemporaryDirectory() - self.addCleanup(td.cleanup) - rd = Path(td.name) - hil_test.accumulate_report(self._rows('boardA', 'cdc_msc'), rd, True, '', '') - self.assertEqual(json.loads((rd / 'hil_report.json').read_text())['scope'], '') -``` - -- [ ] **Step 2: Run it to verify it fails** - -Run: `python3 test/hil/test/test_hil_bounded.py ScopeSurvivesInTheJson` -Expected: FAIL — `KeyError: 'scope'` - -- [ ] **Step 3: Add the field** - -In `accumulate_report`, change the `jpath.write_text(...)` call at `hil_test.py:2149`: - -```python - jpath.write_text(json.dumps({'rows': [{'board': k, 'cells': c, 'duration': d} - for k, (c, d) in acc.items()], - 'banner': banner, - 'scope': scope}, indent=2) + '\n') -``` - -- [ ] **Step 4: Run the tests** - -Run: `python3 test/hil/test/test_hil_bounded.py ScopeSurvivesInTheJson` → PASS -Run: `python3 -m unittest discover -s test/hil/test` → 117 tests OK (the merge at `:2102` reads only `rows` and `banner`, so an older sidecar without `scope` still loads). - -- [ ] **Step 5: Commit** - -```bash -git add test/hil/hil_test.py test/hil/test/test_hil_bounded.py -git commit -m "hil_test: record the run's scope in hil_report.json - -The markdown says a scoped table is scoped; the JSON did not, so a consumer -could not tell a three-board PR run from a full run that lost 24 boards." -``` - ---- - -### Task 2: Render the markdown from the document - -**Files:** -- Modify: `test/hil/hil_test.py:1921` (`render_matrix`), `:2149-2163` (`accumulate_report`'s tail) -- Test: `test/hil/test/test_hil_bounded.py` - -**Interfaces:** -- Consumes: the `scope` key from Task 1. -- Produces: `render_report(doc: dict) -> str`, where `doc` is `{'rows': [{'board','cells','duration'}], 'banner': str, 'scope': str, 'caveat': str}`. `caveat` is optional and empty by default (Task 4 sets it). Order is caveat, banner, scope note, table. - -- [ ] **Step 1: Write the failing test** - -```python -class RenderReportIsPureFunctionOfTheDocument(unittest.TestCase): - def _doc(self, **kw): - d = {'rows': [{'board': 'boardA', 'cells': {'cdc_msc': 'pass'}, 'duration': '1s'}], - 'banner': '', 'scope': '', 'caveat': ''} - d.update(kw) - return d - - def test_table_comes_from_rows(self): - md = hil_test.render_report(self._doc()) - self.assertIn('boardA', md) - self.assertIn('cdc_msc', md) - - def test_scope_note_appears_above_the_table(self): - md = hil_test.render_report(self._doc(scope='-b boardA')) - self.assertLess(md.index('Scoped run'), md.index('boardA')) - - def test_banner_outranks_the_scope_note(self): - md = hil_test.render_report(self._doc(scope='-b boardA', - banner='> **Rig dirty.** x\n')) - self.assertLess(md.index('Rig dirty'), md.index('Scoped run')) - - def test_caveat_is_outermost(self): - md = hil_test.render_report(self._doc(banner='> **Rig dirty.** x\n', - caveat='**HIL run abandoned.**\n')) - self.assertLess(md.index('abandoned'), md.index('Rig dirty')) - - def test_a_document_with_no_rows_still_renders(self): - md = hil_test.render_report(self._doc(rows=[])) - self.assertIn('No tests were run.', md) -``` - -- [ ] **Step 2: Run it to verify it fails** - -Run: `python3 test/hil/test/test_hil_bounded.py RenderReportIsPureFunctionOfTheDocument` -Expected: FAIL — `AttributeError: module 'hil_test' has no attribute 'render_report'` - -- [ ] **Step 3: Add `render_report` and route `accumulate_report` through it** - -Add beside `render_matrix` (after `hil_test.py:1919`): - -```python -def render_report(doc: dict) -> str: - """The markdown IS a rendering of the sidecar. Every writer goes through here, so a - table can never contain something the JSON does not.""" - md = render_matrix([(r['board'], r['cells'], r.get('duration')) - for r in doc.get('rows', [])]) - if doc.get('scope'): - # a scoped run's small table is otherwise indistinguishable from a full one, and - # it replaces the previous full table in the sticky PR comment - md = f'_Scoped run: {doc["scope"]}. Boards/tests not listed were not run._\n\n' + md - # banner, then caveat: a rig-health caveat outranks the table AND the scope note, and an - # abandon notice outranks even that -- the top of the report is where hil/SKILL.md tells - # the agent to look - if doc.get('banner'): - md = doc['banner'] + '\n' + md - if doc.get('caveat'): - md = doc['caveat'] + '\n' + md - return md -``` - -Then replace `accumulate_report`'s tail (`hil_test.py:2153-2163`) with: - -```python - doc = {'rows': [{'board': k, 'cells': c, 'duration': d} for k, (c, d) in acc.items()], - 'banner': banner, 'scope': scope, 'caveat': ''} - jpath.write_text(json.dumps(doc, indent=2) + '\n') - md = render_report(doc) - (report_dir / REPORT_MD).write_text(md + '\n', encoding='utf-8') - return md -``` - -- [ ] **Step 4: Run the tests** - -Run: `python3 -m unittest discover -s test/hil/test` -Expected: 122 OK. `CaveatSurvivesAccumulate` must still pass — it asserts the banner survives a rerun, which is now the `banner` key round-tripping through the document. - -- [ ] **Step 5: Commit** - -```bash -git add test/hil/hil_test.py test/hil/test/test_hil_bounded.py -git commit -m "hil_test: render the markdown from the report document - -One function turns the sidecar into the table, so the markdown cannot carry -anything the JSON lacks. Ordering (caveat > banner > scope > table) is pinned -by tests rather than by the order of three string concatenations." -``` - ---- - -### Task 3: Give the two early-exit paths a document - -**Files:** -- Modify: `test/hil/hil_test.py:2313-2320` (no-boards exit), `test/hil/helper/hil_health.py:346` (`write_timeout_report`) -- Test: `test/hil/test/test_hil_health.py` (beside `WriteTimeoutReport`), `test/hil/test/test_hil_bounded.py` - -**Interfaces:** -- Consumes: `render_report(doc)` from Task 2. -- Produces: `write_report(report_dir: Path, doc: dict) -> None`, which writes `hil_report.json` and `hil_report.md` from one dict. Both early-exit paths call it. - -- [ ] **Step 1: Write the failing test** - -```python -class EveryExitPathLeavesBothArtifacts(unittest.TestCase): - """hil_summary.py builds an agent's verdicts from the JSON. A path that writes only - markdown reports the whole fleet as 'no report row' while a human sees the real story.""" - - def test_the_no_boards_exit_writes_json_too(self): - import json - td = TemporaryDirectory() - self.addCleanup(td.cleanup) - rd = Path(td.name) - hil_test.write_report(rd, {'rows': [], 'banner': '', 'scope': '', - 'caveat': '**HIL run selected no boards.** why\n'}) - self.assertIn('selected no boards', (rd / 'hil_report.md').read_text()) - doc = json.loads((rd / 'hil_report.json').read_text()) - self.assertEqual(doc['rows'], []) - self.assertIn('selected no boards', doc['caveat']) -``` - -and, in `test_hil_health.py`: - -```python - def test_timeout_report_writes_the_sidecar(self): - import json - td = TemporaryDirectory() - self.addCleanup(td.cleanup) - rd = Path(td.name) - hil_health.write_timeout_report(rd, [{'name': 'boardA'}], 3600, 'hil_report.md') - self.assertTrue((rd / 'hil_report.json').is_file()) - self.assertIn('boardA', (rd / 'hil_report.json').read_text()) -``` - -- [ ] **Step 2: Run them to verify they fail** - -Run: `python3 test/hil/test/test_hil_bounded.py EveryExitPathLeavesBothArtifacts` -Expected: FAIL — `AttributeError: module 'hil_test' has no attribute 'write_report'` -Run: `python3 test/hil/test/test_hil_health.py WriteTimeoutReport` -Expected: FAIL — `hil_report.json` is not a file - -- [ ] **Step 3: Add `write_report` and use it in both paths** - -Beside `render_report`: - -```python -def write_report(report_dir: Path, doc: dict) -> None: - """Write both artifacts from one document. Best-effort by design: every caller is on a - failure path where an OSError must not replace the failure being reported.""" - try: - report_dir.mkdir(parents=True, exist_ok=True) - (report_dir / REPORT_JSON).write_text(json.dumps(doc, indent=2) + '\n') - (report_dir / REPORT_MD).write_text(render_report(doc) + '\n', encoding='utf-8') - except OSError: - pass -``` - -Replace the no-boards block at `hil_test.py:2315-2320` with: - -```python - rd = Path(os.environ.get('HIL_REPORT_DIR', '.')) - write_report(rd, {'rows': [], 'banner': '', 'scope': '', - 'caveat': f'**HIL run selected no boards.** {msg}\n'}) -``` - -In `hil_health.write_timeout_report`, after the markdown is composed, write the sidecar next to it with a row per stuck board: - -```python - json_path = report_dir / 'hil_report.json' - json_path.write_text(json.dumps( - {'rows': [{'board': b['name'], 'cells': {'pool-timeout': 'fail'}, - 'duration': None} for b in boards], - 'banner': banner, 'scope': '', 'caveat': prefix}, indent=2) + '\n') -``` - -Keep it inside the function's existing broad `try` — a roster entry without `name` must not escape, which is what that handler exists to prevent. - -- [ ] **Step 4: Run the tests** - -Run: `python3 -m unittest discover -s test/hil/test` -Expected: 124 OK. - -- [ ] **Step 5: Commit** - -```bash -git add test/hil/hil_test.py test/hil/helper/hil_health.py test/hil/test/ -git commit -m "hil_test, hil_health: write the sidecar on the early-exit paths too - -The no-boards exit and the pool-guard fallback wrote markdown only, so a JSON -consumer saw nothing on exactly the runs that failed. hil_summary.py reported -the whole fleet as 'no report row' while the markdown told the real story." -``` - ---- - -### Task 4: Make `_abandon_exit` set a field instead of prepending text - -**Files:** -- Modify: `test/hil/hil_test.py` (`_abandon_exit`, the `if report is not None:` block near `:2622`), and its call site at `:2603` -- Test: `test/hil/test/test_hil_bounded.py` - -**Interfaces:** -- Consumes: `write_report`/`render_report` from Tasks 2–3. -- Produces: `_abandon_exit(pool, mgr, abandoned, err_count, report_dir: Path | None = None)` — the parameter becomes the **directory**, not the markdown path. - -- [ ] **Step 1: Write the failing test** - -```python -class AbandonNoticeLandsInBothArtifacts(unittest.TestCase): - def test_abandon_sets_the_caveat_not_just_the_markdown(self): - import json - td = TemporaryDirectory() - self.addCleanup(td.cleanup) - rd = Path(td.name) - hil_test.accumulate_report( - [('boardA', 0, 0, [('boardA', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') - hil_test.mark_report_abandoned(rd, 'the worker pool would not shut down.') - doc = json.loads((rd / 'hil_report.json').read_text()) - self.assertIn('abandoned', doc['caveat']) - self.assertEqual(len(doc['rows']), 1, 'the finished board must survive') - md = (rd / 'hil_report.md').read_text() - self.assertLess(md.index('abandoned'), md.index('boardA')) - - def test_marking_a_missing_report_is_a_no_op(self): - td = TemporaryDirectory() - self.addCleanup(td.cleanup) - hil_test.mark_report_abandoned(Path(td.name), 'x') # must not raise -``` - -- [ ] **Step 2: Run it to verify it fails** - -Run: `python3 test/hil/test/test_hil_bounded.py AbandonNoticeLandsInBothArtifacts` -Expected: FAIL — `AttributeError: module 'hil_test' has no attribute 'mark_report_abandoned'` - -- [ ] **Step 3: Implement it** - -```python -def mark_report_abandoned(report_dir: Path, why: str) -> None: - """Stamp an existing report as abandoned, in BOTH artifacts. - - Best-effort and silent: this runs while the interpreter is being torn down, and an - exception here hangs the process in multiprocessing's unbounded join().""" - try: - jpath = report_dir / REPORT_JSON - doc = json.loads(jpath.read_text()) if jpath.is_file() else None - if doc is None: - return - doc['caveat'] = (f'**HIL run abandoned: {why}** The table below is this run\'s ' - f'partial result.\n') - write_report(report_dir, doc) - except (OSError, ValueError, TypeError): - pass -``` - -Then in `_abandon_exit`, replace the read-modify-write of the markdown with `mark_report_abandoned(report, ...)` and change the call site at `:2603` from `report_dir / REPORT_MD` to `report_dir`. - -- [ ] **Step 4: Run the tests** - -Run: `python3 -m unittest discover -s test/hil/test` -Expected: 126 OK. - -- [ ] **Step 5: Commit** - -```bash -git add test/hil/hil_test.py test/hil/test/test_hil_bounded.py -git commit -m "hil_test: stamp abandonment into the document, not onto the markdown - -_abandon_exit did a text prepend on a file it had not written, so the caveat -never reached the JSON and an agent reading the sidecar saw a clean partial -report under a red job. Still best-effort and still silent: it runs while the -interpreter is being torn down." -``` - ---- - -### Task 5: Prove the two artifacts cannot disagree - -**Files:** -- Test: `test/hil/test/test_hil_bounded.py` - -- [ ] **Step 1: Write the test** - -```python -class MarkdownIsAlwaysARenderingOfTheJson(unittest.TestCase): - """The property this whole change buys: whatever wrote the report, re-rendering the - sidecar reproduces the markdown byte for byte.""" - - def _check(self, rd): - import json - doc = json.loads((rd / 'hil_report.json').read_text()) - self.assertEqual((rd / 'hil_report.md').read_text(), - hil_test.render_report(doc) + '\n') - - def test_normal_path(self): - td = TemporaryDirectory(); self.addCleanup(td.cleanup); rd = Path(td.name) - hil_test.accumulate_report( - [('boardA', 0, 0, [('boardA', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, - '-b boardA', '> **Rig note.** x\n') - self._check(rd) - - def test_after_an_accumulate_rerun(self): - td = TemporaryDirectory(); self.addCleanup(td.cleanup); rd = Path(td.name) - hil_test.accumulate_report( - [('boardA', 0, 0, [('boardA', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') - hil_test.accumulate_report( - [('boardB', 0, 0, [('boardB', {'cdc_msc': 'OK'}, '1s')], 0)], rd, False, '', '') - self._check(rd) - - def test_after_abandonment(self): - td = TemporaryDirectory(); self.addCleanup(td.cleanup); rd = Path(td.name) - hil_test.accumulate_report( - [('boardA', 0, 0, [('boardA', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') - hil_test.mark_report_abandoned(rd, 'the worker pool would not shut down.') - self._check(rd) - - def test_no_boards_exit(self): - td = TemporaryDirectory(); self.addCleanup(td.cleanup); rd = Path(td.name) - hil_test.write_report(rd, {'rows': [], 'banner': '', 'scope': '', - 'caveat': '**HIL run selected no boards.** why\n'}) - self._check(rd) -``` - -- [ ] **Step 2: Run it** - -Run: `python3 test/hil/test/test_hil_bounded.py MarkdownIsAlwaysARenderingOfTheJson` -Expected: PASS on all four. A failure here means a writer still bypasses `render_report`. - -- [ ] **Step 3: Full gate and commit** - -```bash -python3 -m unittest discover -s test/hil/test # 130 OK -pre-commit run --files test/hil/hil_test.py test/hil/helper/hil_health.py \ - test/hil/test/test_hil_bounded.py test/hil/test/test_hil_health.py -git add test/hil/test/test_hil_bounded.py -git commit -m "test/hil: pin that the markdown is always a rendering of the sidecar - -Four writers, one renderer. This is the invariant the change exists to create, -so it is asserted directly rather than inferred from the writers." -``` - ---- - -## Out of scope - -Deliberately not included, each its own follow-up: - -- **The flat `HIL_POOL_TIMEOUT`.** `hil_test.py:225` is a per-process 3600 s guard that does not scale with board count. It was per board when runs were serial; the origin branch made one run cover the fleet, so a 27-board run shares one budget. Real, and a scheduling change rather than a reporting one. -- **`hil_ci.sh` accumulate in remote mode.** `hil_ci.sh:183` `rm -rf`s `REMOTE_DIR` every run and the copies are one-way, so a remote `--accumulate` retry has no merge base and its one-row report overwrites the local full-fleet one. Fixing that means uploading `hil_report.json` and `.failed` before the run, or keeping `REMOTE_DIR` when `--accumulate` is present. -- **Dropping `hil_report.md` entirely.** Not proposed. It is the PR artifact and what the `hil` skill tells operators to paste; this plan makes it generated, not redundant. diff --git a/docs/superpowers/followup/pr3840-mret-board-result.md b/docs/superpowers/followup/pr3840-mret-board-result.md new file mode 100644 index 000000000..7b8da7b9c --- /dev/null +++ b/docs/superpowers/followup/pr3840-mret-board-result.md @@ -0,0 +1,90 @@ +# Give the HIL worker result a name + +**Origin:** split out of PR #3840 (making `hil_report.md` a rendering of `hil_report.json`). +Delete this file when its own PR lands. + +## What is established + +`test_board()` returns a bare tuple that three producers build and fourteen call sites read +positionally. It has grown 5 → 6 → 7 fields, and the code already works around its own +shape: + +```python +hil_test.py:1992 dirty = [(r[0], r[6]) for r in mret if len(r) > 6 and r[6]] +hil_test.py:2014 blind = [r[0] for r in mret if len(r) > 5 and r[5]] +hil_test.py:2386 for name, _, _, _, dur, *_ in mret: +hil_report.py:306 for name, _, _, rows, *_ in mret: +``` + +Two facts make this worth closing rather than tolerating: + +- **The declared type is already wrong.** `hil_test.py:1711` says + `tuple[str, int, list[str], list, float]` — five fields — while the main return at `:1872` + yields seven (`+ sysfs_blind(), stray`). +- **A wrong slot is a wrong verdict, not a crash.** Field 5 is `blind`, which decides whether + a board's red cells are reported as broken hardware or as "could not tell". Inserting a + field mid-tuple makes `r[5]` read the wrong slot and keep running. + +It has bitten once already: `test_hil_bounded.py`'s +`test_both_row_widths_survive_the_report_writers` exists because the blindness flag widened +the tuple to 6 while the pool-timeout path still synthesised 5-field rows, and *"a +fixed-width unpack in either one raises INSIDE the containment path, which is where a raise +costs every board's results."* That is why the unpacks end in `*_`. + +## What remains + +A `NamedTuple` with defaults. Verified to pickle across the pool boundary and to stay +fully tuple-compatible — existing `r[0]`, `e[1]`, `for name, _, _, rows, *_` and `len(r)` +all keep working, so it lands without touching the fourteen consumers: + +```python +class BoardResult(NamedTuple): + """What one worker returns. Field ORDER is load-bearing: it is unpacked positionally + in a dozen places, and the pool-timeout path synthesises one by hand.""" + name: str + err_count: int + failed_tests: list[str] + rows: list | None # None from the pool-timeout synthesis, never [] + duration: float + blind: bool = False # defaults, so a synthesised result is full-width + stray: int = 0 +``` + +Then a second, smaller step removes the coupling itself: `accumulate_report` takes +`[(name, rows)]` pairs instead of `mret`, and `hil_test` does the extraction because it owns +the shape. One line at each end; the subtle merge logic — stale lock clearing, +`BOUNDARY_CELL`, `duration=None` preservation — is untouched. + +## Sizing + +| | Sites | +|---|---| +| Producers to convert | 4 (`hil_test.py:1724`, `:1872`, `:2283`, `:2327`) | +| Arity guards deleted | 2 (`:1992`, `:2014`) | +| Wrong annotation fixed | 1 (`:1711`) | +| `hil_report`'s coupled line | 1 (`:306`) | +| Positional consumers (optional migration) | 14 | +| **Test fixtures building tuples by hand** | **34** | + +Production code is roughly ten changed lines. **The work is dominated by the test +fixtures**, which is also the risk. + +## Do this first, or the refactor is unverifiable + +`test_hil_report.py` (27 sites) and `test_hil_bounded.py` (7) construct plain tuples by +hand — `('boardA', 0, [], [], 1.0, True)`. A producer that forgot to switch to +`BoardResult`, or a pickling regression, **passes the entire 310-test suite** and surfaces +only on the rig. Convert the fixtures to build `BoardResult` as task 1, before touching any +producer. This ordering is not optional. + +Second trap: `rows` is `None` on the pool-timeout path (`hil_test.py:2283`), never `[]`, and +`accumulate_report` guards with `if rows and ...`. A well-meaning `rows: list = []` default +silently changes that path. Pin it with a test before the conversion. + +## Why it was split out + +PR #3840 touches the report document. This touches `test_board`'s return and the containment +paths, where a raise costs every board's results rather than one board's — a different blast +radius, needing its own review and its own rig run. #3840 is twice-reviewed and dogfooded +ten times on hardware; folding this in would reset that surface for a latent-trap cleanup +that is not causing bugs today. diff --git a/docs/superpowers/followup/pr3840-skill-md-no-boards-drift.md b/docs/superpowers/followup/pr3840-skill-md-no-boards-drift.md new file mode 100644 index 000000000..a039a8c12 --- /dev/null +++ b/docs/superpowers/followup/pr3840-skill-md-no-boards-drift.md @@ -0,0 +1,38 @@ +# `SKILL.md` contradicts the code on no-boards tables + +**Origin:** split out of PR #3840, surfaced by its second review round. Delete this file +when its own PR lands. + +`.claude/skills/hil/SKILL.md:150-151` tells the reading agent: + +> `**HIL run selected no boards.**` — the filters intersected to nothing, so there is **no +> table at all**. Report that (and the filter shown), never `"pass": true`. + +That was true when the no-boards exit wrote a bare notice. It no longer is. An +`--accumulate` no-boards run keeps the accumulated rows — deliberately, because wiping them +destroyed real results — so the artifact now reads: + +``` +**HIL run selected no boards.** filters emptied + +**✅ 1 passed · ❌ 0 failed · ⚪ 0 skipped · blank not run** + +| Board | t | duration | +... +``` + +The behaviour is correct; the documentation is wrong, and wrong in the direction that +matters. An agent is told to expect no table, sees one, and has no rule for whether those +rows are reportable. **They are not this run's** — they are a previous attempt's, carried +forward. + +**What remains:** update that bullet to describe both cases — a fresh run has no table, an +`--accumulate` run shows the previous attempt's rows under the notice and they must not be +reported as this run's. Add a test asserting the fresh case renders no matrix, so the two +halves cannot drift again. + +## Why it was split out + +PR #3840 fixed the findings that changed a verdict. This is a documentation drift: the +behaviour is correct and the doc describing it is not, so it is better reviewed on its own +than appended to a branch already carrying a module consolidation. diff --git a/docs/superpowers/followup/pr3840-write-report-atomicity.md b/docs/superpowers/followup/pr3840-write-report-atomicity.md new file mode 100644 index 000000000..2094207bb --- /dev/null +++ b/docs/superpowers/followup/pr3840-write-report-atomicity.md @@ -0,0 +1,30 @@ +# `write_report` commits the two artifacts non-atomically + +**Origin:** split out of PR #3840, surfaced by its second review round. Delete this file +when its own PR lands. + +```python +md = render_report(doc) + '\n' +report_dir.mkdir(parents=True, exist_ok=True) +(report_dir / REPORT_JSON).write_text(json.dumps(doc, indent=2) + '\n') +(report_dir / REPORT_MD).write_text(md, encoding='utf-8') +``` + +Rendering before writing closed the *render-failure* case: a raise can no longer commit a +sidecar the markdown contradicts. It does not close the *interrupted-between-writes* case. A +kill between those two lines leaves the pair disagreeing — and this runs on the containment +path, on the way to `os._exit`, on a rig whose jobs get cancelled by the GitHub job ceiling. + +**What remains:** write both to temp files, then `os.replace` both. The window shrinks from +two full writes to two renames, and neither file is ever observed half-written. `os.replace` +is atomic per file on POSIX; the pair is still not transactional, which is acceptable and +should be said in the docstring rather than implied away. + +Worth pairing with a test that kills between the writes — or, more practically, one that +asserts no partial file is ever visible by checking the temp-then-rename shape directly. + +## Why it was split out + +A durability edge, not a wrong verdict. PR #3840 closed the render-failure half of this +(nothing is written until the markdown renders); the interrupted-between-writes half needs +a temp-then-rename and is better reviewed on its own. diff --git a/docs/superpowers/plans/2026-08-21-hil-report-module.md b/docs/superpowers/plans/2026-08-21-hil-report-module.md new file mode 100644 index 000000000..5a7c832d8 --- /dev/null +++ b/docs/superpowers/plans/2026-08-21-hil-report-module.md @@ -0,0 +1,658 @@ +# hil_report.py Module Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Fold every function that produces, renders, merges or reads `hil_report.json`/`hil_report.md` into one module, `test/hil/helper/hil_report.py`, and take the two fixes that consolidation enables. + +**Architecture:** A new leaf-ish module owns the report document. `hil_test.py` and `hil_health.py` both import it, which dissolves the circular-import constraint that forced `write_timeout_report` to compose its own markdown. The duplicated cell classifier (`cell_kind` in `hil_test`, `cell_state` in `hil_summary`) collapses into one. `hil_summary.py` is deleted and its CLI moves in. + +**Tech Stack:** Python 3.13 stdlib only (`json`, `argparse`, `pathlib`); existing unit suites under `test/hil/test/` run with plain `unittest`. + +**Spec:** `docs/superpowers/specs/2026-08-21-hil-report-module-design.md` + +## Global Constraints + +- **Behaviour-preserving motion.** `hil_test.py`'s CLI, arguments, output and report format stay byte-identical. The two intended exceptions are named in the spec: the `hil_summary.py` → `hil_report.py` CLI path, and `write_timeout_report` rendering instead of concatenating. +- **`hil_report.py` must work in two modes.** It is imported as `helper.hil_report` by `hil_test.py`, and run as a script by the operator (`python3 test/hil/helper/hil_report.py -b BOARD`). A script run puts `test/hil/helper/` on `sys.path`, *not* `test/hil/`, so `from helper import hil_health` fails in that mode. Task 1 pins both modes with tests. +- **Containment paths must never raise.** `mark_report_abandoned` and `write_timeout_report` run while the interpreter is being torn down or on the way to `os._exit`; anything escaping hangs the process in multiprocessing's unbounded `join()`. Their existing broad handlers move with them unchanged. +- **`hil_ci.sh` stages helpers by an explicit list** (`test/hil/hil_ci.sh:222-228`). A helper module missing from it reaches the rig absent, and the run dies with `ImportError` *after* `REMOTE_DIR` has been wiped. `RemoteStaging.test_import_closure_is_staged_to_the_rig` in `test_hil_bounded.py` already enforces this from the AST import closure; Task 1 only has to add the file to the list. +- Run `python3 -m unittest discover -s test/hil/test` (~82 s) before each commit; `pre-commit run --files ` before pushing. + +--- + +### Task 1: The module, the vocabulary, one classifier, and the render half + +**Files:** +- Create: `test/hil/helper/hil_report.py` +- Create: `test/hil/test/test_hil_report.py` +- Modify: `test/hil/hil_test.py:110` (`REPORT_CELL`), `:1715` (`BOUNDARY_CELL`), `:1902-1903` (`REPORT_MD`/`REPORT_JSON`), `:1921-1978` (`render_matrix`), `:1981-2003` (`render_report`), `:67` (imports) +- Modify: `test/hil/hil_ci.sh:222-228` (scp list) +- Modify: `test/hil/test/test_hil_bounded.py` (move `RenderReportIsPureFunctionOfTheDocument` out) + +**Interfaces:** +- Produces: `helper.hil_report` exposing `REPORT_MD`, `REPORT_JSON`, `REPORT_CELL`, `BOUNDARY_CELL`, `LOCKED_CELL`, `cell_state(v) -> str`, `render_matrix(rows_all) -> str`, `render_report(doc) -> str`. +- `hil_test.py` re-exports nothing: call sites become `hil_report.NAME`. + +- [ ] **Step 1: Write the failing tests** + +Create `test/hil/test/test_hil_report.py`: + +```python +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +# Unit tests for the report document: the vocabulary, the one cell classifier, rendering, +# the four writers, and the fold to per-board verdicts. Split out of test_hil_bounded.py +# and test_hil_health.py when the report code moved into helper/hil_report.py. +# Run directly: +# python3 test/hil/test/test_hil_report.py +import json +import os +import subprocess +import sys +import unittest +from pathlib import Path +from tempfile import TemporaryDirectory + +TEST_DIR = os.path.dirname(os.path.abspath(__file__)) +HIL_DIR = os.path.dirname(TEST_DIR) +sys.path.insert(0, HIL_DIR) + +from helper import hil_report + + +class OneClassifierForBothArtifacts(unittest.TestCase): + """The markdown tally and the agent's verdict used to classify cells with two separate + copies of one rule -- hil_test's cell_kind against REPORT_CELL, and hil_summary's + cell_state against its own re-typed '❌'/'⚪' literals. Change the icons and the table + and the verdict silently disagree.""" + + def test_bare_states(self): + self.assertEqual(hil_report.cell_state('fail'), 'fail') + self.assertEqual(hil_report.cell_state('skip'), 'skip') + self.assertEqual(hil_report.cell_state('pass'), 'pass') + + def test_icon_prefixed_metrics_carry_their_verdict(self): + self.assertEqual(hil_report.cell_state(f'{hil_report.REPORT_CELL["fail"]} 29/30'), 'fail') + self.assertEqual(hil_report.cell_state(f'{hil_report.REPORT_CELL["skip"]} board wedged'), + 'skip') + + def test_an_unprefixed_metric_is_a_pass(self): + """Load-bearing: a passing test may return a plain metric string. Classifying + unknown shapes as fail would publish a green table as a red verdict.""" + self.assertEqual(hil_report.cell_state('480.0 MBps'), 'pass') + self.assertEqual(hil_report.cell_state('1103 KB/s'), 'pass') + + def test_a_non_string_cell_does_not_raise(self): + """render_matrix's copy guarded with isinstance; hil_summary's did not, because its + caller str()'d first. The merged one keeps the guard -- it is the safer superset.""" + self.assertEqual(hil_report.cell_state(None), 'pass') + + def test_the_icons_come_from_REPORT_CELL(self): + """No second copy of the emoji anywhere in the module.""" + src = (Path(HIL_DIR) / 'helper' / 'hil_report.py').read_text(encoding='utf-8') + for icon in ('❌', '⚪', '✅'): + self.assertEqual(src.count(f"'{icon}'"), 1, + f'{icon} is spelled as a literal more than once') + + +class ModuleWorksImportedAndAsAScript(unittest.TestCase): + """It is imported as helper.hil_report by hil_test, and run as a script by the operator + (.claude/agents/hil-operator.md). A script run puts helper/ on sys.path, NOT test/hil, + so a plain `from helper import hil_health` breaks the CLI and only the CLI.""" + + def test_importable_as_a_package_module(self): + r = subprocess.run( + [sys.executable, '-c', + f'import sys; sys.path.insert(0, {HIL_DIR!r}); ' + f'from helper import hil_report; print(hil_report.REPORT_JSON)'], + capture_output=True, text=True, timeout=60) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn('hil_report.json', r.stdout) + + def test_runnable_as_a_script(self): + r = subprocess.run( + [sys.executable, str(Path(HIL_DIR) / 'helper' / 'hil_report.py'), '--help'], + capture_output=True, text=True, timeout=60) + self.assertEqual(r.returncode, 0, r.stderr) + + +class HilCiStagesEveryHelperTheRunImports(unittest.TestCase): + """hil_ci.sh copies helper modules by an EXPLICIT list. One missing module reaches the + rig absent and the run dies with ImportError -- after REMOTE_DIR has already been + rm -rf'd, so the previous run's report and re-run spec are gone too.""" + + def test_the_scp_list_covers_what_hil_test_imports(self): + sh = (Path(HIL_DIR) / 'hil_ci.sh').read_text(encoding='utf-8') + staged = {line.split('helper/')[1].rstrip('" \\\n') + for line in sh.splitlines() if '/test/hil/helper/' in line and '.py' in line} + imported = set() + for mod in (Path(HIL_DIR) / 'hil_test.py', Path(HIL_DIR) / 'helper' / 'hil_report.py'): + src = mod.read_text(encoding='utf-8') + for raw in src.splitlines(): + line = raw.strip() # hil_report's own import is indented in a try + if line.startswith('from helper import '): + imported |= {f'{n.strip()}.py' for n in line.split('import', 1)[1].split(',')} + elif line.startswith('from helper.'): + imported.add(line.split('.')[1].split(' ')[0] + '.py') + missing = imported - staged + self.assertEqual(missing, set(), + f'hil_ci.sh does not stage {missing}; a remote run will ImportError') + + +if __name__ == '__main__': + unittest.main() +``` + +Then **move** the class `RenderReportIsPureFunctionOfTheDocument` from `test/hil/test/test_hil_bounded.py` into this file verbatim, changing only `hil_test.render_report` → `hil_report.render_report` throughout. + +- [ ] **Step 2: Run them to verify they fail** + +Run: `python3 test/hil/test/test_hil_report.py` +Expected: FAIL — `ModuleNotFoundError: No module named 'helper.hil_report'` + +- [ ] **Step 3: Create the module** + +Create `test/hil/helper/hil_report.py`: + +```python +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +"""The HIL report document: one owner for hil_report.json and hil_report.md. + +The markdown IS a rendering of the sidecar -- every writer goes through render_report(), +so a table can never contain something the JSON does not. This module owns the whole life +of that document: the cell vocabulary, the one classifier both artifacts share, rendering, +the four writers, and the fold to one machine-readable verdict per board. + +Dual-mode by design: imported as `helper.hil_report` by hil_test.py, and run as a script by +the operator (see .claude/agents/hil-operator.md). A script run puts test/hil/helper on +sys.path rather than test/hil, hence the guarded hil_health import below. +""" +import argparse +import json +import sys +from pathlib import Path + +try: # imported as part of the helper package + from helper.hil_health import _p +except ImportError: # run as a script: helper/ is sys.path[0] + from hil_health import _p + +REPORT_MD = 'hil_report.md' +REPORT_JSON = 'hil_report.json' +# The status vocabulary, shared by the code that WRITES a cell (hil_test's test runners) and +# the code that reads one back (cell_state). One dict, so the human's table and the agent's +# verdict cannot drift apart. +REPORT_CELL = {'pass': '✅', 'fail': '❌', 'skip': '⚪'} +BOUNDARY_CELL = 'same-PID boundary' +LOCKED_CELL = 'board-locked' + + +def cell_state(v) -> str: + """'pass' | 'fail' | 'skip' for one report cell. + + THE classifier -- the markdown tally and the per-board verdict both call this, so they + cannot disagree. 'fail' or a ❌ prefix is a failure, 'skip' or a ⚪ prefix is a skip, and + EVERYTHING ELSE is a pass. That last arm is load-bearing: a passing test may return a + plain metric string ('480.0 MBps') that lands in the cell unprefixed, while failures are + guaranteed marked -- TestFail's docstring pins that its metric is icon-prefixed precisely + so render and tally treat it as a failure. Classifying unknown shapes as fail here would + publish a green table as a red verdict. + + isinstance-guarded: cells are usually str but a caller may hand over None or a number, + and .startswith on those raises inside a report writer that must not raise.""" + if v == 'fail' or (isinstance(v, str) and v.startswith(REPORT_CELL['fail'])): + return 'fail' + if v == 'skip' or (isinstance(v, str) and v.startswith(REPORT_CELL['skip'])): + return 'skip' + return 'pass' +``` + +Then move, verbatim, from `hil_test.py`: +- `render_matrix` (`hil_test.py:1921-1978`) — with one change: delete its nested `cell_kind` + definition and call the module-level `cell_state` instead. The line + `kinds = [cell_kind(v) for _, cells, _ in rows_all for v in cells.values()]` becomes + `kinds = [cell_state(v) for _, cells, _ in rows_all for v in cells.values()]`. +- `render_report` (`hil_test.py:1981-2003`) — unchanged. + +Add a placeholder CLI so `--help` works (Task 4 fills in `summarize`): + +```python +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument('config_file') + ap.add_argument('-b', '--board', action='append', default=[], + help='boards to report on; default: every board in the config') + ap.add_argument('--report-dir', default='.', help=f'where {REPORT_JSON} lives (default: cwd)') + ap.parse_args() + raise SystemExit('hil_report: summarize() lands in Task 4') + + +if __name__ == '__main__': + sys.exit(main()) +``` + +- [ ] **Step 4: Point `hil_test.py` at the module** + +In `hil_test.py:67`, extend the import: + +```python +from helper import hil_health, hil_lock, hil_report, hil_util +``` + +Delete `REPORT_CELL` (`:110`), `BOUNDARY_CELL` (`:1715`), `REPORT_MD`/`REPORT_JSON` +(`:1902-1903`), `render_matrix` and `render_report` from `hil_test.py`. Then rewrite every +reference to the moved names as `hil_report.`. Find them all with: + +```bash +grep -n "REPORT_CELL\|BOUNDARY_CELL\|REPORT_MD\|REPORT_JSON\|render_matrix\|render_report" \ + test/hil/hil_test.py +``` + +Known sites: `:876`, `:1369`, `:1459`, `:1490`, `:1492`, `:1508`, `:1818`, `:1834`, `:2162`, +`:2191-2192`, `:2209-2210`, `:2403`, `:2592`. + +- [ ] **Step 5: Stage the new module for remote runs** + +In `test/hil/hil_ci.sh:222-228`, add the module to the scp list (keep alphabetical-ish order +with the rest): + +```bash +scp -q "$ROOT_DIR/test/hil/helper/__init__.py" \ + "$ROOT_DIR/test/hil/helper/hil_util.py" \ + "$ROOT_DIR/test/hil/helper/hil_health.py" \ + "$ROOT_DIR/test/hil/helper/hil_lock.py" \ + "$ROOT_DIR/test/hil/helper/hil_report.py" \ + "$ROOT_DIR/test/hil/helper/hil_summary.py" \ + "$ROOT_DIR/test/hil/helper/hil_select.py" \ + "$REMOTE:$REMOTE_DIR/test/hil/helper/" +``` + +- [ ] **Step 6: Run the tests** + +Run: `python3 test/hil/test/test_hil_report.py` → OK +Run: `python3 -m unittest discover -s test/hil/test` → 274 OK (266 + 8 new: 5 classifier, +2 dual-mode, 1 scp guard; `RenderReport…` moves rather than adds) + +- [ ] **Step 7: Commit** + +```bash +git add test/hil/helper/hil_report.py test/hil/hil_test.py test/hil/hil_ci.sh \ + test/hil/test/test_hil_report.py test/hil/test/test_hil_bounded.py +git commit -m "hil_report: new module for the report vocabulary, classifier and rendering + +The markdown tally and the agent's verdict classified cells with two separate +copies of one rule, the second documented as 'the EXACT classifier hil_test.py's +own tally uses'. One cell_state now serves both, keyed off the one REPORT_CELL." +``` + +--- + +### Task 2: Move the three writers + +**Files:** +- Modify: `test/hil/helper/hil_report.py` (add the writers) +- Modify: `test/hil/hil_test.py:2005-2036` (`write_report`, `mark_report_abandoned`), `:2149-2212` (`accumulate_report`) +- Modify: `test/hil/test/test_hil_bounded.py` (move three classes out), `test/hil/test/test_hil_report.py` + +**Interfaces:** +- Consumes: `render_report`, `REPORT_MD`, `REPORT_JSON`, `BOUNDARY_CELL` from Task 1. +- Produces: `hil_report.write_report(report_dir, doc)`, `hil_report.mark_report_abandoned(report_dir, why)`, `hil_report.accumulate_report(mret, report_dir, fresh, scope='', banner='') -> str`. + +- [ ] **Step 1: Move the tests** + +Move these classes from `test/hil/test/test_hil_bounded.py` into `test/hil/test/test_hil_report.py`, +verbatim except `hil_test.` → `hil_report.` for the three moved functions: + +- `ScopeSurvivesInTheJson` +- `EveryExitPathLeavesBothArtifacts` +- `AbandonNoticeLandsInBothArtifacts` +- `CaveatSurvivesAccumulate` +- `MarkdownIsAlwaysARenderingOfTheJson` + +`AbandonNoticeLandsInBothArtifacts.test_an_existing_abandon_caveat_is_not_overwritten` calls +`hil_health.write_timeout_report`; leave that call as-is — Task 3 moves it. + +- [ ] **Step 2: Run them to verify they fail** + +Run: `python3 test/hil/test/test_hil_report.py` +Expected: FAIL — `AttributeError: module 'helper.hil_report' has no attribute 'write_report'` + +- [ ] **Step 3: Move the functions** + +Cut `write_report` (`hil_test.py:2005-2014`), `mark_report_abandoned` (`:2016-2036`) and +`accumulate_report` (`:2149-2212`) from `hil_test.py` and paste them into `hil_report.py` +below `render_report`, unchanged. + +Add to `accumulate_report`'s docstring, after the existing text, so the wart is recorded +where a reader meets it: + +``` + `mret` is hil_test.py's worker-result shape (name, err, fts, rows, ...), so this one + function knows something about its caller that the rest of the module does not. Folding + mret into rows could live in hil_test and only the merge here, but that would rewrite + the subtle parts -- stale board-locked clearing, BOUNDARY_CELL dropping, duration=None + preservation -- for a tidier seam. Data-shape coupling, not an import cycle. +``` + +- [ ] **Step 4: Update the call sites** + +In `hil_test.py`, the three call sites become `hil_report.*`: + +```bash +grep -n "accumulate_report(\|write_report(\|mark_report_abandoned(" test/hil/hil_test.py +``` + +Known sites: `:2260` (inside `_abandon_exit`), `:2351` (no-boards exit), `:2486`, `:2525`, +`:2618`. + +- [ ] **Step 5: Run the tests** + +Run: `python3 -m unittest discover -s test/hil/test` → 274 OK (motion only, no count change) + +- [ ] **Step 6: Commit** + +```bash +git add test/hil/helper/hil_report.py test/hil/hil_test.py \ + test/hil/test/test_hil_report.py test/hil/test/test_hil_bounded.py +git commit -m "hil_report: move the report writers off hil_test + +write_report, mark_report_abandoned and accumulate_report join the renderer they +already call. Pure motion; accumulate_report's knowledge of mret's tuple shape +moves with it and is now documented rather than implicit." +``` + +--- + +### Task 3: `write_timeout_report` renders like everyone else + +**Files:** +- Modify: `test/hil/helper/hil_report.py` (receive the function) +- Modify: `test/hil/helper/hil_health.py:347-398` (remove it), `:19` (drop `import json`) +- Modify: `test/hil/hil_test.py:2498` (call site) +- Modify: `test/hil/test/test_hil_health.py` (move `WriteTimeoutReport` out), `test/hil/test/test_hil_report.py` + +**Interfaces:** +- Consumes: `render_report`, `write_report` from Tasks 1-2. +- Produces: `hil_report.write_timeout_report(report_dir, boards, secs, banner='', prefix='')`. The `md_name` parameter is **gone** — the module owns `REPORT_MD`. + +- [ ] **Step 1: Write the failing tests** + +Move `WriteTimeoutReport` from `test/hil/test/test_hil_health.py` into +`test/hil/test/test_hil_report.py`, changing `hil_health.write_timeout_report` → +`hil_report.write_timeout_report` and dropping the `md_name` argument from every call. Two +of its tests change substantively: + +```python + def test_the_prior_attempts_rows_survive(self): + """Was: the prior MARKDOWN TEXT survives below the banner. It now re-renders from + the merged sidecar, so the guarantee is stated against rows -- one table with the + stuck boards in it, rather than a banner stapled above a duplicate table.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('done', 0, 0, [('done', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600) + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertEqual([r['board'] for r in doc['rows']], ['done', 'stuck']) + md = (rd / hil_report.REPORT_MD).read_text() + self.assertIn('done', md) + self.assertIn('stuck', md) + self.assertIn('abandoned', md) + self.assertLess(md.index('abandoned'), md.index('done')) + self.assertEqual(md.count('| Board'), 1, 'the prior table was duplicated, not merged') + + def test_prefix_carries_the_preflight_diagnosis(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.write_timeout_report(rd, [{'name': 'b1'}], 4200, + prefix='> **wedged usb_hub_wq worker.**\n') + out = (rd / hil_report.REPORT_MD).read_text() + self.assertTrue(out.startswith('> **wedged usb_hub_wq worker.**')) + self.assertIn('timed out after 4200s', out) + self.assertIn('b1', out) +``` + +And in `MarkdownIsAlwaysARenderingOfTheJson`, **delete** +`test_the_pool_guard_fallback_agrees_even_if_it_does_not_render` and add the fifth case in +its place: + +```python + def test_the_pool_guard_fallback(self): + """The last writer to join the invariant: it composed its own markdown only because + hil_health could not import the renderer.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('done', 0, 0, [('done', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600, + prefix='> **wedged usb_hub_wq worker.**\n') + self._check(rd) +``` + +- [ ] **Step 2: Run them to verify they fail** + +Run: `python3 test/hil/test/test_hil_report.py` +Expected: FAIL — `AttributeError: module 'helper.hil_report' has no attribute 'write_timeout_report'` + +- [ ] **Step 3: Move it and make it render** + +Add to `hil_report.py`, and delete `hil_health.py:347-398` plus its now-unused +`import json` at `hil_health.py:19`: + +```python +def write_timeout_report(report_dir: Path, boards, secs: int, + banner: str = '', prefix: str = '') -> None: + """Leave a report behind when the worker pool has to be abandoned. + + map_async is all-or-nothing, so a timeout loses every per-board result and the report + dir would stay empty with no reason for the failure. Any prior attempt's rows are kept + and the stuck boards are merged in beside them. + + `prefix` carries the preflight rig-health verdict: the timeout aborts before + accumulate_report, so without it the report loses the one line saying WHY the pool never + finished.""" + try: + # Built INSIDE the try: a roster entry without a 'name' key raises while assembling + # the board list, and outside the try that escaped and stranded the runner -- which + # is exactly what the broad handler below exists to prevent. + caveat = (prefix + '\n' if prefix else '') + (banner or ( + f'**HIL run abandoned: worker pool timed out after {secs}s.**\n\n' + f'No per-board results could be collected for this attempt, so any rows below ' + f'are from an earlier one. Boards dispatched:\n\n' + + '\n'.join(f'- {b.get("name", "?")}' for b in boards) + '\n')) + # Rows MERGE rather than replace: an earlier attempt's finished boards are real + # results and this attempt has none of its own. Own handler, because a torn sidecar + # must not cost the stuck rows -- losing the old table is a nicety, losing the + # caveat is the failure. + jpath = report_dir / REPORT_JSON + try: + doc = json.loads(jpath.read_text()) if jpath.is_file() else {} + rows = list(doc.get('rows', [])) + except (OSError, ValueError, TypeError, AttributeError): + doc, rows = {}, [] + done = {r.get('board') for r in rows if isinstance(r, dict)} + rows += [{'board': b.get('name', '?'), 'cells': {'pool-timeout': 'fail'}, + 'duration': None} for b in boards if b.get('name', '?') not in done] + write_report(report_dir, {'rows': rows, 'banner': doc.get('banner', ''), + 'scope': doc.get('scope', ''), 'caveat': caveat}) + except Exception as e: # noqa: BLE001 + # Deliberately broad: this is the first statement of the pool-abandon path, so ANY + # escape skips kill_pool_children and os._exit and strands the runner. + _p(f'warning: cannot write {REPORT_MD} to {report_dir}: {e}', flush=True) +``` + +Update `hil_health.py`'s module docstring: its first line reads "Shutting a wedged HIL run +down: kill what the workers spawned, then report." — drop ", then report". + +- [ ] **Step 4: Update the call site** + +`hil_test.py:2498` becomes: + +```python + hil_report.write_timeout_report( + report_dir, [b for b in config_boards + if b['name'] in stuck], POOL_TIMEOUT, + prefix=health_banner) +``` + +- [ ] **Step 5: Run the tests** + +Run: `python3 -m unittest discover -s test/hil/test` → 274 OK (one deleted, one added) + +- [ ] **Step 6: Commit** + +```bash +git add test/hil/helper/hil_report.py test/hil/helper/hil_health.py test/hil/hil_test.py \ + test/hil/test/test_hil_report.py test/hil/test/test_hil_health.py +git commit -m "hil_report: the pool-guard fallback renders like every other writer + +It composed its own markdown for one reason: hil_health cannot import hil_test +back, so it could not reach render_report. With the renderer in a module both +import, that constraint is gone and all five writers are byte-identical -- +MarkdownIsAlwaysARenderingOfTheJson covers the fifth, and the weaker +'agrees even if it does not render' promise is deleted. + +hil_health goes back to doing one thing: killing wedged processes." +``` + +--- + +### Task 4: Fold `hil_summary.py` in and delete it + +**Files:** +- Modify: `test/hil/helper/hil_report.py` (real `summarize` + CLI) +- Delete: `test/hil/helper/hil_summary.py` +- Modify: `test/hil/hil_ci.sh` (drop `hil_summary.py` from the scp list) +- Modify: `.claude/agents/hil-operator.md:71`, `.claude/workflows/hil-validate.js:14,17,54,58,67`, `.claude/workflows/test-hil-validate.mjs:7` +- Modify: `test/hil/test/test_hil_bounded.py` (move `SummaryFoldsReportToBoards` out), `test/hil/test/test_hil_report.py` + +**Interfaces:** +- Consumes: `cell_state`, `LOCKED_CELL`, `REPORT_JSON` from Task 1. +- Produces: `hil_report.variants_of(cfg, board) -> list`, `hil_report.summarize(cfg, boards, report) -> dict` returning `{'results': [...], 'banner': str, 'caveat': str}`; CLI `python3 test/hil/helper/hil_report.py [-b BOARD]... [--report-dir DIR]`. + +- [ ] **Step 1: Move the tests** + +Move `SummaryFoldsReportToBoards` from `test/hil/test/test_hil_bounded.py` into +`test/hil/test/test_hil_report.py`, changing the subprocess target from +`helper/hil_summary.py` to `helper/hil_report.py` in both places (`test_hil_bounded.py:1675` +and `:1757`). Add one test pinning that the old entry point is gone: + +```python + def test_the_old_entry_point_is_gone(self): + """hil_summary.py's CLI moved here. A leftover file would keep working while + drifting from the module that now owns the fold.""" + self.assertFalse((Path(HIL_DIR) / 'helper' / 'hil_summary.py').exists()) +``` + +- [ ] **Step 2: Run them to verify they fail** + +Run: `python3 test/hil/test/test_hil_report.py` +Expected: FAIL — the subprocess exits non-zero with `hil_report: summarize() lands in Task 4` + +- [ ] **Step 3: Move `summarize` in and delete the old file** + +Copy `variants_of` (`hil_summary.py:47-52`) and `summarize` (`:54-92`) into `hil_report.py` +verbatim, with two changes: `cell_state(str(val))` becomes `cell_state(val)` (the merged +classifier is isinstance-guarded, so the `str()` is dead), and the module's own +`FAIL_ICON`/`SKIP_ICON`/`LOCKED_CELL`/`cell_state` definitions are NOT copied — Task 1's +already serve. + +Replace the Task 1 placeholder `main()` with the real one from `hil_summary.py:94-115`, +changing `Path(a.report_dir) / 'hil_report.json'` to `Path(a.report_dir) / REPORT_JSON`. + +Then: + +```bash +git rm test/hil/helper/hil_summary.py +``` + +- [ ] **Step 4: Update the consumers** + +`test/hil/hil_ci.sh` — remove the `hil_summary.py` line from the scp list added in Task 1. + +`.claude/agents/hil-operator.md:71`: + +```bash +python3 test/hil/helper/hil_report.py -b BOARD [-b BOARD...] # from the report dir +``` + +`.claude/workflows/hil-validate.js:58`: + +```javascript + ` python3 test/hil/helper/hil_report.py ${boards.map((b) => `-b ${b}`).join(' ')}\n` + +``` + +In `.claude/workflows/hil-validate.js` lines 14, 17, 54 and 67, and +`.claude/workflows/test-hil-validate.mjs` line 7, replace the prose mentions of +`hil_summary.py` with `hil_report.py`. Change nothing else in those files — the operator's +return contract (`{results, banner, wedged}`) is untouched. + +- [ ] **Step 5: Run the tests** + +Run: `python3 test/hil/test/test_hil_report.py` → OK +Run: `python3 -m unittest discover -s test/hil/test` → 275 OK +Run: `node .claude/workflows/test-hil-validate.mjs` → OK +Run: `grep -rn "hil_summary" . --include=*.py --include=*.sh --include=*.js --include=*.mjs --include=*.md | grep -v docs/superpowers` → no hits + +- [ ] **Step 6: Commit** + +```bash +git add test/hil/helper/hil_report.py test/hil/hil_ci.sh test/hil/test/ \ + .claude/agents/hil-operator.md .claude/workflows/hil-validate.js \ + .claude/workflows/test-hil-validate.mjs +git rm --cached test/hil/helper/hil_summary.py 2>/dev/null || true +git commit -m "hil_report: fold hil_summary in; one module owns the document end to end + +The fold to per-board verdicts is the read half of the artifact the rest of this +module writes, and it carried the second copy of the cell classifier. The CLI +keeps its arguments; only its path changes, which the two harness docs that +invoke it by name follow." +``` + +--- + +## Validation + +- [ ] **Full gate** + +```bash +python3 -m unittest discover -s test/hil/test # 275 OK +pre-commit run --all-files +``` + +- [ ] **Prove the motion changed no behaviour.** Re-render the real fleet report captured + before the refactor and diff it against what the branch produces now: + +```bash +python3 - <<'EOF' +import json, sys +sys.path.insert(0, 'test/hil') +from helper import hil_report +doc = json.load(open('hil_report.json')) # the pair the rig produced pre-refactor +assert open('hil_report.md').read() == hil_report.render_report(doc) + '\n', 'render drifted' +print('render is byte-identical to the pre-refactor artifact') +EOF +``` + +- [ ] **Rig re-check.** `hil_report.py` must reach the rig and the CLI must run there: + +```bash +bash test/hil/hil_ci.sh -b stm32f407disco -b nanoch32v203 +ssh hathach@ci.lan 'cd /tmp/tinyusb-hil && python3 test/hil/helper/hil_report.py \ + test/hil/tinyusb.json -b stm32f407disco -b nanoch32v203' +``` + +Expect a two-board table, `md == render_report(json)`, and a `summarize` verdict naming both +boards — `nanoch32v203` proving the variant fold still works through the moved code. + +## Out of scope + +Each its own follow-up, unchanged from the spec: + +- Splitting `accumulate_report`'s `mret` folding from its merge. +- The flat `HIL_POOL_TIMEOUT` that does not scale with board count. +- Carrying `caveat` through the operator/workflow return contract (`hil-validate.js:34`). diff --git a/docs/superpowers/specs/2026-08-21-hil-report-module-design.md b/docs/superpowers/specs/2026-08-21-hil-report-module-design.md new file mode 100644 index 000000000..41e7000b7 --- /dev/null +++ b/docs/superpowers/specs/2026-08-21-hil-report-module-design.md @@ -0,0 +1,144 @@ +# hil_report.py: one owner for the HIL report document + +**Date:** 2026-08-21 +**Branch:** `hil-report` (continues the report-unification work already on it) + +## Motivation + +`hil_report.json` and `hil_report.md` are now one document rendered two ways, but the code that +produces, renders, merges and reads that document is spread across three modules: + +| Module | Report-related content | +|---|---| +| `hil_test.py` | `REPORT_CELL`, `BOUNDARY_CELL`, `REPORT_MD`, `REPORT_JSON`, `render_matrix`, `render_report`, `write_report`, `mark_report_abandoned`, `accumulate_report` | +| `helper/hil_health.py` | `write_timeout_report` — composes its own markdown | +| `helper/hil_summary.py` | `cell_state`, `variants_of`, `summarize`, CLI | + +Two concrete defects follow from that spread. + +**One classifier, two copies.** `hil_test.py:1966` (`cell_kind`, keyed off `REPORT_CELL`) and +`hil_summary.py:34` (`cell_state`, with its own re-typed `FAIL_ICON, SKIP_ICON = '❌', '⚪'`) +implement the same rule. The latter's docstring says it is *"the EXACT classifier hil_test.py's own +tally uses"* — the duplication was noticed and documented as an obligation to keep in sync, rather +than removed. Change `REPORT_CELL` and the human's table and the agent's verdict silently disagree: +the markdown says ❌ where the JSON says `pass`. That is the same class of defect this branch +exists to eliminate, one layer up. + +**A writer that cannot render.** `hil_test.py` imports `hil_health`, so `hil_health` cannot import +`hil_test` back. That is the only reason `write_timeout_report` composes its own markdown instead of +calling `render_report`, and the only reason the pool-guard fallback is held to a weaker promise +(same boards and caveat in both artifacts, not byte-identical) while the other four writers are +exact. The constraint is structural, not essential: a leaf module both can import dissolves it. + +## Goal / non-goals + +**Goal:** `test/hil/helper/hil_report.py` becomes the single owner of the report document. + +**This is NOT purely code motion, and the distinction matters for review.** Measured against +`master`, `hil_test.py` contains only `render_matrix` and `accumulate_report`. Everything else in +the new module — `render_report`, `write_report`, `mark_report_abandoned`, `mark_report_no_boards`, +`_load`, `_write_stuck_over_prior_md`, `cell_state`, and the `scope`/`caveat` plumbing — is NEW +code, roughly 150 lines of it, and two rounds of review found most of their defects there. Read +those functions as new, not as relocated. `hil_test.py`'s CLI, arguments and table format do stay +unchanged. + +**Deliberate user-visible changes:** +1. `hil_summary.py` is deleted; its CLI moves to `hil_report.py`. The documented command becomes + `python3 test/hil/helper/hil_report.py -b BOARD [-b BOARD…]`. +2. `write_timeout_report` re-renders from the merged sidecar instead of stapling its banner above + the previous attempt's markdown text. Output improves — one table containing the stuck boards, + rather than a fresh banner above a duplicate table — but it is a change (see Testing). + +**Non-goals (explicit follow-ups, not this change):** +- Splitting `accumulate_report`'s `mret` folding from its merge (see "Deliberate wart"). +- The flat `HIL_POOL_TIMEOUT` that does not scale with board count (`hil_test.py:225`). + +## Resulting layout (`test/hil/`) + +| File | ~Lines | Role | +|---|---|---| +| `hil_test.py` | 2390 (−250) | tests + orchestration + CLI | +| `helper/hil_report.py` (new) | ~400 | the report document: vocabulary, render, write, merge, fold, CLI | +| `helper/hil_health.py` | ~345 (−53) | killing wedged processes only | +| `helper/hil_summary.py` | deleted | superseded by `hil_report.py` | + +Import graph: `hil_health` is a leaf; `hil_report` → `hil_health` (for `_p`, the +BrokenPipeError-safe print used on containment paths); `hil_test` → both. No cycles. + +## hil_report.py + +Stdlib only (`json`, `argparse`, `pathlib`) beyond that one `_p` import. Sections, in order: + +**Vocabulary.** `REPORT_MD`, `REPORT_JSON`, `REPORT_CELL`, `BOUNDARY_CELL`, `LOCKED_CELL`. +`REPORT_CELL` becomes the single source of the status icons; `hil_summary.py`'s `FAIL_ICON`/ +`SKIP_ICON` literals are deleted. + +**Classifier.** One `cell_state(v) -> 'pass' | 'fail' | 'skip'`, replacing both `cell_kind` and the +old `cell_state`. Keeps the surviving docstring's warning that the `pass` arm is load-bearing: a +passing test may return an unprefixed metric string (`'480.0 MBps'`), while failures are guaranteed +icon-marked, so classifying unknown shapes as `fail` would publish a green table as a red verdict. + +**Render.** `render_matrix(rows_all)`, `render_report(doc)`. Unchanged; `render_matrix`'s inline +`cell_kind` is replaced by a call to the module-level `cell_state`. + +**Write.** `write_report`, `accumulate_report`, `mark_report_abandoned`, `write_timeout_report`. +Moved verbatim except `write_timeout_report`, which loses its `md_name` parameter (the module owns +`REPORT_MD`) and renders instead of concatenating. + +**Fold.** `variants_of`, `summarize`, and the `main()` CLI from `hil_summary.py`. + +## Deliberate wart + +`accumulate_report` moves wholesale, keeping its knowledge of `mret`'s worker-result tuple shape. +The cleaner boundary would split "fold `mret` → rows" (`hil_test`'s domain) from "merge rows → doc" +(`hil_report`'s), but that rewrites subtle, well-tested logic — stale `board-locked` clearing, +`BOUNDARY_CELL` dropping, `duration=None` preservation — for a tidier seam. It is a data-shape +coupling, not an import cycle. Moving it verbatim keeps the motion reviewable as motion. + +## The sharp edge + +`hil_ci.sh:222-228` stages helper modules by an **explicit scp list**. A new `helper/hil_report.py` +that is not added there reaches the rig missing, and the run dies with `ImportError` *after* +`REMOTE_DIR` has already been wiped — so the previous run's report and re-run spec are gone too. + +This is already guarded: `test_hil_bounded.py`'s `RemoteStaging.test_import_closure_is_staged_to_the_rig` +walks the AST import closure from `hil_test.py`, `usbtest.py` and `mtp_test.py` and requires an exact +scp entry for each file. Adding the module to the list is all this change needs; no new guard is +warranted, and an earlier draft of this document wrongly claimed none existed. + +## Consumers to update + +| File | Change | +|---|---| +| `test/hil/hil_ci.sh:226` | `hil_summary.py` → `hil_report.py` in the scp list | +| `.claude/agents/hil-operator.md:71` | the documented command | +| `.claude/workflows/hil-validate.js:58` | the command the operator is told to run | +| `.claude/workflows/hil-validate.js:14,17,54,67`, `test-hil-validate.mjs:7` | stale `hil_summary.py` mentions in comments | + +No logic in the `.claude` files changes — the operator's return contract +(`{results, banner, wedged}`) is untouched. + +## Testing + +New `test/hil/test/test_hil_report.py`. The report-specific classes move there from +`test_hil_bounded.py` (`CaveatSurvivesAccumulate`, `SummaryFoldsReportToBoards`, +`ScopeSurvivesInTheJson`, `RenderReportIsPureFunctionOfTheDocument`, +`EveryExitPathLeavesBothArtifacts`, `AbandonNoticeLandsInBothArtifacts`, +`MarkdownIsAlwaysARenderingOfTheJson`) and from `test_hil_health.py` (`WriteTimeoutReport`). + +Three test changes are substantive rather than mechanical: + +1. `WriteTimeoutReport.test_keeps_a_previous_attempts_table` asserts the prior **markdown text** + survives. It becomes an assertion that the prior attempt's **rows** survive — the same guarantee + against the new representation. +2. `MarkdownIsAlwaysARenderingOfTheJson` gains a fifth case for the pool-guard fallback, which now + satisfies the byte-identical invariant like the other four. +3. `test_the_pool_guard_fallback_agrees_even_if_it_does_not_render` — the weaker promise — is + deleted, because the promise it encoded no longer applies. + +Gate: `python3 -m unittest discover -s test/hil/test` at 275 — the current 266, minus the one +deleted test, plus the fifth invariant case, the scp-list guard, two dual-mode import tests, +five classifier tests and one pinning that the old entry point is gone — then +`pre-commit run --all-files`. Because this lands on a +branch already validated on hardware, it closes with a rig re-check: the invariant check against a +real report pair and a scoped `--accumulate` run, not the full fleet. diff --git a/test/hil/helper/hil_health.py b/test/hil/helper/hil_health.py index 92f0accc8..aa811eeb4 100644 --- a/test/hil/helper/hil_health.py +++ b/test/hil/helper/hil_health.py @@ -1,6 +1,6 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: MIT -"""Shutting a wedged HIL run down: kill what the workers spawned, then report. +"""Shutting a wedged HIL run down: kill what the workers spawned. A device whose usbfs node is held by a D-state process cannot be freed -- SIGKILL is not delivered in uninterruptible sleep -- so the goal is never to fix the rig from here. It is @@ -341,40 +341,3 @@ def kill_pool_children(pool, *extra) -> int: # SIGKILL is asynchronous and a D-state task ignores it: only a confirmed survivor # justifies the caller's power-cycle wording return len(_kill_and_confirm(killed_pids)) if killed_pids else 0 - - -def write_timeout_report(report_dir: Path, boards, secs: int, md_name: str, - banner: str = '', prefix: str = '') -> None: - """Leave a report behind when the worker pool has to be abandoned. - - map_async is all-or-nothing, so a timeout loses every per-board result and the report - dir would stay empty with no reason for the failure. Any prior attempt's markdown is - kept below the banner.""" - # `prefix` carries the preflight rig-health verdict: the timeout aborts before - # accumulate_report, so without it the report loses the one line saying WHY the pool - # never finished. The '\n' stops Markdown lazy continuation pulling the banner into - # the blockquote. - try: - # Built INSIDE the try: a roster entry without a 'name' key raises KeyError while - # assembling the board list, and outside the try that escaped and stranded the - # runner -- which is exactly what the broad handler below exists to prevent. - head = (prefix + '\n' if prefix else '') + (banner or ( - f'**HIL run abandoned: worker pool timed out after {secs}s.**\n\n' - f'No per-board results could be collected for this attempt, so the ' - f'table below (if any) is from an earlier one. Boards dispatched:\n\n' - + '\n'.join(f'- {b.get("name", "?")}' for b in boards) + '\n')) - report_dir.mkdir(parents=True, exist_ok=True) - md_path = report_dir / md_name - # Its own handler so it cannot take the write down with it: a report torn by an - # attempt killed mid-write raises UnicodeDecodeError (a ValueError, and prior - # reports always contain status emoji), which under a shared try skipped the write - # entirely. Losing the old table is a nicety; losing the banner is the failure. - try: - prior = md_path.read_text(encoding='utf-8') if md_path.is_file() else '' - except (OSError, ValueError): - prior = '' - md_path.write_text(head + (f'\n{prior}' if prior else ''), encoding='utf-8') - except Exception as e: # noqa: BLE001 - # Deliberately broad: this is the first statement of the pool-abandon path, so ANY - # escape skips kill_pool_children and os._exit and strands the runner. - _p(f'warning: cannot write {md_name} to {report_dir}: {e}', flush=True) diff --git a/test/hil/helper/hil_report.py b/test/hil/helper/hil_report.py new file mode 100644 index 000000000..b73c030a9 --- /dev/null +++ b/test/hil/helper/hil_report.py @@ -0,0 +1,544 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +"""The HIL report document: one owner for hil_report.json and hil_report.md. + +The markdown IS a rendering of the sidecar -- every writer goes through render_report(), so +a table can never contain something the JSON does not. This module owns the whole life of +that document: the cell vocabulary, the one classifier both artifacts share, rendering, the +writers, and the fold to one machine-readable verdict per board. + +Dual-mode by design: imported as `helper.hil_report` by hil_test.py, and run as a script by +the operator (see .claude/agents/hil-operator.md). A script run puts test/hil/helper on +sys.path rather than test/hil, hence the guarded hil_health import below. +""" +import argparse +import json +import sys +from pathlib import Path + + + +def _p(*args, **kwargs) -> None: + """Print that cannot raise. Defined here rather than imported from hil_health: this + module is ALSO run as a script (hil-operator.md invokes it by path), and under + PYTHONSAFEPATH=1 -- which the suite's own MTP fixtures set -- sys.path[0] is not the + script dir, so any sibling import dies before argparse runs. Five lines beat that.""" + try: + print(*args, **kwargs) + except (OSError, ValueError): + # ValueError too: printing to a CLOSED stream raises "I/O operation on closed + # file", and escaping here skips the containment path's os._exit. + pass + +REPORT_MD = 'hil_report.md' +REPORT_JSON = 'hil_report.json' +# The status vocabulary, shared by the code that WRITES a cell (hil_test's test runners) and +# the code that reads one back (cell_state). One dict, so the human's table and the agent's +# verdict cannot drift apart. +REPORT_CELL = {'pass': '✅', 'fail': '❌', 'skip': '⚪'} +BOUNDARY_CELL = 'same-PID boundary' +LOCKED_CELL = 'board-locked' +# A pseudo-test column, not a real one: write_timeout_report marks the boards that were +# still dispatched when the pool guard fired. accumulate_report clears it on a retry. +POOL_TIMEOUT_CELL = 'pool-timeout' + + +def _load(report_dir: Path) -> tuple: + """(doc, readable) for the sidecar, coerced to the canonical shape. + + hil_ci.sh uploads a sidecar as the --accumulate merge base, so a non-conforming one is + reachable from OUTSIDE the harness -- and every writer here runs on a path where a + TypeError costs the whole report. Coerce once, at the boundary, instead of guarding + each use: `banner: null` used to kill a fully successful run with a traceback and no + artifact at all, and `cells: null` sent write_timeout_report down its fallback so a + board that ate the whole pool guard was published as a pass. + + `readable` is False only when a sidecar EXISTS but could not be parsed, or is absent -- + both mean its rows are unrecoverable, which callers use to avoid destroying a markdown + that may still hold them.""" + jpath = report_dir / REPORT_JSON + if not jpath.is_file(): + return {'rows': [], 'banner': '', 'scope': '', 'caveat': ''}, False + try: + raw = json.loads(jpath.read_text()) + if not isinstance(raw, dict): + raise ValueError('sidecar is not an object') + except (OSError, ValueError, TypeError): + return {'rows': [], 'banner': '', 'scope': '', 'caveat': ''}, False + rows = [] + # isinstance, not `or []`: a sidecar with `rows: 1` iterates an int and raises outside + # the parse handler above. + for r in (raw.get('rows') if isinstance(raw.get('rows'), list) else []): + if not isinstance(r, dict) or 'board' not in r: + continue + cells = r.get('cells') + dur = r.get('duration') + # VALUES as well as keys: render_matrix does REPORT_CELL.get(v, v), which raises + # TypeError on an unhashable value, and cell_state does v.startswith. A non-str + # cell is corrupt, and dropping it renders blank -- "not run" -- which is the + # honest reading. Coercing it to str would make it classify as a PASS. + rows.append({'board': str(r['board']), + 'cells': {str(k): v for k, v in cells.items() if isinstance(v, str)} + if isinstance(cells, dict) else {}, + 'duration': dur if isinstance(dur, str) else None}) + text = lambda k: raw[k] if isinstance(raw.get(k), str) else '' + return {'rows': rows, 'banner': text('banner'), 'scope': text('scope'), + 'caveat': text('caveat')}, True + + +def cell_state(v) -> str: + """'pass' | 'fail' | 'skip' for one report cell. + + THE classifier -- the markdown tally and the per-board verdict both call this, so they + cannot disagree. 'fail' or a fail-icon prefix is a failure, 'skip' or a skip-icon prefix + is a skip, and EVERYTHING ELSE is a pass. That last arm is load-bearing: a passing test + may return a plain metric string ('480.0 MBps') that lands in the cell unprefixed, while + failures are guaranteed marked -- TestFail's docstring pins that its metric is + icon-prefixed precisely so render and tally treat it as a failure. Classifying unknown + shapes as fail here would publish a green table as a red verdict. + + isinstance-guarded: cells are usually str but a caller may hand over None or a number, + and .startswith on those raises inside a report writer that must not raise.""" + if v == 'fail' or (isinstance(v, str) and v.startswith(REPORT_CELL['fail'])): + return 'fail' + if v == 'skip' or (isinstance(v, str) and v.startswith(REPORT_CELL['skip'])): + return 'skip' + return 'pass' + + +def render_matrix(rows_all: list) -> str: + """Render rows (list of (row_label, {example: status}, duration)) as an aligned + markdown matrix: columns = tests (bare names) centered, boards left-aligned, + per-row duration as the trailing column.""" + seen = set() + for _, cells, _ in rows_all: + seen.update(cells) + if not seen: + return 'No tests were run.' + + # metric-bearing columns pinned first, the rest alphabetical: stable regardless of the + # shuffled execution order + pinned = ['usbtest', 'cdc_msc_throughput', 'msc_file_explorer', 'msc_file_explorer_freertos'] + + def col_key(t): + name = t.rsplit('/', 1)[-1] + return (pinned.index(name) if name in pinned else len(pinned), name, t) + + columns = sorted(seen, key=col_key) + headers = [c.rsplit('/', 1)[-1] for c in columns] + ['duration'] # bare example names + + def cell(cells, col): + v = cells.get(col) + if v is None: + return '' + return REPORT_CELL.get(v, v) # status symbol, or a metric string (e.g. speed) verbatim + + rows_vals = [(lbl, [cell(cells, c) for c in columns] + [dur or '']) + for lbl, cells, dur in rows_all] + board_hdr = 'Board' + board_w = max([len(board_hdr)] + [len(lbl) for lbl, _ in rows_vals]) + col_w = [max([len(h)] + [len(vals[i]) for _, vals in rows_vals]) + for i, h in enumerate(headers)] + + def line(label, values): + padded = [label.ljust(board_w)] + [v.center(w) for v, w in zip(values, col_w)] + return '| ' + ' | '.join(padded) + ' |' + + header = line(board_hdr, headers) + sep = '| ' + '-' * board_w + ' | ' + ' | '.join(':' + '-' * (w - 2) + ':' for w in col_w) + ' |' + body = [line(lbl, vals) for lbl, vals in rows_vals] + + # tally run cells (not-run cells are absent from the dicts). A cell is a bare status or + # a metric string carrying its own icon ("❌ 29/30"), so classify by the leading icon -- + # through cell_state, the same call the per-board verdict makes. + kinds = [cell_state(v) for _, cells, _ in rows_all for v in cells.values()] + failed = kinds.count('fail') + skipped = kinds.count('skip') + passed = kinds.count('pass') + summary = (f'**{REPORT_CELL["pass"]} {passed} passed · {REPORT_CELL["fail"]} {failed} failed · ' + f'{REPORT_CELL["skip"]} {skipped} skipped · blank not run**') + + return summary + '\n\n' + '\n'.join([header, sep] + body) + + +def render_report(doc: dict) -> str: + """The markdown IS a rendering of the sidecar. Every writer goes through here, so a + table can never contain something the JSON does not.""" + # .get throughout, not subscripts: mark_report_abandoned renders a sidecar it did NOT + # write (hil_ci.sh reuses a persistent REMOTE_DIR, so it may be an older version's or + # a torn one) on the way to os._exit, and a KeyError there is not in its handler -- + # it would unwind into multiprocessing's unbounded join and hang the runner it is + # trying to free. Same reason summarize() below reads cells as `r.get('cells') or {}`. + md = render_matrix([(r.get('board', '?'), r.get('cells') or {}, r.get('duration')) + for r in doc.get('rows') or [] if isinstance(r, dict)]) + if doc.get('scope'): + # a scoped run's small table is otherwise indistinguishable from a full one, and + # it replaces the previous full table in the sticky PR comment + md = f'_Scoped run: {doc["scope"]}. Boards/tests not listed were not run._\n\n' + md + # banner, then caveat: a rig-health caveat outranks the table AND the scope note, and an + # abandon notice outranks even that -- the top of the report is where hil/SKILL.md tells + # the agent to look + if doc.get('banner'): + md = doc['banner'] + '\n' + md + if doc.get('caveat'): + md = doc['caveat'] + '\n' + md + return md + + +def write_report(report_dir: Path, doc: dict) -> None: + """Write both artifacts from one document. + + RAISES on failure, deliberately: every caller is on a path whose own handler exists to + report exactly this (write_timeout_report's _p warning, hil_test's fallback-of-the- + fallback). Swallowing OSError here made both of those dead code, so an unwritable or + root-owned report dir produced no artifact AND no message. + + Renders BEFORE writing anything: committing the JSON first and then raising in + render_report left a sidecar saying "abandoned" beside a markdown still reading as a + clean green table -- the one invariant this module exists to hold.""" + md = render_report(doc) + '\n' + report_dir.mkdir(parents=True, exist_ok=True) + (report_dir / REPORT_JSON).write_text(json.dumps(doc, indent=2) + '\n') + (report_dir / REPORT_MD).write_text(md, encoding='utf-8') + + +def _abandon_notice(why: str) -> str: + # Wording is a CONTRACT: .claude/skills/hil/SKILL.md pins this banner as the case where + # "the table below IS this run's ... Report the results AND the abandonment". Calling + # the table partial would send the reading agent to re-run boards that already passed. + return (f'**HIL run abandoned: {why}** The table below was collected before the ' + f'abandon; treat board results as unverified.\n') + + +def _already_abandoned(doc: dict) -> bool: + """Whether THIS attempt already recorded how it ended. + + `caveat` only. It used to check `banner` too, because hil_test.py folded its abandon + notices in there -- but banner is carried across an --accumulate retry by design, so a + stale notice from an earlier attempt silenced a genuinely new abandon and the run's own + failure went unrecorded. banner now carries rig HEALTH (which describes the conditions + the cells were collected under, and so must persist); caveat carries the run's OUTCOME + (which must not).""" + return '**HIL run ab' in doc.get('caveat', '') + + +def _stamp_markdown(report_dir: Path, notice: str) -> None: + """Last line of defence: prepend the notice to the markdown itself. + + pr_comment.yml cats only hil_report.md, so a path that gives up here publishes a clean + green table under an abandoned, non-zero job. Master did this unconditionally.""" + mpath = report_dir / REPORT_MD + if not mpath.is_file(): + return + # errors='replace' and catch ValueError: a torn report or a LANG=C locale raises + # UnicodeDecodeError -- NOT an OSError -- straight past os._exit. + body = mpath.read_text(encoding='utf-8', errors='replace') + if '**HIL run ab' not in body[:2000]: + mpath.write_text(notice + '\n' + body, encoding='utf-8') + + +def mark_report_abandoned(report_dir: Path, why: str) -> None: + """Stamp an existing report as abandoned, in BOTH artifacts. + + Best-effort and silent: this runs while the interpreter is being torn down, and an + exception here hangs the process in multiprocessing's unbounded join().""" + notice = _abandon_notice(why) + try: + doc, readable = _load(report_dir) + if readable: + if _already_abandoned(doc): + return # whoever got there first wins, WRITE included + doc['caveat'] = notice + write_report(report_dir, doc) + return + except (OSError, ValueError, TypeError, AttributeError): + pass # fall through -- a failure here must not cost the stamp entirely + # Unreadable sidecar, or the document write failed. Either way the markdown is what + # the PR comment reads, so stamp it directly rather than giving up. + try: + _stamp_markdown(report_dir, notice) + except (OSError, ValueError, TypeError, AttributeError): + pass + + +def mark_report_no_boards(report_dir: Path, msg: str, fresh: bool = True) -> None: + """Record that the board filters intersected to nothing. + + `fresh` mirrors hil_test's own flag, because this runs BEFORE the fresh wipe: without + it a fresh run whose filter emptied re-published the PREVIOUS run's green rows under + this run's red job -- the stale-table failure it exists to prevent. An --accumulate run + keeps them, since nothing this attempt did invalidates them.""" + try: + doc, _ = _load(report_dir) + if not fresh and _already_abandoned(doc): + # SKILL.md gives the two notices OPPOSITE rules, and an abandon outranks a + # filter that matched nothing -- do not overwrite the record of a failed run. + # Only while ACCUMULATING, though: this runs before the fresh wipe, so guarding + # a fresh run would leave the previous attempt's rows AND its abandon notice + # published as this run's. + return + # A fresh run carries NOTHING from the prior sidecar -- rows, banner and scope + # alike, matching accumulate_report, which builds from an empty prior when fresh. + # Resetting only rows republished a stale rig-health note and a stale scope line + # under this run's notice, from a leftover or uploaded sidecar. + prior = {'rows': [], 'banner': '', 'scope': ''} if fresh else doc + write_report(report_dir, {'rows': prior['rows'], 'banner': prior['banner'], + 'scope': prior['scope'], + 'caveat': f'**HIL run selected no boards.** {msg}\n'}) + except (OSError, ValueError, TypeError, AttributeError): + pass # loud on stdout already; the exit code is what the job reads + + +def accumulate_report(mret: list, report_dir: Path, fresh: bool, scope: str = '', + banner: str = '', caveat: str = '') -> str: + """Merge this run's results into json in report_dir, then (re)write + the markdown matrix to md. `fresh` (a first run, no --accumulate) + starts a new report; otherwise a re-run accumulates so boards/tests that + already passed are preserved while re-run cells are updated. `scope` names the + board filter, if any, so a scoped table is not mistaken for a full one. + Returns the md. + + `mret` is hil_test.py's worker-result shape (name, err, fts, rows, ...), so this one + function knows something about its caller that the rest of the module does not. Folding + mret into rows could live in hil_test and only the merge here, but that would rewrite + the subtle parts -- stale board-locked clearing, BOUNDARY_CELL dropping, duration=None + preservation -- for a tidier seam. Data-shape coupling, not an import cycle.""" + # ONE canonical load: a sidecar reaching here may have been uploaded by hil_ci.sh as + # the merge base, so it is untrusted input. `banner` carries forward -- it describes + # the conditions the earlier cells were collected under, and the .failed spec re-runs + # only FAILURES so those passes are never re-earned. `caveat` does NOT: it records how + # a RUN ENDED, and this attempt has not ended yet. Carrying it made a clean retry + # publish "HIL run abandoned" over a run where nothing was abandoned. + prior = {'rows': [], 'banner': ''} + if not fresh: + prior, _ = _load(report_dir) + acc = {r['board']: [dict(r['cells']), r['duration']] for r in prior['rows']} + prior_banner = prior['banner'] + + # current cells override prior for boards/tests that ran; a filtered run reports + # duration None, keeping the previous full-run value + for name, _, _, rows, *_ in mret: + if rows and not any(LOCKED_CELL in cells for _, cells, _ in rows): + # board ran for real: clear a stale lock-failure cell (its row is keyed by + # board name; test rows may be variant names) + stale = acc.get(name) + if stale is not None: + stale[0].pop(LOCKED_CELL, None) + # and the pool-timeout mark: write_timeout_report stamps it on a board that + # never reported, and update() below MERGES, so without this a board that + # passed clean on the retry kept a red cell for ever. + stale[0].pop(POOL_TIMEOUT_CELL, None) + if not stale[0]: + # variant-keyed boards never repopulate the board-name row, so drop it + # or it renders as a blank ghost row + del acc[name] + for row_label, cells, dur in rows: + row = acc.setdefault(row_label, [{}, None]) + # a row that ran is no longer pool-timed-out, whatever it is keyed by + row[0].pop(POOL_TIMEOUT_CELL, None) + # the boundary cell is only ever written on failure, so a re-run of this + # variant that cleared the boundary must drop the previous attempt's ❌ + if BOUNDARY_CELL not in cells: + row[0].pop(BOUNDARY_CELL, None) + row[0].update(cells) + if dur is not None: + row[1] = dur + + report_dir.mkdir(parents=True, exist_ok=True) + # by LINE, deduped: attempts repeat the same caveat far more often than they add a new + # one, and three copies of the D-state note reads as three incidents + seen, merged = set(), [] + for line in (prior_banner + banner).splitlines(): + if line.strip() and line not in seen: + seen.add(line) + merged.append(line) + banner = '\n'.join(merged) + '\n' if merged else '' + doc = {'rows': [{'board': k, 'cells': c, 'duration': d} for k, (c, d) in acc.items()], + 'banner': banner, 'scope': scope, 'caveat': caveat} + # through write_report, not hand-rolled: writing the JSON and only then rendering is + # the ordering write_report exists to forbid -- a render failure left the sidecar ahead + # of the markdown, which is the one invariant this module holds. + write_report(report_dir, doc) + return render_report(doc) + + +def _write_stuck_over_prior_md(report_dir: Path, doc: dict) -> None: + """Sidecar unrecoverable: rebuild it from the stuck rows alone, but leave the + markdown's existing table beneath the caveat rather than throwing real results away. + + The one place the md-is-a-rendering-of-the-json invariant is deliberately suspended, + because there is no readable json left for it to be a rendering of.""" + try: + prior = (report_dir / REPORT_MD).read_text(encoding='utf-8') + except (OSError, ValueError): + prior = '' + # Say so explicitly: those rows exist only as rendered text, so no later --accumulate + # can merge them back. Claiming the sidecar represents them would be false. + note = ('_The table below is a previous attempt\'s rendered output. The sidecar could ' + 'not be read, so those rows are NOT in it and will not survive another run._\n') + head = (doc['banner'] + '\n' if doc['banner'] else '') + doc['caveat'] + '\n' + note + body = prior if prior.strip() else render_matrix( + [(r['board'], r['cells'], r['duration']) for r in doc['rows']]) + report_dir.mkdir(parents=True, exist_ok=True) + (report_dir / REPORT_JSON).write_text(json.dumps(doc, indent=2) + '\n') + (report_dir / REPORT_MD).write_text(head + '\n' + body, encoding='utf-8') + + +def write_timeout_report(report_dir: Path, boards, secs: int, + banner: str = '', prefix: str = '') -> None: + """Leave a report behind when the worker pool has to be abandoned. + + map_async is all-or-nothing, so a timeout loses every per-board result and the report + dir would stay empty with no reason for the failure. Any prior attempt's rows are kept + and each stuck board is marked with a POOL_TIMEOUT_CELL beside them. + + `prefix` is the preflight rig-health verdict and goes to the BANNER, where rig health + lives and where an --accumulate retry carries it forward; the abandon notice goes to + the caveat, which does not carry. Folding both into the caveat is what made a clean + retry report an abandonment that had not happened.""" + try: + # names INSIDE the try: a roster entry that is not a dict raises here, and outside + # it that escaped and stranded the runner. + names = [b.get('name', '?') if isinstance(b, dict) else '?' for b in boards] + caveat = banner or ( + f'**HIL run abandoned: worker pool timed out after {secs}s.**\n\n' + f'No per-board results could be collected for this attempt. Rows other than ' + f'the {POOL_TIMEOUT_CELL} cells below are from an earlier attempt. Boards ' + f'dispatched:\n\n' + '\n'.join(f'- {n}' for n in names) + '\n') + doc, readable = _load(report_dir) + rows = doc['rows'] + by_board = {r['board']: r for r in rows} + for name in names: + row = by_board.get(name) + if row is None: + rows.append({'board': name, 'cells': {POOL_TIMEOUT_CELL: 'fail'}, + 'duration': None}) + else: + # _load guarantees `cells` is a dict, so a null-cells row from an uploaded + # sidecar can no longer send this down the fallback and publish a board + # that ate the whole pool guard as a pass. + row['cells'][POOL_TIMEOUT_CELL] = 'fail' + out = {'rows': rows, 'scope': doc['scope'], 'caveat': caveat, + 'banner': ((doc['banner'] + prefix) if prefix not in doc['banner'] + else doc['banner'])} + if not readable and (report_dir / REPORT_MD).is_file(): + # `readable` covers ABSENT as well as torn: an absent sidecar beside an intact + # markdown used to re-render from the stuck row alone and destroy real results. + _write_stuck_over_prior_md(report_dir, out) + return + write_report(report_dir, out) + except Exception as e: # noqa: BLE001 + # Deliberately broad: this is the first statement of the pool-abandon path, so ANY + # escape skips kill_pool_children and os._exit and strands the runner. + _p(f'warning: cannot write {REPORT_MD} to {report_dir}: {e}', flush=True) + try: + # Same wording as above and the same guarded name extraction -- the fallback + # used to re-derive b.get("name") outside any try and raise identically, so a + # malformed roster left NO artifact at all. + names = [b.get('name', '?') if isinstance(b, dict) else '?' for b in boards] + head = (prefix + '\n' if prefix else '') + (banner or ( + f'**HIL run abandoned: worker pool timed out after {secs}s.**\n\n' + f'No per-board results could be collected for this attempt, so the table ' + f'below (if any) is from an earlier one. Boards dispatched:\n\n' + + '\n'.join(f'- {n}' for n in names) + '\n')) + try: + prior = (report_dir / REPORT_MD).read_text(encoding='utf-8') + except (OSError, ValueError): + prior = '' + report_dir.mkdir(parents=True, exist_ok=True) + (report_dir / REPORT_MD).write_text( + head + (f'\n{prior}' if prior else ''), encoding='utf-8') + except Exception as e2: # noqa: BLE001 + _p(f'warning: fallback {REPORT_MD} write failed too: {e2}', flush=True) + + +def variants_of(cfg: dict, board: str) -> list: + for b in cfg.get('boards', []): + if b['name'] == board: + return [v['name'] for v in (b.get('variant') or [])] or [board] + return [board] + + +def summarize(cfg: dict, boards: list, report: dict) -> dict: + # .get, not a subscript: this is the one reader an agent's verdict depends on, and a + # row without 'board' used to kill the CLI with a traceback and no results at all -- + # hil-validate.js then reports every board as "hil-operator returned no entry". + rows = {r['board']: r.get('cells') or {} + for r in (report.get('rows') or []) + if isinstance(r, dict) and 'board' in r} + owner = {v['name']: b['name'] for b in cfg.get('boards', []) + for v in (b.get('variant') or [])} + results = [] + for board in boards: + names = variants_of(cfg, board) + mine = {n: rows[n] for n in names if n in rows} + # a variant name that is neither declared nor prefixed cannot be attributed; the + # `-` fallback only helps ad-hoc builds, it is not the primary path. It must + # also never steal a row DECLARED by another board: a declared variant need not start + # with its own board's name, so it may happen to start with this board's name plus '-'. + mine.update({n: c for n, c in rows.items() + if n.startswith(f'{board}-') and n not in mine + and owner.get(n, board) == board}) + # the BOARD-name row too: hil_test writes lock contention and pool timeouts keyed + # by board name, but variants_of returns only DECLARED variant names -- and + # nanoch32v203 / ch32v307v_r1_1v0 declare none equal to their board name. Without + # this those rows are invisible, so a lock held by concurrent CI is published as a + # hardware FAIL and hil-validate.js never retries it. + if board in rows and board not in mine: + mine[board] = rows[board] + if not mine: + results.append({'board': board, 'ran': False, 'pass': False, 'locked': False, + 'detail': 'no report row for this board'}) + continue + # a wedge outranks lock contention: `locked` short-circuits `detail` below, so a + # stale board-locked cell from an earlier attempt used to mask the pool-timeout + # cell the retry added -- publishing a board that hung the rig as LOCKED, which + # hil-validate.js then RE-RUNS, paying another pool guard on it. + wedged = any(POOL_TIMEOUT_CELL in cells for cells in mine.values()) + locked = not wedged and any(LOCKED_CELL in cells for cells in mine.values()) + bad = [] + for vname, cells in sorted(mine.items()): + for test, val in sorted(cells.items()): + if test == LOCKED_CELL: + continue + if cell_state(val) == 'fail': + bad.append(f'{vname} {test}: {val}') + ok = not bad and not locked + if locked: + detail = 'held by another holder; not flashed' + elif bad: + detail = '; '.join(bad) + else: + detail = f'{len(mine)} variant(s), {sum(len(c) for c in mine.values())} cell(s) ok' + results.append({'board': board, 'ran': True, 'pass': ok, 'locked': locked, + 'detail': detail}) + # `caveat` too: an abandoned or no-boards run says so THERE, and this JSON is all + # an agent gets -- leaving it in the sidecar puts it back where only a human looks. + return {'results': results, 'banner': report.get('banner', ''), + 'caveat': report.get('caveat', '')} + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument('config_file') + ap.add_argument('-b', '--board', action='append', default=[], + help='boards to report on; default: every board in the config') + ap.add_argument('--report-dir', default='.', help=f'where {REPORT_JSON} lives (default: cwd)') + a = ap.parse_args() + + cfg = json.loads(Path(a.config_file).read_text()) + boards = a.board or [b['name'] for b in cfg.get('boards', [])] + jpath = Path(a.report_dir) / REPORT_JSON + if not jpath.is_file(): + print(f'error: {jpath} not found -- did hil_test.py run in this directory?', + file=sys.stderr) + return 1 + # through _load, like every writer: feeding raw JSON to summarize left the one reader an + # agent's verdict depends on crashing on the malformed sidecars the writers tolerate. + doc, _ = _load(Path(a.report_dir)) + json.dump(summarize(cfg, boards, doc), sys.stdout, indent=2) + print() + return 0 + +if __name__ == '__main__': + sys.exit(main()) diff --git a/test/hil/helper/hil_summary.py b/test/hil/helper/hil_summary.py deleted file mode 100644 index e566bead0..000000000 --- a/test/hil/helper/hil_summary.py +++ /dev/null @@ -1,115 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-License-Identifier: MIT -"""Fold hil_report.json into one machine-readable verdict per BOARD. - -A workflow driving hil_test.py through an operator agent has no filesystem access, so the -agent has to carry the results across. It must carry them, not retype them: the previous -design asked the agent to transcribe the markdown table, and every defect found in four -review rounds came from re-parsing that prose -- variant row names vs board names, -`board locked` vs `board-locked`, folding several variant rows into one verdict, rows that -matched no board. All of it is a join, and the join belongs here, where the roster is. - -Report rows are named per VARIANT (hil_test.py builds them from `vname`), and a variant name -is not required to start with the board name -- nanoch32v203 produces only `-fsdev`/`-usbfs`, -ch32v307v_r1_1v0 only `-usbhs`/`-usbfs`. The config is what maps them back. - -Emits, on stdout: - {"results": [{"board", "ran", "pass", "locked", "detail"}...], "banner": str} - -`locked` is a field, not a prefix to grep for. `ran` false means the board produced no row at -all, which is not the same as failing. - -Usage: hil_summary.py [-b BOARD]... [--report-dir DIR] -""" -import argparse -import json -import sys -from pathlib import Path - -FAIL_ICON, SKIP_ICON = '❌', '⚪' # a pass needs no icon: unmarked = pass -LOCKED_CELL = 'board-locked' - - -def cell_state(v: str) -> str: - """'pass' | 'fail' | 'skip' -- the EXACT classifier hil_test.py's own tally uses - (cell_kind in render_matrix): 'fail' or a ❌ prefix is a failure, 'skip' or a ⚪ - prefix is a skip, and EVERYTHING ELSE is a pass. That last arm is load-bearing: a - passing test may return a plain metric string ('480.0 MBps') that lands in the cell - unprefixed, while failures are guaranteed marked -- TestFail's docstring pins that its - metric is icon-prefixed precisely so render/tally treat it as a failure. Classifying - unknown shapes as fail here would publish a green table as a red verdict.""" - if v == 'fail' or v.startswith(FAIL_ICON): - return 'fail' - if v == 'skip' or v.startswith(SKIP_ICON): - return 'skip' - return 'pass' - - -def variants_of(cfg: dict, board: str) -> list: - for b in cfg.get('boards', []): - if b['name'] == board: - return [v['name'] for v in (b.get('variant') or [])] or [board] - return [board] - - -def summarize(cfg: dict, boards: list, report: dict) -> dict: - rows = {r['board']: r.get('cells') or {} for r in report.get('rows', [])} - owner = {v['name']: b['name'] for b in cfg.get('boards', []) - for v in (b.get('variant') or [])} - results = [] - for board in boards: - names = variants_of(cfg, board) - mine = {n: rows[n] for n in names if n in rows} - # a variant name that is neither declared nor prefixed cannot be attributed; the - # `-` fallback only helps ad-hoc builds, it is not the primary path. It must - # also never steal a row DECLARED by another board: a declared variant need not start - # with its own board's name, so it may happen to start with this board's name plus '-'. - mine.update({n: c for n, c in rows.items() - if n.startswith(f'{board}-') and n not in mine - and owner.get(n, board) == board}) - if not mine: - results.append({'board': board, 'ran': False, 'pass': False, 'locked': False, - 'detail': 'no report row for this board'}) - continue - locked = any(LOCKED_CELL in cells for cells in mine.values()) - bad = [] - for vname, cells in sorted(mine.items()): - for test, val in sorted(cells.items()): - if test == LOCKED_CELL: - continue - if cell_state(str(val)) == 'fail': - bad.append(f'{vname} {test}: {val}') - ok = not bad and not locked - if locked: - detail = 'held by another holder; not flashed' - elif bad: - detail = '; '.join(bad) - else: - detail = f'{len(mine)} variant(s), {sum(len(c) for c in mine.values())} cell(s) ok' - results.append({'board': board, 'ran': True, 'pass': ok, 'locked': locked, - 'detail': detail}) - return {'results': results, 'banner': report.get('banner', '')} - - -def main() -> int: - ap = argparse.ArgumentParser() - ap.add_argument('config_file') - ap.add_argument('-b', '--board', action='append', default=[], - help='boards to report on; default: every board in the config') - ap.add_argument('--report-dir', default='.', help='where hil_report.json lives (default: cwd)') - a = ap.parse_args() - - cfg = json.loads(Path(a.config_file).read_text()) - boards = a.board or [b['name'] for b in cfg.get('boards', [])] - jpath = Path(a.report_dir) / 'hil_report.json' - if not jpath.is_file(): - print(f'error: {jpath} not found -- did hil_test.py run in this directory?', - file=sys.stderr) - return 1 - json.dump(summarize(cfg, boards, json.loads(jpath.read_text())), sys.stdout, indent=2) - print() - return 0 - - -if __name__ == '__main__': - sys.exit(main()) diff --git a/test/hil/hil_ci.sh b/test/hil/hil_ci.sh index 514b0f174..daa787242 100644 --- a/test/hil/hil_ci.sh +++ b/test/hil/hil_ci.sh @@ -210,6 +210,69 @@ rm -rf -- "$1" mkdir -p -- "$1/test/hil/helper" "$1/examples" REMOTE +# The --accumulate merge base. The wipe above just cleared REMOTE_DIR, and +# accumulate_report merges onto the sidecar in the RUN's cwd (hil_test.py:2193 sets +# `fresh = not args.accumulate`, and only a non-fresh run reads it) -- so without this a +# remote retry starts from nothing and its one-row table REPLACES the full-fleet one it was +# meant to extend. The copy-back at the end of this script has always existed; this is the +# other half of it. +# +# Gated, not unconditional: a fresh run unlinks the sidecar anyway (hil_test.py:2244), so +# uploading there is wasted work that also obscures what the wipe means. +# +# .failed is deliberately NOT uploaded: hil_test.py only ever writes it, never +# reads it -- the retry spec reaches the rig as the -b/-bt arguments the caller expanded +# from it (`hil_ci.sh $(cat .failed)`). +# argparse decides, not a case arm: hil_test.py declares `-a, --accumulate`, so argparse +# also accepts `-av`, `-va`, `--accum` and `--acc` -- and hil-validate.js tells the +# operator to retry "adding -v", which makes `-av` the natural spelling. A hand-rolled +# match missed all four: no upload, and the else-branch warning never fired either, so the +# one-row table replaced the full-fleet one in silence. +ACCUMULATE=$(python3 - ${ARGS[@]+"${ARGS[@]}"} <<'PY' +import argparse, sys +p = argparse.ArgumentParser(add_help=False) +p.add_argument('-a', '--accumulate', action='store_true') +p.add_argument('-v', '--verbose', action='store_true') # so -av/-va bundle as they do there +print(1 if p.parse_known_args(sys.argv[1:])[0].accumulate else 0) +PY +) || ACCUMULATE=0 +if [ "$ACCUMULATE" = 1 ]; then + if [ -f "$ROOT_DIR/hil_report.json" ]; then + # Provenance: hil_report.json is not namespaced by CONFIG or REMOTE (build.yml and + # pr_comment.yml read that exact name), so a `REMOTE=hifiphile CONFIG=.../hfp.json` + # run leaves an hfp sidecar behind that a later ci.lan retry would merge, publishing + # boards that never ran here. Require at least one row to belong to THIS roster. + if python3 - "$ROOT_DIR/hil_report.json" "$CONFIG" <<'PY' +import json, sys +try: + rows = json.load(open(sys.argv[1])).get('rows') or [] + cfg = json.load(open(sys.argv[2])).get('boards') or [] +except Exception: + sys.exit(1) +known = set() +for b in cfg: + known.add(b.get('name')) + known.update(v.get('name') for v in (b.get('variant') or [])) +sys.exit(0 if not rows or any(r.get('board') in known for r in rows if isinstance(r, dict)) + else 1) +PY + then + echo "==> Uploading hil_report.json as the --accumulate merge base" + scp -q "$ROOT_DIR/hil_report.json" "$REMOTE:$REMOTE_DIR/" + else + echo "==> warning: $ROOT_DIR/hil_report.json holds no board from $(basename "$CONFIG")" \ + "-- it is from another rig or config, so it is NOT being uploaded; this run's" \ + "table will REPLACE rather than extend" >&2 + fi + else + # Loud, because this is the failure mode: the run still succeeds, and quietly + # publishes a small table where a full one used to be. + echo "==> warning: --accumulate was requested but $ROOT_DIR/hil_report.json does not" \ + "exist, so there is nothing to merge onto -- this run's table will REPLACE the" \ + "previous one rather than extend it" >&2 + fi +fi + # Copy HIL test script and config echo "==> Copying test scripts" scp -q "$ROOT_DIR/test/hil/hil_test.py" \ @@ -223,7 +286,7 @@ scp -q "$ROOT_DIR/test/hil/helper/__init__.py" \ "$ROOT_DIR/test/hil/helper/hil_util.py" \ "$ROOT_DIR/test/hil/helper/hil_health.py" \ "$ROOT_DIR/test/hil/helper/hil_lock.py" \ - "$ROOT_DIR/test/hil/helper/hil_summary.py" \ + "$ROOT_DIR/test/hil/helper/hil_report.py" \ "$REMOTE:$REMOTE_DIR/test/hil/helper/" # Copy only firmware binaries (elf/bin/hex) plus esptool metadata @@ -316,19 +379,41 @@ REMOTE # Copy the generated report back to the local checkout (best-effort; the run's # exit code is preserved regardless of whether a report was produced). -scp -q "$REMOTE:$REMOTE_DIR/hil_report.md" "$ROOT_DIR/hil_report.md" \ - && echo "==> Report copied to $ROOT_DIR/hil_report.md" \ - || echo "==> warning: no hil_report.md copied back" >&2 +# rm -f FIRST, exactly as the sidecar loop below does: the markdown and the JSON are two +# halves of ONE document now, so leaving a stale table behind when the copy fails -- beside +# a sidecar that was correctly removed -- publishes last run's green results under this +# run's red job, and the operator's hil_report.py call exits 1 against the missing sidecar. +# Fetch BOTH halves to temps and commit them as a pair. Separate fetch/rename meant a +# markdown that arrived beside a sidecar that did not left the local pair failing the +# rendering invariant, and the next --accumulate retry merging the wrong base. Deleting +# first and then scp'ing was worse still: an ssh drop at the end of a 60-minute run +# destroyed the report outright. +md_ok=0; json_ok=0 +scp -q "$REMOTE:$REMOTE_DIR/hil_report.md" "$ROOT_DIR/hil_report.md.tmp" 2>/dev/null \ + && [ -f "$ROOT_DIR/hil_report.md.tmp" ] && md_ok=1 +scp -q "$REMOTE:$REMOTE_DIR/hil_report.json" "$ROOT_DIR/hil_report.json.tmp" 2>/dev/null \ + && [ -f "$ROOT_DIR/hil_report.json.tmp" ] && json_ok=1 +if [ "$md_ok" = 1 ] && [ "$json_ok" = 1 ]; then + mv -f "$ROOT_DIR/hil_report.md.tmp" "$ROOT_DIR/hil_report.md" + mv -f "$ROOT_DIR/hil_report.json.tmp" "$ROOT_DIR/hil_report.json" + echo "==> Report copied to $ROOT_DIR/hil_report.md (+ sidecar)" +else + rm -f "$ROOT_DIR/hil_report.md.tmp" "$ROOT_DIR/hil_report.json.tmp" + # All or nothing: a half-copied pair is worse than none. The stale local markdown goes + # because that is what gets pasted into a PR as this run's results; the stale sidecar + # goes with it so the two cannot disagree. + rm -f "$ROOT_DIR/hil_report.md" "$ROOT_DIR/hil_report.json" + echo "==> warning: report copy-back incomplete (md=$md_ok json=$json_ok); removed the" \ + "stale local pair -- an --accumulate retry has no merge base until a run succeeds" >&2 +fi -# The re-run spec and the JSON sidecar live in the run's cwd on the rig (REMOTE_DIR), and the -# next invocation rm -rf's it. Without copying them back, the `--accumulate` retry every doc on -# this branch prescribes has nothing to read and nothing to merge onto. Delete the local copies -# FIRST: a green run writes no .failed, so a silent no-op scp would leave last run's spec in -# the checkout looking current, and "retry from the spec" would re-flash boards that passed. -for extra in "$(basename "$CONFIG").failed" hil_report.json; do - rm -f "$ROOT_DIR/$extra" - scp -q "$REMOTE:$REMOTE_DIR/$extra" "$ROOT_DIR/$extra" 2>/dev/null \ - && echo "==> $extra copied to $ROOT_DIR/$extra" || true -done +# The re-run spec lives in the run's cwd on the rig and the next invocation rm -rf's it. +# Delete the local copy first: a green run writes no .failed, so a silent no-op scp would +# leave last run's spec looking current and "retry from the spec" would re-flash boards +# that passed. +spec="$(basename "$CONFIG").failed" +rm -f "$ROOT_DIR/$spec" +scp -q "$REMOTE:$REMOTE_DIR/$spec" "$ROOT_DIR/$spec" 2>/dev/null \ + && echo "==> $spec copied to $ROOT_DIR/$spec" || true exit $rc diff --git a/test/hil/hil_test.py b/test/hil/hil_test.py index fcd7c7e6f..83e5082d4 100755 --- a/test/hil/hil_test.py +++ b/test/hil/hil_test.py @@ -64,7 +64,7 @@ from multiprocessing import TimeoutError as MpTimeoutError sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # PYTHONSAFEPATH drops it import hil_flash -from helper import hil_health, hil_lock, hil_util +from helper import hil_health, hil_lock, hil_report, hil_util from helper.hil_util import device_tests, dual_tests, host_test # Raw Lock/Semaphore objects in Pool initargs are inheritable only under fork @@ -106,9 +106,6 @@ STATUS_OK = "\033[32mOK\033[0m" STATUS_FAILED = "\033[31mFailed\033[0m" STATUS_SKIPPED = "\033[33mSkipped\033[0m" -# Plain (non-ANSI) cell symbols for hil_report.md; a missing binary counts as skipped. -REPORT_CELL = {'pass': '✅', 'fail': '❌', 'skip': '⚪'} - class TestFail(AssertionError): """Fail a test but still surface a metric string in its report cell (e.g. usbtest's '❌ 29/30' @@ -871,7 +868,7 @@ def test_device_cdc_msc_throughput(board): # payload, so an HS board reads as suspiciously slow. Say so rather than publish a green # cell whose scale is a guess. scale = '' if speed_known else ' FS?' - return f'{REPORT_CELL["pass"]} C {pair(cdc_r, cdc_w)} M {pair(msc_r, msc_w)}{scale}' + return f'{hil_report.REPORT_CELL["pass"]} C {pair(cdc_r, cdc_w)} M {pair(msc_r, msc_w)}{scale}' def test_device_dfu(board): @@ -1364,7 +1361,7 @@ def test_device_usbtest(board): f'no cafe:4010 device with serial {uid}' if seen is False else f'cannot tell whether cafe:4010 {uid} is present: the bounded sysfs reads did ' f'not answer{hil_util.sysfs_blind_note()}', - metric=f'{REPORT_CELL["fail"]} 0/30') + metric=f'{hil_report.REPORT_CELL["fail"]} 0/30') # settle: right after flashing the enumeration can bounce once (and on dual-port parts # the other port's stale node — same serial and PID — lingers), and testusb run into # that gap sees the device drop mid-case @@ -1454,7 +1451,7 @@ def test_device_usbtest(board): board_wedged = (f'{board["name"]}: usbtest reported a hang and was killed ' f'before it could report a verdict') raise TestFail(f'usbtest did not run: {detail}', - metric=f'{REPORT_CELL["fail"]} 0/30') + metric=f'{hil_report.REPORT_CELL["fail"]} 0/30') # A HUNG case that recovery could not clear leaves a D-state holder on this board's # usbfs node. Latch it: the remaining examples would each flash THROUGH that node, @@ -1485,9 +1482,9 @@ def test_device_usbtest(board): # the re-run spec. parsed=True: a retry re-pays the whole battery to re-observe a # wedge, and flashes through the poisoned node to do it. raise TestFail(f'usbtest {passed}/{total} but the device wedged ({board_wedged})', - metric=f'{REPORT_CELL["fail"]} {passed}/{total}', parsed=True) + metric=f'{hil_report.REPORT_CELL["fail"]} {passed}/{total}', parsed=True) if failed == 0 and notrun == 0 and total > 0: - return f'{REPORT_CELL["pass"]} {passed}/{total}' + return f'{hil_report.REPORT_CELL["pass"]} {passed}/{total}' bad = [c.get('num') for c in data.get('cases', []) if c.get('status') not in ('PASS', 'BUDGET')] why = f'usbtest {passed}/{total}' @@ -1503,7 +1500,7 @@ def test_device_usbtest(board): why += f'; {notrun} case(s) never ran ({reason}), so this says nothing about them' # parsed ONLY when every case ran: an aborted battery (budget expiry, kernel hang, bus # drop) leaves BUDGET entries, and those are exactly what a reflash retry can fix. - raise TestFail(why, metric=f'{REPORT_CELL["fail"]} {passed}/{total}', + raise TestFail(why, metric=f'{hil_report.REPORT_CELL["fail"]} {passed}/{total}', parsed=(notrun == 0)) @@ -1705,10 +1702,6 @@ def build_board(board: Board) -> tuple[str, int]: return name, failed -# pseudo-test column for a variant boundary the park-flash could not clear (see below) -BOUNDARY_CELL = 'same-PID boundary' - - def test_board(board: Board) -> tuple[str, int, list[str], list, float]: swept = False name = board['name'] @@ -1722,7 +1715,7 @@ def test_board(board: Board) -> tuple[str, int, list[str], list, float]: log_line(f'{name:25} {STATUS_FAILED}: {e}') # visible report row so the ❌ matches the exit code; failed-tests stays empty so a # re-run repeats the whole board (no bogus -bt filter) - return name, 1, [], [(name, {'board-locked': 'fail'}, None)], 0.0 + return name, 1, [], [(name, {hil_report.LOCKED_CELL: 'fail'}, None)], 0.0 # after the lock: flock wait behind a concurrent run is not board cost t_board = time.monotonic() try: @@ -1809,7 +1802,7 @@ def test_board(board: Board) -> tuple[str, int, list[str], list, float]: # charging again would double-count one incident in the exit code if not wedge_skip: err_count += 1 - cells[BOUNDARY_CELL] = 'fail' + cells[hil_report.BOUNDARY_CELL] = 'fail' # blaming run_list[0] would re-run an innocent test that then passes, # leaving the boundary unretested; re-run the whole board instead board_wide_fail = True @@ -1825,7 +1818,7 @@ def test_board(board: Board) -> tuple[str, int, list[str], list, float]: # Do NOT flash through a poisoned node: each attempt enumerates into # it, blocks uninterruptibly and leaves another stray behind. Report # the skip so the cell is not mistaken for a pass. - cells[test] = f'{REPORT_CELL["skip"]} board wedged' + cells[test] = f'{hil_report.REPORT_CELL["skip"]} board wedged' # ...and re-run the WHOLE board, like the boundary-failure path above: # these tests never executed, so naming them individually in the .failed # spec is not enough -- an --accumulate re-run that fixes only the wedged @@ -1893,8 +1886,6 @@ def test_board(board: Board) -> tuple[str, int, list[str], list, float]: _lock_fh.close() -REPORT_MD = 'hil_report.md' -REPORT_JSON = 'hil_report.json' # controller hints from previous runs: uid -> {'name', 'pci', 'duration'}. Only 'pci' is # consumed (dispatch order and first-flash budgeting, never battery serialization). PCI # addresses are boot-stable, so the cache survives reboots and goes stale on re-cabling. @@ -1912,66 +1903,6 @@ def schedule_boards(boards: list, pci_of_uid: dict) -> list: return [b for grp in itertools.zip_longest(*buckets.values()) for b in grp if b is not None] -def render_matrix(rows_all: list) -> str: - """Render rows (list of (row_label, {example: status}, duration)) as an aligned - markdown matrix: columns = tests (bare names) centered, boards left-aligned, - per-row duration as the trailing column.""" - seen = set() - for _, cells, _ in rows_all: - seen.update(cells) - if not seen: - return 'No tests were run.' - - # metric-bearing columns pinned first, the rest alphabetical: stable regardless of the - # shuffled execution order - pinned = ['usbtest', 'cdc_msc_throughput', 'msc_file_explorer', 'msc_file_explorer_freertos'] - - def col_key(t): - name = t.rsplit('/', 1)[-1] - return (pinned.index(name) if name in pinned else len(pinned), name, t) - - columns = sorted(seen, key=col_key) - headers = [c.rsplit('/', 1)[-1] for c in columns] + ['duration'] # bare example names - - def cell(cells, col): - v = cells.get(col) - if v is None: - return '' - return REPORT_CELL.get(v, v) # status symbol, or a metric string (e.g. speed) verbatim - - rows_vals = [(lbl, [cell(cells, c) for c in columns] + [dur or '']) - for lbl, cells, dur in rows_all] - board_hdr = 'Board' - board_w = max([len(board_hdr)] + [len(lbl) for lbl, _ in rows_vals]) - col_w = [max([len(h)] + [len(vals[i]) for _, vals in rows_vals]) - for i, h in enumerate(headers)] - - def line(label, values): - padded = [label.ljust(board_w)] + [v.center(w) for v, w in zip(values, col_w)] - return '| ' + ' | '.join(padded) + ' |' - - header = line(board_hdr, headers) - sep = '| ' + '-' * board_w + ' | ' + ' | '.join(':' + '-' * (w - 2) + ':' for w in col_w) + ' |' - body = [line(lbl, vals) for lbl, vals in rows_vals] - - # tally run cells (not-run cells are absent from the dicts). A cell is a bare status or - # a metric string carrying its own icon ("❌ 29/30"), so classify by the leading icon. - def cell_kind(v): - if v == 'fail' or (isinstance(v, str) and v.startswith(REPORT_CELL['fail'])): - return 'fail' - if v == 'skip' or (isinstance(v, str) and v.startswith(REPORT_CELL['skip'])): - return 'skip' - return 'pass' - kinds = [cell_kind(v) for _, cells, _ in rows_all for v in cells.values()] - failed = kinds.count('fail') - skipped = kinds.count('skip') - passed = kinds.count('pass') - summary = (f'**{REPORT_CELL["pass"]} {passed} passed · {REPORT_CELL["fail"]} {failed} failed · ' - f'{REPORT_CELL["skip"]} {skipped} skipped · blank not run**') - - return summary + '\n\n' + '\n'.join([header, sep] + body) - - def _write_failed_spec(failed_fname: Path, report_dir: Path, mret: list) -> None: """Re-run spec: only the failed boards (-b), each restricted to its own failed tests (-bt); a board with failures but no test list re-runs entirely. @@ -2083,87 +2014,13 @@ def _blind_note(mret: list) -> str: f'{", ".join(blind)}. See the usb-kernel-recover skill.\n') -def accumulate_report(mret: list, report_dir: Path, fresh: bool, scope: str = '', - banner: str = '') -> str: - """Merge this run's results into hil_report.json in report_dir, then (re)write - the markdown matrix to hil_report.md. `fresh` (a first run, no --accumulate) - starts a new report; otherwise a re-run accumulates so boards/tests that - already passed are preserved while re-run cells are updated. `scope` names the - board filter, if any, so a scoped table is not mistaken for a full one. - Returns the md.""" - acc = {} # ordered {row_label: [cells dict, duration str|None]} - prior_banner = '' - jpath = report_dir / REPORT_JSON - if not fresh and jpath.is_file(): - try: - saved = json.loads(jpath.read_text()) - # CI keys the report dir by run id, so the sidecar is from an earlier attempt - for entry in saved.get('rows', []): - acc[entry['board']] = [dict(entry['cells']), entry.get('duration')] - # ... and so is the caveat those cells were collected under. A rerun on a rig - # that has since recovered contributes no banner, and the .failed spec reruns - # only FAILURES -- so the earlier attempt's passes are never re-earned and - # would be published as clean results of a rig that was not. - prior_banner = saved.get('banner', '') - except (ValueError, KeyError, TypeError): - pass # corrupt/old sidecar: start fresh - - # current cells override prior for boards/tests that ran; a filtered run reports - # duration None, keeping the previous full-run value - for name, _, _, rows, *_ in mret: - if rows and not any('board-locked' in cells for _, cells, _ in rows): - # board ran for real: clear a stale lock-failure cell (its row is keyed by - # board name; test rows may be variant names) - stale = acc.get(name) - if stale is not None: - stale[0].pop('board-locked', None) - if not stale[0]: - # variant-keyed boards never repopulate the board-name row, so drop it - # or it renders as a blank ghost row - del acc[name] - for row_label, cells, dur in rows: - row = acc.setdefault(row_label, [{}, None]) - # the boundary cell is only ever written on failure, so a re-run of this - # variant that cleared the boundary must drop the previous attempt's ❌ - if BOUNDARY_CELL not in cells: - row[0].pop(BOUNDARY_CELL, None) - row[0].update(cells) - if dur is not None: - row[1] = dur - - report_dir.mkdir(parents=True, exist_ok=True) - # by LINE, deduped: attempts repeat the same caveat far more often than they add a new - # one, and three copies of the D-state note reads as three incidents - seen, merged = set(), [] - for line in (prior_banner + banner).splitlines(): - if line.strip() and line not in seen: - seen.add(line) - merged.append(line) - banner = '\n'.join(merged) + '\n' if merged else '' - jpath.write_text(json.dumps({'rows': [{'board': k, 'cells': c, 'duration': d} - for k, (c, d) in acc.items()], - 'banner': banner}, indent=2) + '\n') - - md = render_matrix([(k, c, d) for k, (c, d) in acc.items()]) - if scope: - # a scoped run's small table is otherwise indistinguishable from a full one, and - # it replaces the previous full table in the sticky PR comment - md = f'_Scoped run: {scope}. Boards/tests not listed were not run._\n\n' + md - # LAST, so it is outermost: a rig-health caveat outranks the table AND the scope note, - # and the top of the report is where hil/SKILL.md tells the agent to look for it. - if banner: - md = banner + '\n' + md - (report_dir / REPORT_MD).write_text(md + '\n', encoding='utf-8') - return md - - # containment paths print through hil_health._p: stdout may already be a dead pipe (a # dropped ssh session), and a BrokenPipeError there would skip os._exit _p = hil_health._p def _abandon_exit(pool, mgr, abandoned: bool, err_count: int, - report: Path | None = None) -> None: + report_dir: Path | None = None) -> None: """Free the runner when the pool could not be shut down. Returns only if not abandoned. Must run even while an exception is propagating: multiprocessing's atexit handler @@ -2199,24 +2056,11 @@ def _abandon_exit(pool, mgr, abandoned: bool, err_count: int, 'stay locked.', flush=True) # A report already written by accumulate_report says nothing about the abandon, and a # green table under a red job is how an agent ends up pasting it as this run's result. - # Prepend the caveat; best-effort, never at the cost of exiting. - if report is not None: - try: - if report.exists(): - # utf-8 explicitly (the cells are ✅/❌/⚪) and catch ValueError too: a torn - # report or a LANG=C locale raises UnicodeDecodeError -- NOT an OSError -- - # straight past os._exit, stranding the runner. - body = report.read_text(encoding='utf-8', errors='replace') - # Only when no banner is there yet, searched anywhere in the head rather - # than at char 0: write_timeout_report's banner must stay FIRST (its table - # is a PREVIOUS attempt's) and it puts the rig-health quote above itself. - if '**HIL run ab' not in body[:2000]: - report.write_text( - '**HIL run abandoned: the worker pool would not shut down.** The ' - 'table below was collected before the abandon; treat board ' - 'results as unverified.\n\n' + body, encoding='utf-8') - except (OSError, ValueError): - pass + # Set the caveat in the DOCUMENT -- prepending to the markdown alone left the sidecar, + # which is all hil_report.summarize() and therefore an agent ever sees, saying nothing. + # Best-effort, never at the cost of exiting. + if report_dir is not None: + hil_report.mark_report_abandoned(report_dir, 'the worker pool would not shut down.') try: sys.stdout.flush() except OSError: @@ -2305,13 +2149,11 @@ def main() -> None: print(msg, flush=True) # loud AND leaving evidence: exiting with no report at all lets the PR comment # keep the previous push's stale table under a red job - try: - rd = Path(os.environ.get('HIL_REPORT_DIR', '.')) - rd.mkdir(parents=True, exist_ok=True) - (rd / REPORT_MD).write_text(f'**HIL run selected no boards.** {msg}\n', - encoding='utf-8') - except OSError: - pass + rd = Path(os.environ.get('HIL_REPORT_DIR', '.')) + # fresh must be threaded through: this runs BEFORE the `if fresh:` wipe below, so + # defaulting it here wiped an --accumulate run's accumulated rows -- the exact + # regression the parameter exists to prevent. + hil_report.mark_report_no_boards(rd, msg, fresh=not args.accumulate) sys.exit(1) @@ -2387,14 +2229,14 @@ def main() -> None: # BEFORE Manager()/Pool(), not inside the try: hil_ci.sh reuses a persistent REMOTE_DIR # and scp's the report back unconditionally, so if a fork failure (OSError/EAGAIN right # after a convoy -- the case this whole block guards) skipped the wipe, the finally's - # _abandon_exit would prepend "HIL run abandoned" to the PREVIOUS run's table and + # _abandon_exit would stamp "HIL run abandoned" onto the PREVIOUS run's report and # publish last night's board results as this run's. Nothing is live yet here, so an # OSError from the wipe itself just exits with its traceback -- it cannot strand the # interpreter in multiprocessing's unbounded atexit join, which is what deferring it # was protecting against. if fresh: report_dir.mkdir(parents=True, exist_ok=True) - for f in (REPORT_JSON, REPORT_MD): + for f in (hil_report.REPORT_JSON, hil_report.REPORT_MD): (report_dir / f).unlink(missing_ok=True) failed_fname.unlink(missing_ok=True) try: @@ -2446,9 +2288,9 @@ def main() -> None: f'are this run\'s; {len(stuck)} never reported and are NOT in ' f'the table: {", ".join(stuck)}. Re-run covers those.\n') try: - accumulate_report(mret, report_dir, fresh, '', + hil_report.accumulate_report(mret, report_dir, fresh, '', health_banner + _blind_note(mret) - + _stray_note(mret) + banner) + + _stray_note(mret), caveat=banner) except Exception as rerr: # noqa: BLE001 - the raise below must still happen # FALL BACK, do not just warn: accumulate_report can raise on an # unwritable/root-owned report dir or a torn JSON, and _abandon_exit @@ -2458,9 +2300,9 @@ def main() -> None: print(f'warning: partial report failed: {type(rerr).__name__}: {rerr}; ' f'falling back to the board list', flush=True) try: - hil_health.write_timeout_report( + hil_report.write_timeout_report( report_dir, [b for b in config_boards - if b['name'] in stuck], POOL_TIMEOUT, REPORT_MD, + if b['name'] in stuck], POOL_TIMEOUT, prefix=health_banner) except Exception as re2: # noqa: BLE001 print(f'warning: fallback report failed too: ' @@ -2485,9 +2327,9 @@ def main() -> None: f'{len(mret)} board(s) below finished and are this run\'s; ' f'{len(stuck)} did not report: {", ".join(stuck)}.\n') try: - accumulate_report(mret, report_dir, fresh, '', + hil_report.accumulate_report(mret, report_dir, fresh, '', health_banner + _blind_note(mret) - + _stray_note(mret) + banner) + + _stray_note(mret), caveat=banner) except Exception as re2: # noqa: BLE001 - the raise below must still happen print(f'warning: partial report failed: {type(re2).__name__}: {re2}', flush=True) @@ -2578,12 +2420,12 @@ def main() -> None: # looks exactly like a full run that happened to be small scoped = sorted(set(args.board) | set(board_test)) scope = f'{len(scoped)} board(s) — {", ".join(scoped)}' if scoped else '' - report = accumulate_report(mret, report_dir, fresh, scope, + report = hil_report.accumulate_report(mret, report_dir, fresh, scope, health_banner + _blind_note(mret) + _stray_note(mret)) print() print(report) - print(f'\nReport written to {(report_dir / REPORT_MD).resolve()}') + print(f'\nReport written to {(report_dir / hil_report.REPORT_MD).resolve()}') duration = time.time() - duration print() @@ -2594,7 +2436,7 @@ def main() -> None: # In the finally, not after: any raise above (accumulate_report sits outside the # OSError handler) would skip the abandon path and unwind into multiprocessing's # unbounded atexit join, hanging the runner. - _abandon_exit(pool, mgr, pool_abandoned, err_count, report_dir / REPORT_MD) + _abandon_exit(pool, mgr, pool_abandoned, err_count, report_dir) # Same clamp: exit status is a byte either way, so 256 failures would report green. sys.exit(min(err_count, 125)) diff --git a/test/hil/test/test_ci_select.py b/test/hil/test/test_ci_select.py index c34bccd1f..f3e000cb4 100644 --- a/test/hil/test/test_ci_select.py +++ b/test/hil/test/test_ci_select.py @@ -984,6 +984,7 @@ class TestTheHarnessTestsAreNotTheHarness(unittest.TestCase): 'test/hil/test/test_ci_select.py', 'test/hil/test/test_hil_bounded.py', 'test/hil/test/test_hil_health.py', + 'test/hil/test/test_hil_report.py', 'test/hil/test/test_hil_util.py', ], 'test/hil/test/ gained or lost a file; it is carved out of rule 2, so confirm ' 'the rig still does not read anything in there before updating this list') diff --git a/test/hil/test/test_hil_bounded.py b/test/hil/test/test_hil_bounded.py index c6d454f0e..715d520db 100644 --- a/test/hil/test/test_hil_bounded.py +++ b/test/hil/test/test_hil_bounded.py @@ -39,6 +39,7 @@ serial_stub.SerialTimeoutException = type('SerialTimeoutException', (Exception,) sys.modules.setdefault('serial', serial_stub) import hil_flash import hil_test +from helper import hil_report def write_script(path: Path, body: str) -> None: @@ -936,11 +937,17 @@ class RunWhileContract(unittest.TestCase): def boom(): raise AssertionError('x') + # a duration no other process would plausibly pick: `pgrep -f` searches the WHOLE + # machine, so a bare `sleep 20` matched an unrelated background job -- another + # agent session's retry loop, in the case that exposed this -- and failed a test + # about our own child. Observed failing 3/3 in isolation while that loop ran. + sentinel = '20.0451' with self.assertRaises(AssertionError): - self.hil_util.run_alongside(['sleep', '20'], boom, 1) + self.hil_util.run_alongside(['sleep', sentinel], boom, 1) # nothing of ours is left running: the reap ran on the error path too import subprocess - out = subprocess.run(['pgrep', '-f', '^sleep 20'], capture_output=True, text=True) + out = subprocess.run(['pgrep', '-f', f'^sleep {sentinel}'], + capture_output=True, text=True) seen['strays'] = [p for p in out.stdout.split() if p] self.assertEqual(seen['strays'], [], 'work() raising leaked the child') @@ -1146,10 +1153,16 @@ class AbandonExitSurvivesAFailedFork(unittest.TestCase): def test_none_pool_and_manager_still_write_the_banner(self): # a subprocess, because _abandon_exit ends in os._exit: in-process it would take # the test runner with it, before any assertion could run + import json import subprocess with TemporaryDirectory() as td: - report = Path(td) / 'hil_report.md' - report.write_text('| board | test |\n|---|---|\n', encoding='utf-8') + rd = Path(td) + # it takes the report DIRECTORY now and re-renders both artifacts from the + # sidecar, so seed the sidecar -- the markdown is output, not input + (rd / 'hil_report.json').write_text(json.dumps( + {'rows': [{'board': 'boardA', 'cells': {'cdc_msc': 'pass'}, + 'duration': '1s'}], + 'banner': '', 'scope': '', 'caveat': ''})) src = ( 'import sys, types\n' f'sys.path.insert(0, {str(Path(TEST_DIR).parents[0])!r})\n' @@ -1160,12 +1173,14 @@ class AbandonExitSurvivesAFailedFork(unittest.TestCase): 'sys.modules.setdefault("serial", st)\n' 'import hil_test\n' f'hil_test._abandon_exit(None, None, True, 1, __import__("pathlib")' - f'.Path({str(report)!r}))\n') + f'.Path({str(rd)!r}))\n') r = subprocess.run([sys.executable, '-c', src], capture_output=True, text=True, timeout=120) self.assertEqual(r.returncode, 1, r.stderr) - self.assertTrue(report.read_text().startswith('**HIL run abandoned'), - 'the abandon banner never reached the report') + self.assertTrue((rd / 'hil_report.md').read_text().startswith( + '**HIL run abandoned'), 'the abandon banner never reached the report') + self.assertIn('abandoned', + json.loads((rd / 'hil_report.json').read_text())['caveat']) def test_kill_pool_children_tolerates_a_pool_that_never_existed(self): from helper import hil_health @@ -1647,124 +1662,6 @@ class StagingCoversEveryBoardForm(unittest.TestCase): "last run's re-run spec survived a green run") -class SummaryFoldsReportToBoards(unittest.TestCase): - """hil_summary.py replaces the agent retyping the markdown table. Report rows are named per - VARIANT and a variant need not start with the board name, so the config is what maps them - back -- the previous string-matching design produced a defect in each of four review rounds.""" - - def _sum(self, boards, rows, cfg_boards=None, banner=''): - import json - import subprocess - td = TemporaryDirectory() - self.addCleanup(td.cleanup) - d = Path(td.name) - (d / 'hil_report.json').write_text(json.dumps( - {'rows': [{'board': b, 'cells': c, 'duration': '1s'} for b, c in rows], - 'banner': banner})) - cfg = d / 'cfg.json' - cfg.write_text(json.dumps({'boards': cfg_boards or [{'name': b} for b in boards]})) - args = [a for b in boards for a in ('-b', b)] - r = subprocess.run(['python3', str(Path(TEST_DIR).parents[0] / 'helper' / 'hil_summary.py'), - str(cfg), *args, '--report-dir', str(d)], - capture_output=True, text=True, timeout=60) - self.assertEqual(r.returncode, 0, r.stderr) - return json.loads(r.stdout)['results'] - - def test_variant_rows_fold_onto_their_board(self): - """nanoch32v203 never produces a row named after the board.""" - got = self._sum(['nanoch32v203'], - [('nanoch32v203-fsdev', {'usbtest': 'pass'}), - ('nanoch32v203-usbfs', {'usbtest': 'pass'})], - cfg_boards=[{'name': 'nanoch32v203', - 'variant': [{'name': 'nanoch32v203-fsdev'}, - {'name': 'nanoch32v203-usbfs'}]}]) - self.assertEqual([r['board'] for r in got], ['nanoch32v203']) - self.assertTrue(got[0]['pass']) - self.assertTrue(got[0]['ran']) - - def test_one_failing_variant_fails_the_board(self): - got = self._sum(['nano'], - [('nano-a', {'usbtest': 'pass'}), ('nano-b', {'usbtest': '❌ 29/30'})], - cfg_boards=[{'name': 'nano', 'variant': [{'name': 'nano-a'}, - {'name': 'nano-b'}]}]) - self.assertFalse(got[0]['pass']) - self.assertIn('29/30', got[0]['detail']) - - def test_lock_contention_is_a_field_not_a_prefix(self): - got = self._sum(['alpha'], [('alpha', {'board-locked': 'fail'})]) - self.assertTrue(got[0]['locked']) - self.assertFalse(got[0]['pass']) - - def test_a_board_with_no_row_is_marked_not_run(self): - got = self._sum(['alpha', 'beta'], [('alpha', {'usbtest': 'pass'})]) - self.assertTrue(got[0]['ran']) - self.assertFalse(got[1]['ran']) - self.assertFalse(got[1]['pass']) - - def test_a_metric_cell_counts_by_its_icon(self): - got = self._sum(['a', 'b'], [('a', {'cdc_msc_throughput': '✅ C 1.2 M 3.4'}), - ('b', {'cdc_msc_throughput': '❌ C 0.0 M 0.0'})]) - self.assertTrue(got[0]['pass']) - self.assertFalse(got[1]['pass']) - - def test_skipped_cells_do_not_fail_a_board(self): - got = self._sum(['a'], [('a', {'usbtest': 'skip', 'cdc_msc': 'pass'})]) - self.assertTrue(got[0]['pass']) - - def test_a_plain_metric_cell_is_a_pass(self): - """Mirrors hil_test.py's own tally (cell_kind): failures are ALWAYS marked -- 'fail' - or a ❌ prefix, per TestFail's docstring -- while a passing test may return a plain - metric string that lands in the cell unprefixed. Treating unknown shapes as fail - would publish a green table as a red verdict.""" - got = self._sum(['a'], [('a', {'device_speed': '480.0 MBps'})]) - self.assertTrue(got[0]['pass']) - - def test_a_declared_variant_of_another_board_is_not_stolen(self): - """A declared variant need not start with its own board's name, so it may start with - a DIFFERENT board's name plus '-'. The prefix fallback must not attribute it twice.""" - got = self._sum(['alpha', 'beta'], - [('beta-x', {'usbtest': 'fail'})], - cfg_boards=[{'name': 'alpha', 'variant': [{'name': 'beta-x'}]}, - {'name': 'beta'}]) - self.assertTrue(got[0]['ran']) - self.assertFalse(got[0]['pass']) - self.assertFalse(got[1]['ran'], "beta must not inherit alpha's row") - - -class CaveatSurvivesAccumulate(unittest.TestCase): - """CI reruns with --accumulate: the sidecar keeps every earlier attempt's cells, but the - banner was recomputed per attempt. A first attempt on a degraded rig and a clean rerun - therefore published the degraded attempt's PASSES with no caveat on them -- and the - generated .failed spec reruns only failures, so those cells are never re-earned.""" - - def _rows(self, board, cell): - return [(board, 0, 0, [(board, {cell: 'OK'}, '1s')], 0)] - - def test_an_earlier_attempts_caveat_is_still_on_the_report(self): - td = TemporaryDirectory() - self.addCleanup(td.cleanup) - rd = Path(td.name) - banner = '> **Rig note.** 2 process(es) in D state at start.\n' - - hil_test.accumulate_report(self._rows('boardA', 'cdc_msc'), rd, True, '', banner) - self.assertIn('Rig note', (rd / hil_test.REPORT_MD).read_text()) - - # the rerun: clean rig, so this attempt contributes no banner of its own - md = hil_test.accumulate_report(self._rows('boardB', 'cdc_msc'), rd, False, '', '') - self.assertIn('boardA', md) # the earlier cells are kept ... - self.assertIn('Rig note', md, - 'the caveat the earlier cells were collected under was dropped') - - def test_the_same_caveat_twice_is_not_stacked(self): - td = TemporaryDirectory() - self.addCleanup(td.cleanup) - rd = Path(td.name) - banner = '> **Rig note.** 2 process(es) in D state at start.\n' - hil_test.accumulate_report(self._rows('boardA', 'cdc_msc'), rd, True, '', banner) - md = hil_test.accumulate_report(self._rows('boardB', 'cdc_msc'), rd, False, '', banner) - self.assertEqual(md.count('Rig note'), 1) - - class BlindWorkerReachesTheReport(unittest.TestCase): """A worker that exhausts its bounded-read budget answers SYSFS_UNKNOWN for every attribute, so its "device not found" means "could not tell". That reached the log and @@ -1796,7 +1693,7 @@ class BlindWorkerReachesTheReport(unittest.TestCase): wide = ('boardA', 1, ['device/cdc_msc'], [('boardA', {'cdc_msc': '❌'}, '2s')], 2.0, True) narrow = ('stuck', 1, [], None, 0) # what the timeout path builds hil_test._write_failed_spec(rd / 'x.failed', rd, [wide, narrow]) - md = hil_test.accumulate_report([wide], rd, True, '', hil_test._blind_note([wide])) + md = hil_report.accumulate_report([wide], rd, True, '', hil_test._blind_note([wide])) self.assertIn('boardA', md) self.assertIn('not all verdicts are evidence', md.lower()) diff --git a/test/hil/test/test_hil_health.py b/test/hil/test/test_hil_health.py index 5695cad6d..c7864f568 100644 --- a/test/hil/test/test_hil_health.py +++ b/test/hil/test/test_hil_health.py @@ -467,49 +467,6 @@ class KillPoolChildren(PatchCase): self.assertEqual(hil_health.kill_pool_children(NoPool()), 0) -class WriteTimeoutReport(unittest.TestCase): - def test_prefix_carries_the_preflight_diagnosis(self): - """The timeout aborts before accumulate_report, so without the prefix the artifact - and the PR comment lose the one line saying WHY the pool never finished.""" - with TemporaryDirectory() as td: - d = Path(td) - hil_health.write_timeout_report(d, [{'name': 'b1'}], 4200, 'r.md', - prefix='> **wedged usb_hub_wq worker.**\n') - out = (d / 'r.md').read_text() - self.assertTrue(out.startswith('> **wedged usb_hub_wq worker.**')) - self.assertIn('timed out after 4200s', out) - self.assertIn('- b1', out) - - def test_writes_a_report_where_there_would_be_none(self): - with TemporaryDirectory() as td: - hil_health.write_timeout_report(Path(td), [{'name': 'ra6m5_ek'}], 4200, - 'hil_report.md') - md = (Path(td) / 'hil_report.md').read_text() - self.assertIn('4200s', md) - self.assertIn('ra6m5_ek', md) - - def test_keeps_a_previous_attempts_table(self): - with TemporaryDirectory() as td: - path = Path(td) / 'hil_report.md' - path.write_text('| board | cdc_msc |\n') - hil_health.write_timeout_report(Path(td), [{'name': 'b1'}], 4200, 'hil_report.md') - md = path.read_text() - self.assertIn('abandoned', md) - self.assertIn('| board | cdc_msc |', md) - self.assertLess(md.index('abandoned'), md.index('| board |')) - - def test_custom_banner_is_used(self): - with TemporaryDirectory() as td: - hil_health.write_timeout_report(Path(td), [], 0, 'hil_report.md', - banner='**refused to start.**\n') - self.assertIn('refused to start', (Path(td) / 'hil_report.md').read_text()) - - def test_unwritable_dir_does_not_raise(self): - """The caller may be about to os._exit; losing the report must not also lose the - exit path.""" - hil_health.write_timeout_report(Path('/proc/nonexistent/nope'), [], 0, 'x.md') - - class WorkerSweepsItsOwnChildren(unittest.TestCase): """maxtasksperchild=1 makes a worker exit the moment its task returns, so by the time main()'s finally sweeps, the strays have been reparented to init and are off the pool's diff --git a/test/hil/test/test_hil_report.py b/test/hil/test/test_hil_report.py new file mode 100644 index 000000000..9ab39bde3 --- /dev/null +++ b/test/hil/test/test_hil_report.py @@ -0,0 +1,1162 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +# Unit tests for the report document: the vocabulary, the one cell classifier, rendering, +# the four writers, and the fold to per-board verdicts. Split out of test_hil_bounded.py +# and test_hil_health.py when the report code moved into helper/hil_report.py. +# Run directly: +# python3 test/hil/test/test_hil_report.py +import json +import os +import subprocess +import sys +import unittest +from pathlib import Path +from tempfile import TemporaryDirectory + +TEST_DIR = os.path.dirname(os.path.abspath(__file__)) +HIL_DIR = os.path.dirname(TEST_DIR) +sys.path.insert(0, HIL_DIR) + +from helper import hil_report + + +class OneClassifierForBothArtifacts(unittest.TestCase): + """The markdown tally and the agent's verdict used to classify cells with two separate + copies of one rule -- hil_test's cell_kind against REPORT_CELL, and hil_summary's + cell_state against its own re-typed '❌'/'⚪' literals. Change the icons and the table + and the verdict silently disagree.""" + + def test_bare_states(self): + self.assertEqual(hil_report.cell_state('fail'), 'fail') + self.assertEqual(hil_report.cell_state('skip'), 'skip') + self.assertEqual(hil_report.cell_state('pass'), 'pass') + + def test_icon_prefixed_metrics_carry_their_verdict(self): + self.assertEqual(hil_report.cell_state(f'{hil_report.REPORT_CELL["fail"]} 29/30'), 'fail') + self.assertEqual(hil_report.cell_state(f'{hil_report.REPORT_CELL["skip"]} board wedged'), + 'skip') + + def test_an_unprefixed_metric_is_a_pass(self): + """Load-bearing: a passing test may return a plain metric string. Classifying + unknown shapes as fail would publish a green table as a red verdict.""" + self.assertEqual(hil_report.cell_state('480.0 MBps'), 'pass') + self.assertEqual(hil_report.cell_state('1103 KB/s'), 'pass') + + def test_a_non_string_cell_does_not_raise(self): + """render_matrix's copy guarded with isinstance; hil_summary's did not, because its + caller str()'d first. The merged one keeps the guard -- it is the safer superset.""" + self.assertEqual(hil_report.cell_state(None), 'pass') + + def test_the_icons_come_from_REPORT_CELL(self): + """No second copy of the emoji anywhere in the module.""" + src = (Path(HIL_DIR) / 'helper' / 'hil_report.py').read_text(encoding='utf-8') + # CODE only: prose may quote an icon to explain a rule. The old assertion counted + # the single-quoted spelling `'❌'`, which a second copy written as "❌" would have + # sailed past. + code = '\n'.join(line.split('#', 1)[0] for line in src.splitlines()) + for icon in ('❌', '⚪', '✅'): + self.assertEqual(code.count(icon), 1, + f'{icon} is spelled in code more than once; REPORT_CELL is' + f' meant to be the one source') + + +class ModuleWorksImportedAndAsAScript(unittest.TestCase): + """It is imported as helper.hil_report by hil_test, and run as a script by the operator + (.claude/agents/hil-operator.md). A script run puts helper/ on sys.path, NOT test/hil, + so a plain `from helper import hil_health` breaks the CLI and only the CLI.""" + + def test_importable_as_a_package_module(self): + r = subprocess.run( + [sys.executable, '-c', + f'import sys; sys.path.insert(0, {HIL_DIR!r}); ' + f'from helper import hil_report; print(hil_report.REPORT_JSON)'], + capture_output=True, text=True, timeout=60) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn('hil_report.json', r.stdout) + + def test_runnable_as_a_script(self): + r = subprocess.run( + [sys.executable, str(Path(HIL_DIR) / 'helper' / 'hil_report.py'), '--help'], + capture_output=True, text=True, timeout=60) + self.assertEqual(r.returncode, 0, r.stderr) + + +class RenderReportIsPureFunctionOfTheDocument(unittest.TestCase): + """Four writers used to compose the markdown independently, so a table could carry + something the sidecar did not. One renderer, and the ordering it guarantees, is what + stops that -- pinned here rather than left to the order of three concatenations.""" + + def _doc(self, **kw): + d = {'rows': [{'board': 'boardA', 'cells': {'cdc_msc': 'pass'}, 'duration': '1s'}], + 'banner': '', 'scope': '', 'caveat': ''} + d.update(kw) + return d + + def test_table_comes_from_rows(self): + md = hil_report.render_report(self._doc()) + self.assertIn('boardA', md) + self.assertIn('cdc_msc', md) + + def test_scope_note_appears_above_the_table(self): + md = hil_report.render_report(self._doc(scope='-b boardA')) + self.assertLess(md.index('Scoped run'), md.index('boardA')) + + def test_banner_outranks_the_scope_note(self): + md = hil_report.render_report(self._doc(scope='-b boardA', + banner='> **Rig dirty.** x\n')) + self.assertLess(md.index('Rig dirty'), md.index('Scoped run')) + + def test_caveat_is_outermost(self): + md = hil_report.render_report(self._doc(banner='> **Rig dirty.** x\n', + caveat='**HIL run abandoned.**\n')) + self.assertLess(md.index('abandoned'), md.index('Rig dirty')) + + def test_a_document_with_no_rows_still_renders(self): + md = hil_report.render_report(self._doc(rows=[])) + self.assertIn('No tests were run.', md) + + def test_a_malformed_row_does_not_raise(self): + """mark_report_abandoned renders a sidecar it did not write -- hil_ci.sh reuses a + persistent REMOTE_DIR, so it can be an older version's or a torn one -- and it runs + on the way to os._exit, where a KeyError hangs the runner in multiprocessing's + unbounded join() instead of freeing it.""" + md = hil_report.render_report(self._doc( + rows=[{'board': 'boardA', 'cells': {'cdc_msc': 'pass'}}, {'board': 'half'}, + {}])) + self.assertIn('boardA', md) # the intact row still renders ... + self.assertIn('half', md) # ... and a cell-less one becomes a blank row + + +class ScopeSurvivesInTheJson(unittest.TestCase): + """A scoped run's small table is indistinguishable from a full run that lost boards. + The markdown says so; the JSON did not, so any JSON consumer could not tell.""" + + def _rows(self, board, cell): + return [(board, 0, 0, [(board, {cell: 'OK'}, '1s')], 0)] + + def test_scope_is_recorded_in_the_sidecar(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report(self._rows('boardA', 'cdc_msc'), rd, True, + '-b boardA', '') + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertEqual(doc['scope'], '-b boardA') + + def test_an_unscoped_run_records_an_empty_scope(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report(self._rows('boardA', 'cdc_msc'), rd, True, '', '') + self.assertEqual(json.loads((rd / hil_report.REPORT_JSON).read_text())['scope'], '') + + +class EveryExitPathLeavesBothArtifacts(unittest.TestCase): + """summarize() builds an agent's verdicts from the JSON. A path that writes only + markdown reports the whole fleet as 'no report row' while a human sees the real story.""" + + def test_the_no_boards_exit_writes_json_too(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.write_report(rd, {'rows': [], 'banner': '', 'scope': '', + 'caveat': '**HIL run selected no boards.** why\n'}) + self.assertIn('selected no boards', (rd / hil_report.REPORT_MD).read_text()) + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertEqual(doc['rows'], []) + self.assertIn('selected no boards', doc['caveat']) + + def test_write_report_raises_so_its_callers_can_report_it(self): + """write_report is NOT best-effort. Swallowing the OSError made + write_timeout_report's _p warning and hil_test's fallback-of-the-fallback dead + code -- an unwritable report dir produced no artifact and no message.""" + with self.assertRaises(OSError): + hil_report.write_report(Path('/proc/nonexistent/nope'), + {'rows': [], 'banner': '', 'scope': '', 'caveat': 'x\n'}) + + def test_the_guarded_callers_still_do_not_raise(self): + """They are the ones on the way to os._exit, where a raise hangs the interpreter + in multiprocessing's unbounded join().""" + bad = Path('/proc/nonexistent/nope') + hil_report.mark_report_abandoned(bad, 'the worker pool would not shut down.') + hil_report.mark_report_no_boards(bad, 'filters intersected to nothing') + import io + from contextlib import redirect_stdout + with redirect_stdout(io.StringIO()): + hil_report.write_timeout_report(bad, [{'name': 'b1'}], 3600) + + +class AbandonNoticeLandsInBothArtifacts(unittest.TestCase): + """_abandon_exit did a text prepend on a file it had not written, so the caveat never + reached the JSON and an agent reading the sidecar saw a clean partial report under a + red job.""" + + def test_abandon_sets_the_caveat_not_just_the_markdown(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('boardA', 0, 0, [('boardA', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertIn('abandoned', doc['caveat']) + self.assertEqual(len(doc['rows']), 1, 'the finished board must survive') + md = (rd / hil_report.REPORT_MD).read_text() + self.assertLess(md.index('abandoned'), md.index('boardA')) + + def test_marking_a_missing_report_is_a_no_op(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + hil_report.mark_report_abandoned(Path(td.name), 'x') # must not raise + + def test_a_sidecar_with_a_malformed_row_still_gets_stamped(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_JSON).write_text(json.dumps( + {'rows': [{'board': 'boardA'}], 'banner': '', 'scope': '', 'caveat': ''})) + hil_report.mark_report_abandoned(rd, 'x') + self.assertIn('abandoned', + json.loads((rd / hil_report.REPORT_JSON).read_text())['caveat']) + self.assertIn('abandoned', (rd / hil_report.REPORT_MD).read_text()) + + def test_a_torn_sidecar_is_a_no_op(self): + """This runs while the interpreter is being torn down: a raise here hangs the + process in multiprocessing's unbounded join().""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_JSON).write_text('{ truncated mid-') + hil_report.mark_report_abandoned(rd, 'x') # must not raise + + def test_an_existing_abandon_caveat_is_not_overwritten(self): + """The pool-timeout path names the stuck boards and the rig-health verdict; this + one only knows the pool would not shut down. Whoever got there first wins -- + the guard _abandon_exit used to spell as "'**HIL run ab' not in body[:2000]".""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600, + prefix='> **wedged usb_hub_wq worker.**\n') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertIn('timed out after 3600s', doc['caveat']) + self.assertIn('wedged usb_hub_wq worker', doc['banner']) # rig health, not outcome + self.assertNotIn('would not shut down', doc['caveat']) + + +class CaveatSurvivesAccumulate(unittest.TestCase): + """CI reruns with --accumulate: the sidecar keeps every earlier attempt's cells, but the + banner was recomputed per attempt. A first attempt on a degraded rig and a clean rerun + therefore published the degraded attempt's PASSES with no caveat on them -- and the + generated .failed spec reruns only failures, so those cells are never re-earned.""" + + def _rows(self, board, cell): + return [(board, 0, 0, [(board, {cell: 'OK'}, '1s')], 0)] + + def test_an_earlier_attempts_caveat_is_still_on_the_report(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + banner = '> **Rig note.** 2 process(es) in D state at start.\n' + + hil_report.accumulate_report(self._rows('boardA', 'cdc_msc'), rd, True, '', banner) + self.assertIn('Rig note', (rd / hil_report.REPORT_MD).read_text()) + + # the rerun: clean rig, so this attempt contributes no banner of its own + md = hil_report.accumulate_report(self._rows('boardB', 'cdc_msc'), rd, False, '', '') + self.assertIn('boardA', md) # the earlier cells are kept ... + self.assertIn('Rig note', md, + 'the caveat the earlier cells were collected under was dropped') + + def test_the_same_caveat_twice_is_not_stacked(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + banner = '> **Rig note.** 2 process(es) in D state at start.\n' + hil_report.accumulate_report(self._rows('boardA', 'cdc_msc'), rd, True, '', banner) + md = hil_report.accumulate_report(self._rows('boardB', 'cdc_msc'), rd, False, '', banner) + self.assertEqual(md.count('Rig note'), 1) + + +class MarkdownIsAlwaysARenderingOfTheJson(unittest.TestCase): + """The property this whole change buys: whatever wrote the report, re-rendering the + sidecar reproduces the markdown byte for byte. Four writers, one renderer -- asserted + directly rather than inferred from the writers.""" + + def _check(self, rd): + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertEqual((rd / hil_report.REPORT_MD).read_text(), + hil_report.render_report(doc) + '\n') + + def test_normal_path(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('boardA', 0, 0, [('boardA', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, + '-b boardA', '> **Rig note.** x\n') + self._check(rd) + + def test_after_an_accumulate_rerun(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('boardA', 0, 0, [('boardA', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') + hil_report.accumulate_report( + [('boardB', 0, 0, [('boardB', {'cdc_msc': 'OK'}, '1s')], 0)], rd, False, '', '') + self._check(rd) + + def test_after_abandonment(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('boardA', 0, 0, [('boardA', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + self._check(rd) + + def test_no_boards_exit(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.write_report(rd, {'rows': [], 'banner': '', 'scope': '', + 'caveat': '**HIL run selected no boards.** why\n'}) + self._check(rd) + + def test_the_pool_guard_fallback(self): + """The last writer to join the invariant: it composed its own markdown only because + hil_health could not import the renderer.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('done', 0, 0, [('done', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600, + prefix='> **wedged usb_hub_wq worker.**\n') + self._check(rd) + +class WriteTimeoutReport(unittest.TestCase): + def test_prefix_carries_the_preflight_diagnosis(self): + """The timeout aborts before accumulate_report, so without the prefix the artifact + and the PR comment lose the one line saying WHY the pool never finished.""" + with TemporaryDirectory() as td: + d = Path(td) + hil_report.write_timeout_report(d, [{'name': 'b1'}], 4200, + prefix='> **wedged usb_hub_wq worker.**\n') + out = (d / hil_report.REPORT_MD).read_text() + # the abandon notice leads (run outcome), the rig-health prefix follows in the + # banner -- prefix used to be folded INTO the caveat, which is what made a clean + # --accumulate retry inherit an abandonment that had not happened + self.assertTrue(out.startswith('**HIL run abandoned: worker pool timed out'), out[:80]) + self.assertIn('> **wedged usb_hub_wq worker.**', out) + self.assertIn('timed out after 4200s', out) + self.assertIn('- b1', out) + + def test_writes_a_report_where_there_would_be_none(self): + with TemporaryDirectory() as td: + hil_report.write_timeout_report(Path(td), [{'name': 'ra6m5_ek'}], 4200) + md = (Path(td) / hil_report.REPORT_MD).read_text() + self.assertIn('4200s', md) + self.assertIn('ra6m5_ek', md) + + def test_the_prior_attempts_rows_survive(self): + """Was: the prior MARKDOWN TEXT survives below the banner. It now re-renders from + the merged sidecar, so the guarantee is stated against rows -- one table with the + stuck boards in it, rather than a banner stapled above a duplicate table.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('done', 0, 0, [('done', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600) + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertEqual([r['board'] for r in doc['rows']], ['done', 'stuck']) + md = (rd / hil_report.REPORT_MD).read_text() + self.assertIn('done', md) + self.assertIn('stuck', md) + self.assertIn('abandoned', md) + self.assertLess(md.index('abandoned'), md.index('done')) + self.assertEqual(md.count('| Board'), 1, 'the prior table was duplicated, not merged') + + def test_custom_banner_is_used(self): + with TemporaryDirectory() as td: + hil_report.write_timeout_report(Path(td), [], 0, + banner='**refused to start.**\n') + self.assertIn('refused to start', (Path(td) / hil_report.REPORT_MD).read_text()) + + def test_timeout_report_writes_the_sidecar(self): + """This path used to write markdown only, so summarize() -- which is all an + agent gets -- reported the whole fleet as 'no report row' on exactly the runs + that failed.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.write_timeout_report(rd, [{'name': 'boardA'}], 3600) + self.assertTrue((rd / hil_report.REPORT_JSON).is_file()) + self.assertIn('boardA', (rd / hil_report.REPORT_JSON).read_text()) + + def test_the_sidecar_keeps_a_previous_attempts_rows(self): + """An earlier attempt's finished boards are real results and this attempt has none + of its own, so the rows merge rather than replace.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_JSON).write_text(json.dumps( + {'rows': [{'board': 'done', 'cells': {'cdc_msc': 'pass'}, 'duration': '1s'}], + 'banner': '', 'scope': '', 'caveat': ''})) + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600) + rows = json.loads((rd / hil_report.REPORT_JSON).read_text())['rows'] + self.assertEqual([r['board'] for r in rows], ['done', 'stuck']) + + def test_a_torn_sidecar_does_not_lose_the_stuck_boards(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_JSON).write_text('{ truncated mid-') + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600) + rows = json.loads((rd / hil_report.REPORT_JSON).read_text())['rows'] + self.assertEqual([r['board'] for r in rows], ['stuck']) + + def test_a_roster_entry_without_a_name_does_not_escape(self): + """The broad handler exists to stop a KeyError here stranding the runner, but a + report that silently loses its only board is worse than one saying '?'.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.write_timeout_report(rd, [{}], 3600) + rows = json.loads((rd / hil_report.REPORT_JSON).read_text())['rows'] + self.assertEqual([r['board'] for r in rows], ['?']) + + def test_unwritable_dir_does_not_raise(self): + """The caller may be about to os._exit; losing the report must not also lose the + exit path.""" + hil_report.write_timeout_report(Path('/proc/nonexistent/nope'), [], 0) + + +class SummaryFoldsReportToBoards(unittest.TestCase): + """summarize() replaces the agent retyping the markdown table. Report rows are named per + VARIANT and a variant need not start with the board name, so the config is what maps them + back -- the previous string-matching design produced a defect in each of four review rounds.""" + + def _sum(self, boards, rows, cfg_boards=None, banner=''): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + d = Path(td.name) + (d / 'hil_report.json').write_text(json.dumps( + {'rows': [{'board': b, 'cells': c, 'duration': '1s'} for b, c in rows], + 'banner': banner})) + cfg = d / 'cfg.json' + cfg.write_text(json.dumps({'boards': cfg_boards or [{'name': b} for b in boards]})) + args = [a for b in boards for a in ('-b', b)] + r = subprocess.run(['python3', str(Path(TEST_DIR).parents[0] / 'helper' / 'hil_report.py'), + str(cfg), *args, '--report-dir', str(d)], + capture_output=True, text=True, timeout=60) + self.assertEqual(r.returncode, 0, r.stderr) + return json.loads(r.stdout)['results'] + + def test_variant_rows_fold_onto_their_board(self): + """nanoch32v203 never produces a row named after the board.""" + got = self._sum(['nanoch32v203'], + [('nanoch32v203-fsdev', {'usbtest': 'pass'}), + ('nanoch32v203-usbfs', {'usbtest': 'pass'})], + cfg_boards=[{'name': 'nanoch32v203', + 'variant': [{'name': 'nanoch32v203-fsdev'}, + {'name': 'nanoch32v203-usbfs'}]}]) + self.assertEqual([r['board'] for r in got], ['nanoch32v203']) + self.assertTrue(got[0]['pass']) + self.assertTrue(got[0]['ran']) + + def test_one_failing_variant_fails_the_board(self): + got = self._sum(['nano'], + [('nano-a', {'usbtest': 'pass'}), ('nano-b', {'usbtest': '❌ 29/30'})], + cfg_boards=[{'name': 'nano', 'variant': [{'name': 'nano-a'}, + {'name': 'nano-b'}]}]) + self.assertFalse(got[0]['pass']) + self.assertIn('29/30', got[0]['detail']) + + def test_lock_contention_is_a_field_not_a_prefix(self): + got = self._sum(['alpha'], [('alpha', {'board-locked': 'fail'})]) + self.assertTrue(got[0]['locked']) + self.assertFalse(got[0]['pass']) + + def test_a_board_with_no_row_is_marked_not_run(self): + got = self._sum(['alpha', 'beta'], [('alpha', {'usbtest': 'pass'})]) + self.assertTrue(got[0]['ran']) + self.assertFalse(got[1]['ran']) + self.assertFalse(got[1]['pass']) + + def test_a_metric_cell_counts_by_its_icon(self): + got = self._sum(['a', 'b'], [('a', {'cdc_msc_throughput': '✅ C 1.2 M 3.4'}), + ('b', {'cdc_msc_throughput': '❌ C 0.0 M 0.0'})]) + self.assertTrue(got[0]['pass']) + self.assertFalse(got[1]['pass']) + + def test_skipped_cells_do_not_fail_a_board(self): + got = self._sum(['a'], [('a', {'usbtest': 'skip', 'cdc_msc': 'pass'})]) + self.assertTrue(got[0]['pass']) + + def test_a_plain_metric_cell_is_a_pass(self): + """Mirrors hil_test.py's own tally (cell_kind): failures are ALWAYS marked -- 'fail' + or a ❌ prefix, per TestFail's docstring -- while a passing test may return a plain + metric string that lands in the cell unprefixed. Treating unknown shapes as fail + would publish a green table as a red verdict.""" + got = self._sum(['a'], [('a', {'device_speed': '480.0 MBps'})]) + self.assertTrue(got[0]['pass']) + + def test_a_declared_variant_of_another_board_is_not_stolen(self): + """A declared variant need not start with its own board's name, so it may start with + a DIFFERENT board's name plus '-'. The prefix fallback must not attribute it twice.""" + got = self._sum(['alpha', 'beta'], + [('beta-x', {'usbtest': 'fail'})], + cfg_boards=[{'name': 'alpha', 'variant': [{'name': 'beta-x'}]}, + {'name': 'beta'}]) + self.assertTrue(got[0]['ran']) + self.assertFalse(got[0]['pass']) + self.assertFalse(got[1]['ran'], "beta must not inherit alpha's row") + + + def test_the_caveat_reaches_the_agents_verdict(self): + """The abandon/no-boards notice lives in the document now, and this JSON is all an + agent gets -- dropping it here puts the caveat back where only a human sees it.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + d = Path(td.name) + (d / 'hil_report.json').write_text(json.dumps( + {'rows': [{'board': 'boardA', 'cells': {'cdc_msc': 'pass'}, 'duration': '1s'}], + 'banner': '', 'scope': '', + 'caveat': '**HIL run abandoned: the worker pool would not shut down.**\n'})) + cfg = d / 'cfg.json' + cfg.write_text(json.dumps({'boards': [{'name': 'boardA'}]})) + r = subprocess.run( + ['python3', str(Path(TEST_DIR).parents[0] / 'helper' / 'hil_report.py'), + str(cfg), '-b', 'boardA', '--report-dir', str(d)], + capture_output=True, text=True, timeout=60) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn('abandoned', json.loads(r.stdout)['caveat']) + + def test_an_older_sidecar_without_a_caveat_still_summarises(self): + got = self._sum(['boardA'], [('boardA', {'cdc_msc': 'pass'})]) + self.assertTrue(got[0]['pass']) + + def test_the_old_entry_point_is_gone(self): + """hil_summary.py's CLI moved here. A leftover file would keep working while + drifting from the module that now owns the fold.""" + self.assertFalse((Path(HIL_DIR) / 'helper' / 'hil_summary.py').exists()) + + +class AbandonStampIsNotDestructive(unittest.TestCase): + """mark_report_abandoned runs on the way to os._exit, on a report it did not write. + Every case here was a live regression found by review.""" + + def _doc(self, **kw): + d = {'rows': [{'board': 'OLD', 'cells': {'t': 'pass'}, 'duration': '9s'}], + 'banner': '', 'scope': '', 'caveat': ''} + d.update(kw) + return d + + def test_declining_to_stamp_does_not_republish_the_markdown(self): + """The guard skipped the caveat assignment but write_report ran anyway, so a + no-op call still overwrote THIS run's table with a re-render of an older sidecar.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_JSON).write_text(json.dumps(self._doc( + caveat='**HIL run abandoned: worker pool timed out after 3600s.**\n'))) + (rd / hil_report.REPORT_MD).write_text('THIS RUN table with boardX\n') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + self.assertEqual((rd / hil_report.REPORT_MD).read_text(), + 'THIS RUN table with boardX\n') + + def test_a_banner_borne_abandon_notice_also_wins(self): + """The pool-timeout path puts its notice in `banner` (hil_test.py:2300), not + `caveat`. SKILL.md gives the two notices OPPOSITE rules, so stamping the vaguer + one on top tells the agent to publish rows it is meant to discard.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('a', 0, 0, [('a', {'t': 'OK'}, '1s')], 0)], rd, True, '', '', + caveat='**HIL run abandoned: worker pool timed out after 3600s.** 2 never' + ' reported.\n') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + md = (rd / hil_report.REPORT_MD).read_text() + self.assertTrue(md.startswith('**HIL run abandoned: worker pool timed out'), md[:80]) + self.assertNotIn('would not shut down', md) + + def test_a_missing_sidecar_still_stamps_the_markdown(self): + """Master read the MARKDOWN and prepended unconditionally, so it always stamped. + pr_comment.yml cats only hil_report.md -- giving up here publishes a clean green + table under an abandoned, non-zero job.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_MD).write_text('**✅ 27 passed · ❌ 0 failed**\n') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + md = (rd / hil_report.REPORT_MD).read_text(encoding='utf-8') + self.assertIn('abandoned', md) + self.assertIn('27 passed', md) + + def test_a_torn_sidecar_still_stamps_the_markdown(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_JSON).write_text('{ truncated mid-') + (rd / hil_report.REPORT_MD).write_text('**✅ 27 passed**\n') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + self.assertIn('abandoned', (rd / hil_report.REPORT_MD).read_text(encoding='utf-8')) + + def test_the_wording_matches_the_skill_contract(self): + """SKILL.md pins this banner as 'the table below IS this run's ... Report the + results AND the abandonment'. Calling it 'partial' sends the agent to re-run + boards that already passed.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_JSON).write_text(json.dumps(self._doc())) + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + caveat = json.loads((rd / hil_report.REPORT_JSON).read_text())['caveat'] + self.assertNotIn('partial', caveat) + self.assertIn('unverified', caveat) + + +class WriteReportFailsLoudly(unittest.TestCase): + def test_a_render_failure_does_not_leave_a_committed_json(self): + """It wrote the JSON, then rendered. A render raise left the sidecar saying + 'abandoned' beside a markdown that still read as a clean green table -- breaking + the one invariant this module exists to hold.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_MD).write_text('STALE GREEN TABLE\n') + (rd / hil_report.REPORT_JSON).write_text(json.dumps( + {'rows': ['boardA'], 'banner': '', 'scope': '', 'caveat': ''})) + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + md = (rd / hil_report.REPORT_MD).read_text() + # either both moved or neither did -- never a sidecar the markdown contradicts + self.assertEqual('abandoned' in doc.get('caveat', ''), 'abandoned' in md, + 'the sidecar was committed without its markdown') + + def test_a_non_dict_row_does_not_cost_the_abandon_stamp(self): + """A row that is a bare string raised out of render_report, so the stamp was lost + entirely -- the failure mode this whole function exists to prevent.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_JSON).write_text(json.dumps( + {'rows': ['boardA', {'board': 'good', 'cells': {'t': 'pass'}}], + 'banner': '', 'scope': '', 'caveat': ''})) + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + md = (rd / hil_report.REPORT_MD).read_text() + self.assertIn('abandoned', md) + self.assertIn('good', md) + + def test_an_unwritable_dir_reaches_the_callers_warning(self): + """write_report swallowing OSError made write_timeout_report's broad handler -- + and hil_test's fallback-of-the-fallback -- dead code: no artifact, no message.""" + import io + from contextlib import redirect_stdout + buf = io.StringIO() + with redirect_stdout(buf): + hil_report.write_timeout_report(Path('/proc/nonexistent/nope'), + [{'name': 'b1'}], 3600, prefix='x\n') + self.assertIn('warning', buf.getvalue().lower(), 'the failure was silent') + + +class PoolTimeoutCellIsHonest(unittest.TestCase): + def test_a_stuck_board_with_a_prior_row_still_gets_the_cell(self): + """`not in done` skipped the cell for any board carrying an earlier attempt's row, + so a board that just ate the 60-minute guard summarized as pass:true.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('stm32f4', 0, 0, [('stm32f4', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') + hil_report.write_timeout_report(rd, [{'name': 'stm32f4'}], 3600) + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + verdict = hil_report.summarize({'boards': [{'name': 'stm32f4'}]}, ['stm32f4'], doc) + self.assertFalse(verdict['results'][0]['pass'], + 'a board that hung the pool was published as a pass') + + def test_a_clean_retry_clears_the_cell(self): + """accumulate_report clears stale board-locked and BOUNDARY_CELL cells but not + this one, so a board that passed clean on the retry stayed red forever.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600) + hil_report.accumulate_report( + [('stuck', 0, 0, [('stuck', {'cdc_msc': 'OK'}, '2s')], 0)], rd, False, '', '') + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertNotIn('pool-timeout', doc['rows'][0]['cells']) + verdict = hil_report.summarize({'boards': [{'name': 'stuck'}]}, ['stuck'], doc) + self.assertTrue(verdict['results'][0]['pass']) + + def test_a_torn_sidecar_does_not_destroy_an_intact_markdown(self): + """Re-rendering from an unusable sidecar threw away real results the human copy + still had. Master concatenated below its banner and kept them.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_MD).write_text('| Board | t |\n| a | OK |\n| b | OK |\n') + (rd / hil_report.REPORT_JSON).write_text('{ truncated') + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600) + md = (rd / hil_report.REPORT_MD).read_text() + self.assertIn('| a | OK |', md, "an earlier attempt's real results were destroyed") + self.assertIn('stuck', md) + + +class SummarizeSeesEveryRow(unittest.TestCase): + def test_board_name_rows_reach_a_variant_boards_verdict(self): + """hil_test writes lock-contention and pool-timeout rows keyed by BOARD name, but + variants_of returns only declared variant names -- so for nanoch32v203 and + ch32v307v_r1_1v0 those rows were invisible and a held lock published as a + hardware FAIL that hil-validate.js never retried.""" + cfg = {'boards': [{'name': 'nano', + 'variant': [{'name': 'nano-fsdev'}, {'name': 'nano-usbfs'}]}]} + doc = {'rows': [{'board': 'nano', 'cells': {'board-locked': 'fail'}, + 'duration': None}], 'banner': '', 'scope': '', 'caveat': ''} + r = hil_report.summarize(cfg, ['nano'], doc)['results'][0] + self.assertTrue(r['ran']) + self.assertTrue(r['locked'], 'a held lock was published as a hardware failure') + + def test_a_malformed_row_does_not_kill_the_cli(self): + """summarize is the one reader with no defense, and it is the only one an agent's + verdict depends on.""" + out = hil_report.summarize({'boards': [{'name': 'a'}]}, ['a'], + {'rows': [{'cells': {}}, {'board': 'a', + 'cells': {'t': 'pass'}}]}) + self.assertTrue(out['results'][0]['pass']) + + +class NoBoardsExitKeepsWhatRan(unittest.TestCase): + def test_it_does_not_wipe_an_accumulated_sidecar(self): + """Master wrote only markdown here, so the sidecar survived. Writing rows:[] + unconditionally makes an --accumulate rerun whose filters empty erase every + board that had already passed.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('a', 0, 0, [('a', {'t': 'OK'}, '1s')], 0)], rd, True, '', '') + hil_report.mark_report_no_boards(rd, 'No boards left after the flasher filter', + fresh=False) + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertEqual([r['board'] for r in doc['rows']], ['a']) + self.assertIn('selected no boards', doc['caveat']) + self.assertIn('selected no boards', (rd / hil_report.REPORT_MD).read_text()) + + +class TheMergeBehavioursAreActuallyPinned(unittest.TestCase): + """accumulate_report's docstring cites these three as the reason not to split it, yet + deleting any of them left the whole suite green. Mutation-verified.""" + + def test_a_cleared_boundary_drops_the_previous_attempts_mark(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('b', 0, 0, [('b-v', {hil_report.BOUNDARY_CELL: 'fail'}, '1s')], 0)], + rd, True, '', '') + hil_report.accumulate_report( + [('b', 0, 0, [('b-v', {'cdc_msc': 'OK'}, '2s')], 0)], rd, False, '', '') + cells = json.loads((rd / hil_report.REPORT_JSON).read_text())['rows'][0]['cells'] + self.assertNotIn(hil_report.BOUNDARY_CELL, cells) + + def test_a_board_that_really_ran_drops_its_stale_lock_cell(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('b', 0, 0, [('b', {hil_report.LOCKED_CELL: 'fail'}, None)], 0)], rd, True, '', '') + hil_report.accumulate_report( + [('b', 0, 0, [('b', {'cdc_msc': 'OK'}, '2s')], 0)], rd, False, '', '') + rows = json.loads((rd / hil_report.REPORT_JSON).read_text())['rows'] + self.assertEqual([r['board'] for r in rows], ['b']) + self.assertNotIn(hil_report.LOCKED_CELL, rows[0]['cells']) + + def test_a_filtered_rerun_keeps_the_previous_duration(self): + """A -t-filtered re-run reports duration None; blanking the column loses the only + record of how long the full run took.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('b', 0, 0, [('b', {'cdc_msc': 'OK'}, '119s')], 0)], rd, True, '', '') + hil_report.accumulate_report( + [('b', 0, 0, [('b', {'cdc_msc': 'OK'}, None)], 0)], rd, False, '', '') + self.assertEqual(json.loads( + (rd / hil_report.REPORT_JSON).read_text())['rows'][0]['duration'], '119s') + + +class TheFooterCountsAreNotSwapped(unittest.TestCase): + """SKILL.md tells the operator to paste the footer counts verbatim, and swapping the + failed/skipped tallies left the suite green.""" + + def test_each_kind_is_counted_under_its_own_label(self): + md = hil_report.render_matrix([ + ('b', {'p1': 'pass', 'p2': 'pass', 'f1': 'fail', + 's1': f'{hil_report.REPORT_CELL["skip"]} board wedged'}, '1s')]) + self.assertIn(f'{hil_report.REPORT_CELL["pass"]} 2 passed', md) + self.assertIn(f'{hil_report.REPORT_CELL["fail"]} 1 failed', md) + self.assertIn(f'{hil_report.REPORT_CELL["skip"]} 1 skipped', md) + + +class HilCiUploadsTheAccumulateMergeBase(unittest.TestCase): + """hil_ci.sh rm -rf's REMOTE_DIR at the start of every run, and accumulate_report + merges onto the sidecar in the run's cwd -- so without an upload a remote + `--accumulate` retry silently starts from nothing and its one-row table REPLACES the + full-fleet one. The copy-back at the end has always existed; the upload did not.""" + + def _gate(self, *args): + """Run the real gate block out of hil_ci.sh and return its ACCUMULATE verdict. + + Executed, not grepped: the previous pair of tests searched the source text and + stayed green when `if [ "$ACCUMULATE" = 1 ]` was mutated to `if true`, because the + comment block above it mentions --accumulate five times.""" + sh = (Path(HIL_DIR) / 'hil_ci.sh').read_text(encoding='utf-8') + a = sh.index('ACCUMULATE=$(python3 -') + b = sh.index(') || ACCUMULATE=0', a) + len(') || ACCUMULATE=0') + script = 'ARGS=("$@")\n' + sh[a:b] + '\necho "$ACCUMULATE"' + r = subprocess.run(['bash', '-c', script, '_', *args], + capture_output=True, text=True, timeout=60) + self.assertEqual(r.returncode, 0, r.stderr) + return r.stdout.strip() + + def test_every_spelling_argparse_accepts_is_detected(self): + """hil_test.py declares `-a, --accumulate`, so argparse also takes -av, -va, + --accum and --acc; hil-validate.js tells the operator to retry 'adding -v'.""" + for spelling in ('--accumulate', '-a', '-av', '-va', '--accum', '--acc'): + self.assertEqual(self._gate(spelling), '1', f'{spelling} was not detected') + + def test_a_run_without_it_is_not_treated_as_accumulate(self): + for spelling in ('-b', '-v', '--retry'): + self.assertEqual(self._gate(spelling), '0', f'{spelling} falsely detected') + + def test_the_sidecar_is_uploaded_and_gated(self): + sh = (Path(HIL_DIR) / 'hil_ci.sh').read_text(encoding='utf-8') + up = [ln for ln in sh.splitlines() + if 'scp' in ln and 'hil_report.json' in ln and '$REMOTE:' in ln] + self.assertTrue(up, 'nothing uploads hil_report.json; --accumulate has no merge base') + self.assertIn('if [ "$ACCUMULATE" = 1 ]', sh, 'the upload is not gated') + + def test_a_missing_merge_base_is_loud(self): + """The damage: --accumulate with nothing to merge onto succeeds and quietly + publishes a small table where a full one used to be.""" + warn = [ln for ln in (Path(HIL_DIR) / 'hil_ci.sh').read_text().splitlines() + if 'warning' in ln.lower() and 'accumulate' in ln.lower()] + self.assertTrue(warn, 'no warning when --accumulate has no local sidecar') + + +class RunOutcomeAndRigHealthAreSeparate(unittest.TestCase): + """`banner` describes the CONDITIONS cells were collected under, so it carries across a + retry. `caveat` describes how a RUN ENDED, so it must not: a clean retry that reports + an earlier attempt's abandonment tells the agent a green run failed.""" + + def test_a_clean_retry_drops_the_previous_abandon_notice(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report([('a', 0, 0, [('a', {'t': 'OK'}, '1s')], 0)], + rd, True, '', '') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + hil_report.accumulate_report([('a', 0, 0, [('a', {'t': 'OK'}, '2s')], 0)], + rd, False, '', '') + self.assertEqual(json.loads((rd / hil_report.REPORT_JSON).read_text())['caveat'], '') + + def test_rig_health_still_carries_across_the_retry(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.write_timeout_report(rd, [{'name': 's'}], 3600, + prefix='> **Rig note.** wedged\n') + hil_report.accumulate_report([('a', 0, 0, [('a', {'t': 'OK'}, '1s')], 0)], + rd, False, '', '') + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertIn('Rig note', doc['banner']) + + def test_a_second_attempts_abandon_is_recorded(self): + """_already_abandoned matched a notice carried forward from an EARLIER attempt, so + a genuinely new abandon wrote nothing and the run's own failure vanished.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('a', 0, 0, [('a', {'t': 'OK'}, '1s')], 0)], rd, True, '', + '> **Rig note.** x\n', + caveat='**HIL run abandoned: worker pool timed out after 3600s.**\n') + hil_report.accumulate_report([('a', 0, 0, [('a', {'t': 'OK'}, '2s')], 0)], + rd, False, '', '') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + self.assertIn('would not shut down', + json.loads((rd / hil_report.REPORT_JSON).read_text())['caveat']) + + +class AMalformedSidecarNeverCostsTheReport(unittest.TestCase): + """hil_ci.sh now uploads a sidecar as the merge base, so a non-conforming one is + reachable from outside the harness.""" + + def _write(self, rd, doc): + (rd / hil_report.REPORT_JSON).write_text(json.dumps(doc)) + + def test_a_null_banner_does_not_kill_a_successful_run(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + self._write(rd, {'rows': [{'board': 'a', 'cells': {'t': 'pass'}, 'duration': '1s'}], + 'banner': None, 'caveat': None, 'scope': ''}) + hil_report.accumulate_report([('b', 0, 0, [('b', {'t': 'OK'}, '1s')], 0)], + rd, False, '', '') + self.assertTrue((rd / hil_report.REPORT_MD).is_file()) + + def test_a_null_cells_row_still_gets_its_pool_timeout_cell(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + self._write(rd, {'rows': [{'board': 'boardA', 'cells': None, 'duration': '61s'}], + 'banner': '', 'caveat': '', 'scope': ''}) + hil_report.write_timeout_report(rd, [{'name': 'boardA'}], 3600) + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + v = hil_report.summarize({'boards': [{'name': 'boardA'}]}, ['boardA'], doc) + self.assertFalse(v['results'][0]['pass'], + 'a board that ate the whole pool guard was published as a pass') + + def test_an_awkward_sidecar_still_gets_the_abandon_stamp(self): + """Any raise inside the dict branch was swallowed and the markdown fallback was + unreachable, so the stamp was lost from BOTH artifacts.""" + for bad in ({'rows': [{'board': 'a', 'cells': {'t': 'p'}, 'duration': 120}], + 'banner': '', 'caveat': '', 'scope': ''}, + {'rows': [{'board': 'a', 'cells': {'t': ['x']}, 'duration': '1s'}], + 'banner': None, 'caveat': '', 'scope': ''}): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + self._write(rd, bad) + (rd / hil_report.REPORT_MD).write_text('**✅ 27 passed**\n') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + self.assertIn('abandoned', (rd / hil_report.REPORT_MD).read_text(encoding='utf-8'), + f'no stamp for {bad}') + + def test_a_malformed_roster_entry_still_leaves_an_artifact(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + import io + from contextlib import redirect_stdout + with redirect_stdout(io.StringIO()): + hil_report.write_timeout_report(rd, ['plainstring'], 3600) + self.assertTrue((rd / hil_report.REPORT_MD).is_file(), 'no artifact at all') + + +class PoolTimeoutOutranksAStaleLock(unittest.TestCase): + def test_a_wedge_is_not_published_as_lock_contention(self): + """locked was computed across every cell and short-circuited detail, so a board + that wedged the rig on the retry was reported as LOCKED -- and hil-validate.js + re-runs those, paying another pool guard on a board that just hung it.""" + doc = {'rows': [{'board': 'boardX', + 'cells': {'board-locked': 'fail', 'pool-timeout': 'fail'}, + 'duration': None}], 'banner': '', 'caveat': '', 'scope': ''} + r = hil_report.summarize({'boards': [{'name': 'boardX'}]}, ['boardX'], doc)['results'][0] + self.assertFalse(r['locked'], 'a wedge was published as lock contention') + self.assertFalse(r['pass']) + + +class NoBoardsExitRespectsFreshness(unittest.TestCase): + def test_a_fresh_run_does_not_republish_the_previous_rows(self): + """It is called BEFORE the fresh wipe, so it re-published last run's green table + under this run's red job -- the stale-table failure it exists to prevent.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report([('a', 0, 0, [('a', {'t': 'OK'}, '1s')], 0)], + rd, True, '', '') + hil_report.mark_report_no_boards(rd, 'filters emptied', fresh=True) + self.assertEqual(json.loads((rd / hil_report.REPORT_JSON).read_text())['rows'], []) + + def test_an_accumulate_run_keeps_them(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report([('a', 0, 0, [('a', {'t': 'OK'}, '1s')], 0)], + rd, True, '', '') + hil_report.mark_report_no_boards(rd, 'filters emptied', fresh=False) + self.assertEqual([r['board'] for r in json.loads( + (rd / hil_report.REPORT_JSON).read_text())['rows']], ['a']) + + def test_it_does_not_overwrite_an_abandon_notice(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report([('a', 0, 0, [('a', {'t': 'OK'}, '1s')], 0)], + rd, True, '', '') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + hil_report.mark_report_no_boards(rd, 'filters emptied', fresh=False) + self.assertIn('abandoned', + json.loads((rd / hil_report.REPORT_JSON).read_text())['caveat']) + + +class TheNoBoardsCallSiteIsWired(unittest.TestCase): + """The fresh/accumulate branches of mark_report_no_boards were tested by calling it + DIRECTLY, so both passed while hil_test.py's one real call site never passed the flag + at all -- an --accumulate run whose filter emptied still wiped the accumulated rows. + This drives hil_test.py itself; the no-boards exit needs only a config and a filter + that matches nothing, so it costs no hardware.""" + + def _run(self, *extra): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / 'cfg.json').write_text(json.dumps( + {'boards': [{'name': 'alpha', 'uid': '1', 'flasher': {'name': 'jlink', 'uid': '2'}}]})) + (rd / hil_report.REPORT_JSON).write_text(json.dumps( + {'rows': [{'board': 'earlier', 'cells': {'t': 'pass'}, 'duration': '1s'}], + 'banner': '', 'scope': '', 'caveat': ''})) + r = subprocess.run( + [sys.executable, str(Path(HIL_DIR) / 'hil_test.py'), + '--flasher', 'nonexistent', *extra, str(rd / 'cfg.json')], + capture_output=True, text=True, timeout=120, + env={**os.environ, 'HIL_REPORT_DIR': str(rd)}) + self.assertEqual(r.returncode, 1, r.stdout + r.stderr) + return json.loads((rd / hil_report.REPORT_JSON).read_text()) + + def test_an_accumulate_run_keeps_the_accumulated_rows(self): + doc = self._run('--accumulate') + self.assertEqual([r['board'] for r in doc['rows']], ['earlier'], + "the call site did not pass fresh=not args.accumulate") + self.assertIn('selected no boards', doc['caveat']) + + def test_a_fresh_run_does_not_republish_them(self): + doc = self._run() + self.assertEqual(doc['rows'], []) + self.assertIn('selected no boards', doc['caveat']) + + +class EveryWriterRendersBeforeItCommits(unittest.TestCase): + def test_accumulate_report_does_not_commit_json_then_fail_to_render(self): + """accumulate_report hand-rolled the write instead of calling write_report, so a + render failure left the sidecar ahead of the markdown -- the exact ordering + write_report's docstring forbids.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_JSON).write_text(json.dumps( + {'rows': [{'board': 'boardA', 'cells': {'t': 'pass'}, 'duration': 119.0}], + 'banner': '', 'caveat': '', 'scope': ''})) + hil_report.accumulate_report([('boardB', 0, 0, [('boardB', {'t': 'OK'}, '1s')], 0)], + rd, False, '', '') + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertEqual((rd / hil_report.REPORT_MD).read_text(), + hil_report.render_report(doc) + '\n') + + +class MissingSidecarDoesNotDestroyTheMarkdown(unittest.TestCase): + def test_an_absent_sidecar_keeps_the_prior_table(self): + """`recovered` was only cleared when the sidecar was TORN, not when it was absent + -- reachable from hil_ci.sh's asymmetric copy-back and build.yml's skip marker.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_MD).write_text('| Board | t |\n| a | OK |\n| b | OK |\n') + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600) + self.assertIn('| a | OK |', (rd / hil_report.REPORT_MD).read_text()) + + +class LoadIsTheOnlyTrustBoundary(unittest.TestCase): + """hil_ci.sh uploads a sidecar as the merge base, so these shapes arrive from OUTSIDE + the harness. Every one of these raised past a handler before.""" + + def _seed(self, rd, raw): + (rd / hil_report.REPORT_JSON).write_text(raw if isinstance(raw, str) + else json.dumps(raw)) + + def test_a_non_list_rows_does_not_raise(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + self._seed(rd, {'rows': 1, 'banner': '', 'caveat': '', 'scope': ''}) + hil_report.accumulate_report([('a', 0, 0, [('a', {'t': 'OK'}, '1s')], 0)], + rd, False, '', '') + self.assertTrue((rd / hil_report.REPORT_MD).is_file()) + + def test_an_unhashable_cell_value_does_not_raise(self): + """render_matrix does REPORT_CELL.get(v, v); an unhashable value raised TypeError + on the NORMAL accumulate path.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + self._seed(rd, {'rows': [{'board': 'a', 'cells': {'t': ['x'], 'u': 'pass'}, + 'duration': '1s'}], + 'banner': '', 'caveat': '', 'scope': ''}) + hil_report.accumulate_report([('b', 0, 0, [('b', {'t': 'OK'}, '1s')], 0)], + rd, False, '', '') + cells = {r['board']: r['cells'] + for r in json.loads((rd / hil_report.REPORT_JSON).read_text())['rows']} + self.assertNotIn('t', cells['a'], 'a corrupt cell must drop, not become a pass') + self.assertIn('u', cells['a']) + + def test_summarize_survives_a_malformed_sidecar_via_load(self): + """The CLI is the one reader an agent's verdict depends on.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / 'cfg.json').write_text(json.dumps({'boards': [{'name': 'a'}]})) + self._seed(rd, {'rows': [{'board': 1, 'cells': 'notadict'}, + {'board': 'a', 'cells': {'t': 'pass'}}], + 'banner': '', 'caveat': '', 'scope': ''}) + r = subprocess.run( + [sys.executable, str(Path(HIL_DIR) / 'helper' / 'hil_report.py'), + str(rd / 'cfg.json'), '-b', 'a', '--report-dir', str(rd)], + capture_output=True, text=True, timeout=60) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertTrue(json.loads(r.stdout)['results'][0]['pass']) + + +class NoBoardsGuardOnlyAppliesWhenAccumulating(unittest.TestCase): + def test_a_fresh_run_carries_nothing_from_the_prior_sidecar(self): + """rows were reset on fresh but banner and scope were not, so a leftover or + uploaded sidecar republished a stale rig-health note and a stale scope line under + this run's notice.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report([('old', 0, 0, [('old', {'t': 'OK'}, '1s')], 0)], + rd, True, '3 board(s) — a, b, c', + '> **Rig note.** stale D-state holder\n') + hil_report.mark_report_no_boards(rd, 'filters emptied', fresh=True) + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertEqual(doc['rows'], []) + self.assertEqual(doc['banner'], '', 'a stale rig-health banner was republished') + self.assertEqual(doc['scope'], '', 'a stale scope note was republished') + self.assertNotIn('Rig note', (rd / hil_report.REPORT_MD).read_text()) + + def test_an_accumulate_run_keeps_banner_and_scope(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report([('old', 0, 0, [('old', {'t': 'OK'}, '1s')], 0)], + rd, True, '3 board(s) — a, b, c', + '> **Rig note.** real\n') + hil_report.mark_report_no_boards(rd, 'filters emptied', fresh=False) + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertIn('Rig note', doc['banner']) + self.assertEqual([r['board'] for r in doc['rows']], ['old']) + + def test_a_fresh_run_is_not_blocked_by_a_prior_abandon(self): + """The guard runs BEFORE the fresh wipe, so guarding a fresh run left the previous + attempt's rows AND its abandon notice published as this run's.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report([('old', 0, 0, [('old', {'t': 'OK'}, '1s')], 0)], + rd, True, '', '') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + hil_report.mark_report_no_boards(rd, 'filters emptied', fresh=True) + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertEqual(doc['rows'], []) + self.assertIn('selected no boards', doc['caveat']) + + +if __name__ == '__main__': + unittest.main() -- cgit v1.3.1