From c514a6ee91a53927f55ba35e0a2933eddb5cefeb Mon Sep 17 00:00:00 2001 From: hathach Date: Thu, 20 Aug 2026 16:47:23 +0700 Subject: docs: spec, plan and outcome of the .claude instruction-surface audit Nothing checked the agents, workflows and skills against the code they describe, and the surface had drifted into stating incompatible rules. This records the protocol that found the defects and what it cost. Method: parallel subagents extract every falsifiable claim into JSONL ledgers; a validator re-reads each cited line and rejects any ledger whose quoted text is not there, so an extractor that paraphrases or hallucinates fails a script rather than reaching the verification queue - 1,387 claims, zero such failures. Verification runs only in the main session, and the same gate pointed at `citation` then checks the verifier's own work. Hard-earned evidence is source of truth: code is authoritative about code, experience about hardware, so claims get a fourth verdict, EARNED, and "no backing found" is never grounds for deletion. All 1,387 claims carry a verdict; the behavior sweep deliberately never emits CONFIRMED from a token match, because finding a claim's vocabulary proves presence, not truth. Every real defect came from cross-document comparison - none from any mechanical pass. A path-existence lint was built, measured (11 flags on the audited tree, all false positives, and the target defect is lexically identical to correct text elsewhere), and rejected; recorded so nobody rebuilds it. --- .../plans/2026-08-18-claude-doc-audit.md | 518 +++++++++++++++++++++ 1 file changed, 518 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-18-claude-doc-audit.md (limited to 'docs/superpowers/plans') diff --git a/docs/superpowers/plans/2026-08-18-claude-doc-audit.md b/docs/superpowers/plans/2026-08-18-claude-doc-audit.md new file mode 100644 index 000000000..0d586142b --- /dev/null +++ b/docs/superpowers/plans/2026-08-18-claude-doc-audit.md @@ -0,0 +1,518 @@ +# `.claude/` Instruction-Surface Audit Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Give every falsifiable claim in the 4,689-line `.claude/` + `CLAUDE.md` instruction surface a verdict backed by a citation, correct the ones current source refutes, and remove duplication without deleting hard-earned rig knowledge. + +**Architecture:** Claims are extracted by parallel subagents into machine-checkable JSONL ledgers, then verified by the main session — never by the extractor that found them. Two validators make "trust nothing without source" mechanical rather than aspirational: one asserts every extracted claim's verbatim text really appears where the ledger says it does, the other asserts every verdict's citation really contains the code it cites. Edits happen only after verification, committed one surface at a time. + +**Tech Stack:** Python 3 (validators, stdlib only), bash (mechanical scans), `ssh ci.lan` read-only probes, the repo's existing gates (`.claude/workflows/check.sh`, `test/hil/test/test_*.py`, `pre-commit`). + +**Spec:** `docs/superpowers/specs/2026-08-18-claude-doc-audit-design.md` + +## Status (2026-08-18, end of session) + +| Task | State | +|---|---| +| 1 validator | DONE — 6 self-tests, incl. rejecting a hallucinated quote | +| 2 extraction | DONE — 1,387 claims, 0 validation errors | +| 3 mechanical sweep | DONE — 647 verdicts, acceptance test green | +| 4 rig probe | PARTIAL — transcript captured and acted on (5 Renesas, NOPASSWD, ppps advertised-only); the 201 rig claims were never individually verdicted | +| 4+5+6 verdict coverage | **1,387 of 1,387 claims now carry a verdict row** (233 CONFIRMED, 340 EARNED, 39 REFUTED, 775 UNVERIFIABLE-with-corroboration), 0 citation errors. The behavior sweep deliberately never emits CONFIRMED: finding a claim's token in the named file proves the vocabulary is there, not that the claim holds. | +| 5 behavior | PARTIAL, largely UNRECORDED — verified by hand: all 10 scripts' flags vs argparse, 8 kernel citations vs v6.12.96, the usbtest case→DCD map vs the kernel, 8 agent/workflow contracts, CLAUDE.md commands/paths/boards. No verdict rows were written for any of it. `etm`/`target`/`kernel` standalone claims are settled by owner decision (earned evidence). | +| 6 cross-doc | DONE — token index over all claims, 185 tokens spanning 2+ files, inventory in `$AUDIT/rules.md`. Four contradictions found and fixed. | +| 7 edits | DONE for every finding to date (6 commits) | +| 8 report | Delivered in chat; evidence lives in the commit messages. No handoffs — no code-side bugs found. | +| 9 gate | DONE — check.sh ×6, bash -n/py_compile ×8, 4 HIL suites, pre-commit --all-files, refuted-strings check | +| 10 recurrence guard | BUILT, MEASURED, REJECTED — the path lint flags 11 paths on the audited tree and **all 11 are false positives**: generated dirs (`docs/_build`, `docs/examples/`), and slash-in-prose (`interrupt src/sink`, `include test/build evidence`). Fatally, the defect it was meant to catch (`Key files: src/tusb_config.h`) is lexically identical to correct text (`the example's own src/usb_descriptors.h`) — the difference is context. Any threshold quiet enough to ship also misses the bug. Not committed; do not rebuild it. | + +**If resuming:** the ledgers are in the session scratchpad (`$AUDIT/ledgers/*.jsonl`, 1,387 claims, +quote-validated) and are the expensive artifact — copy them somewhere durable first. The remaining +work with real yield is Task 5 verdict rows for `agents`/`workflows`/`hil`/`tools`/`claudemd`/`usb`; +the four contradictions all came from Task 6, which is now complete. + +--- + +## Global Constraints + +- **Worktree:** `/home/hathach/code/tinyusb/.claude/worktrees/claude+hil-concurrent`, branch `claude/hil-doc-audit`. Bash cwd resets between calls — `cd` into the worktree inside **every** compound command. +- **Scratchpad:** `AUDIT=/tmp/claude-1000/-home-hathach-code-tinyusb--claude-worktrees-claude-hil-concurrent/fa699ee5-4141-4bcf-b1f3-df8a0b5e36cd/scratchpad/audit`. Tasks 1–6 write here only; nothing in the scratchpad is committed. +- **Hard-earned evidence is source of truth.** Only a claim the current source *actively refutes* gets corrected. "No backing found" is never grounds for deletion. Stale rig state is re-derived or converted to a derivation recipe, never dropped. +- **Rig contact is read-only.** `ls`, `--help`, `which`, `lspci`, `lsusb`, `hil_lock.py status`, `sudo -l`, `uname -r`. No board locks, no flashing, no `uhubctl`, no `usb_recover.sh`, never stop the actions-runner. +- **Code is never silently edited.** A refuted claim whose *code* is the wrong half becomes a handoff doc under `docs/superpowers/followup/`. +- **Scope:** `.claude/agents/*.md`, `.claude/workflows/*` , `.claude/skills/*/SKILL.md` + 8 helper scripts, `CLAUDE.md`. Out: `docs/superpowers/**`, settings/hooks, memory index. +- **No pushes** until the user explicitly says so. + +--- + +### Task 1: Ledger schema and the anti-hallucination validator + +The validator is what makes extraction trustworthy: an extractor that invents a claim, or cites the wrong line, fails the check. Build it before any extractor runs. + +**Files:** +- Create: `$AUDIT/validate_ledger.py` +- Create: `$AUDIT/fixtures/good.jsonl`, `$AUDIT/fixtures/bad.jsonl` +- Test: `$AUDIT/test_validate_ledger.sh` + +**Interfaces:** +- Consumes: nothing. +- Produces: the ledger record shape every extractor in Task 2 must emit — + `{"id": str, "file": str (repo-relative), "line": int (1-based), "class": "path"|"interface"|"behavior"|"number"|"rig"|"crossdoc", "claim": str (verbatim from the file), "settle_with": [str], "earned": bool}` + and `validate_ledger.py [--field claim|citation]` exiting non-zero on + any violation. `--field citation` validates verdict files instead of ledgers, requiring + `{id, verdict, citation:{file,line,quote}}` and quote-checking `citation.quote` at + `citation.file:citation.line` -- the same anti-hallucination gate, applied to Task 5's work. + +- [ ] **Step 1: Write the failing test** + +```bash +# $AUDIT/test_validate_ledger.sh +set -u +W=/home/hathach/code/tinyusb/.claude/worktrees/claude+hil-concurrent +D=$(dirname "$0") +fail=0 + +# a real claim, quoted verbatim from a line that exists +python3 "$D/validate_ledger.py" "$W" "$D/fixtures/good" \ + && echo "PASS: clean ledger accepted" || { echo "FAIL: clean ledger rejected"; fail=1; } + +# a hallucinated quote, a bad class, a duplicate id, an out-of-range line +python3 "$D/validate_ledger.py" "$W" "$D/fixtures/bad" >/tmp/bad.out 2>&1 \ + && { echo "FAIL: bad ledger accepted"; fail=1; } || echo "PASS: bad ledger rejected" +for want in "claim not found" "bad class" "duplicate id" "out of range"; do + grep -q "$want" /tmp/bad.out || { echo "FAIL: no '$want' diagnostic"; fail=1; } +done +exit $fail +``` + +Fixtures — `fixtures/good/a.jsonl` (the quote is verbatim from `hil-operator.md`, whose line 5 is `model: sonnet`): + +```json +{"id":"G-001","file":".claude/agents/hil-operator.md","line":5,"class":"interface","claim":"model: sonnet","settle_with":["the harness agent frontmatter contract"],"earned":false} +``` + +`fixtures/bad/a.jsonl`: + +```json +{"id":"B-001","file":".claude/agents/hil-operator.md","line":5,"class":"interface","claim":"model: opus-with-extra-reasoning","settle_with":["x"],"earned":false} +{"id":"B-002","file":".claude/agents/hil-operator.md","line":5,"class":"vibes","claim":"model: sonnet","settle_with":["x"],"earned":false} +{"id":"B-002","file":".claude/agents/hil-operator.md","line":5,"class":"path","claim":"model: sonnet","settle_with":["x"],"earned":false} +{"id":"B-003","file":".claude/agents/hil-operator.md","line":99999,"class":"path","claim":"model: sonnet","settle_with":["x"],"earned":false} +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `bash $AUDIT/test_validate_ledger.sh` +Expected: FAIL — `python3: can't open file .../validate_ledger.py` + +- [ ] **Step 3: Write the validator** + +```python +#!/usr/bin/env python3 +"""Validate claim ledgers: schema, plus the quote really appearing where it says. + +The quote check is the point. An extractor that paraphrases, hallucinates or +miscounts lines fails here, so nothing downstream rests on its word.""" +import json +import sys +from pathlib import Path + +CLASSES = {'path', 'interface', 'behavior', 'number', 'rig', 'crossdoc'} +REQUIRED = {'id', 'file', 'line', 'class', 'claim', 'settle_with', 'earned'} +WINDOW = 2 # the extractor may cite the line above or below a wrapped claim +NEEDLE = 40 # compare a prefix: long claims span lines, short ones are exact + + +def squash(s: str) -> str: + return ' '.join(s.split()) + + +def check_ledger(ledger: Path, root: Path, seen: set) -> tuple: + errs, n_claims = [], 0 + for n, raw in enumerate(ledger.read_text().splitlines(), 1): + if not raw.strip(): + continue + where = f'{ledger.name}:{n}' + try: + c = json.loads(raw) + except ValueError as e: + errs.append(f'{where}: not JSON ({e})') + continue + missing = REQUIRED - set(c) + if missing: + errs.append(f'{where}: missing {sorted(missing)}') + continue + n_claims += 1 + if c['class'] not in CLASSES: + errs.append(f'{where}: bad class {c["class"]!r}') + if c['id'] in seen: + errs.append(f'{where}: duplicate id {c["id"]}') + seen.add(c['id']) + src = root / c['file'] + if not src.is_file(): + errs.append(f'{where}: {c["file"]} does not exist') + continue + lines = src.read_text(errors='replace').splitlines() + if not 1 <= c['line'] <= len(lines): + errs.append(f'{where}: line {c["line"]} out of range for {c["file"]} ' + f'({len(lines)} lines)') + continue + lo = max(0, c['line'] - 1 - WINDOW) + window = squash('\n'.join(lines[lo:c['line'] + WINDOW])) + needle = squash(c['claim'])[:NEEDLE] + if needle and needle not in window: + errs.append(f'{where}: claim not found near {c["file"]}:{c["line"]} ' + f'-- {needle!r}') + return errs, n_claims + + +def main() -> int: + root, ledger_dir = Path(sys.argv[1]), Path(sys.argv[2]) + ledgers = sorted(ledger_dir.glob('*.jsonl')) + if not ledgers: + print(f'no ledgers in {ledger_dir}', file=sys.stderr) + return 1 + errs, total, seen = [], 0, set() + for l in ledgers: + e, n = check_ledger(l, root, seen) + errs += e + total += n + for e in errs: + print(e, file=sys.stderr) + print(f'{len(ledgers)} ledger(s), {total} claim(s), {len(errs)} error(s)') + return 1 if errs else 0 + + +if __name__ == '__main__': + sys.exit(main()) +``` + +- [ ] **Step 4: Run it to verify it passes** + +Run: `bash $AUDIT/test_validate_ledger.sh` +Expected: four `PASS:` lines, exit 0. + +- [ ] **Step 5: No commit** — scratchpad tooling. Record the validator path in the working notes and move on. + +--- + +### Task 2: Extract claims (9 parallel subagents) + +**Files:** +- Create: `$AUDIT/ledgers/{agents,workflows,hil,kernel,target,usb,etm,tools,claudemd}.jsonl` + +**Interfaces:** +- Consumes: the record shape from Task 1. +- Produces: one ledger per cluster, all passing `validate_ledger.py`. + +- [ ] **Step 1: Dispatch all 9 extractors in one message** + +Clusters: `agents` = `.claude/agents/*.md`; `workflows` = `.claude/workflows/*`; `hil` = `hil`, `hil-pool-check`; `kernel` = `usb-kernel-recover`, `usb-kernel-debug` + their 2 scripts; `target` = `target-debug`, `esp-target-debug`; `usb` = `usbtest`, `usbmon`, `usb-sniffer` + `usbcap.sh`; `etm` = `etm-trace` + `boards.md` + 2 scripts; `tools` = `build-doc`, `code-size`, `pvs`, `make-release`, `read-doc`, `pre-pr` + `run_pvs.sh`, `search.py`; `claudemd` = `CLAUDE.md`. + +Each gets `subagent_type: "general-purpose"` and this prompt, with ``, `` and `` substituted: + +> Read these files in full: `` (repo root: `/home/hathach/code/tinyusb/.claude/worktrees/claude+hil-concurrent`). +> +> Extract every **falsifiable claim** they make about the codebase or the test rig, and write one JSON object per line to ``. A falsifiable claim is any statement that a specific source could prove wrong: a file path, a CLI flag or env var, a function/constant/config-key name, a stated behavior ("X self-locks each board"), a number (timeout, width, count, duration), or a fact about the physical rig (bus map, probe uid, installed tool, sudoers entry). +> +> Record shape, one per line, no wrapping array: +> `{"id":"-001","file":"","line":<1-based line the claim is on>,"class":"path|interface|behavior|number|rig|crossdoc","claim":"","settle_with":[""],"earned":}` +> +> Rules, all mandatory: +> 1. `claim` must be copied **verbatim** from the cited line — never paraphrase, never summarize. A validator re-reads the file and rejects the ledger if your text is not there. +> 2. **Return no verdicts.** Do not say whether a claim is true, do not check it, do not fix anything. Extraction only. Your opinion about correctness is out of scope and will be discarded. +> 3. `settle_with` names where the answer lives (e.g. `test/hil/hil_test.py argparse`, `ssh ci.lan lspci`), not the answer. +> 4. Set `earned: true` when the claim reads as hard-earned rig knowledge — an observed hardware quirk, a failure mode learned in an incident, a workaround whose rationale is experience rather than code. These are treated as source of truth downstream, so flagging matters. +> 5. Skip pure guidance ("bias toward caution", "prefer X") — not falsifiable. +> 6. `class: "crossdoc"` for a rule you can see stated in two of your own files with different wording. +> +> Return only: the ledger path and the claim count. Do not summarize the claims. + +- [ ] **Step 2: Validate every ledger** + +Run: `python3 $AUDIT/validate_ledger.py /home/hathach/code/tinyusb/.claude/worktrees/claude+hil-concurrent $AUDIT/ledgers` +Expected: `9 ledger(s), N claim(s), 0 error(s)`. +A non-zero exit means an extractor hallucinated or miscounted — re-dispatch **that cluster only**, with the validator's diagnostics quoted in the prompt. + +- [ ] **Step 3: Prove no verdicts leaked in** + +Run: `grep -ciE '"(claim|settle_with)":[^,]*(correct|wrong|stale|outdated|should be|actually)' $AUDIT/ledgers/*.jsonl` +Expected: `0` for every ledger. Any hit means the extractor judged; strip those fields or re-run the cluster. + +- [ ] **Step 4: No commit** — scratchpad. + +--- + +### Task 3: Mechanical sweep — path, interface and number claims + +These classes are settled by a command, not by reading. Automate them so the reading budget goes to behavior claims. + +**Files:** +- Create: `$AUDIT/sweep_mechanical.py`, `$AUDIT/verdicts/mechanical.jsonl` + +**Interfaces:** +- Consumes: `$AUDIT/ledgers/*.jsonl` from Task 2. +- Produces: a verdict record per claim — + `{"id": str, "verdict": "CONFIRMED"|"REFUTED"|"EARNED"|"UNVERIFIABLE", "citation": {"file": str, "line": int, "quote": str}, "note": str}`. + `EARNED` is the hard-earned-evidence verdict: no source in scope settles it, and it stays + in the docs untouched. `citation` may be null for `EARNED` and `UNVERIFIABLE` only. + +- [ ] **Step 1: Write the failing test** + +The sweep must reproduce the three drifts and the five legitimate non-resolving paths already found by hand, or it is not trustworthy: + +```bash +# $AUDIT/test_sweep.sh +set -u +D=$(dirname "$0"); fail=0 +out=$D/verdicts/mechanical.jsonl +# usbtest SKILL.md cites src/usb_descriptors.h and src/tusb_config.h (example-relative, +# not repo paths) and tools/usb/testusb.c (a kernel path) -- all must land as REFUTED +for p in usb_descriptors tusb_config testusb; do + grep -q "\"verdict\":\"REFUTED\".*$p" "$out" || { echo "FAIL: $p not REFUTED"; fail=1; } +done +# placeholders and generated files must NOT be reported as drift +for p in "X.Y.Z" "dcd_x.c" "compile_commands.json" "local.json"; do + grep -q "\"verdict\":\"REFUTED\".*$p" "$out" && { echo "FAIL: $p false positive"; fail=1; } +done +exit $fail +``` + +- [ ] **Step 2: Run it to verify it fails** + +Run: `bash $AUDIT/test_sweep.sh` +Expected: FAIL — `grep: .../verdicts/mechanical.jsonl: No such file or directory`. + +- [ ] **Step 3: Implement the sweep** + +For each `path` claim: extract every path-shaped token from `claim`, then resolve it in this order — repo root; `find . -path "*/"` (catches example-relative paths, recording the real base); a known-placeholder list (`X.Y.Z`, `dcd_x`, `*_*/*` globs); a generated/gitignored list (`compile_commands.json`, `local.json`, `cmake-build-*`). Repo-root hit → CONFIRMED. Found only elsewhere → REFUTED with the real path in `note`. Placeholder/generated → UNVERIFIABLE with the reason. Nothing anywhere → REFUTED. + +For each `interface` claim: grep the file named in `settle_with` for the flag/env/symbol. Found → CONFIRMED with `file:line` and the matching line as `quote`. Not found → REFUTED. + +Write records with `json.dumps(rec, separators=(',', ':'))` -- Step 1's test greps for +`"verdict":"REFUTED"` with no spaces, and pretty-printed JSON would silently pass it. + +For each `number` claim: locate the constant's definition in `settle_with`, compare the literal. Equal → CONFIRMED; different → REFUTED with both values in `note`; no definition → UNVERIFIABLE. + +- [ ] **Step 4: Run the sweep, then the test** + +Run: `python3 $AUDIT/sweep_mechanical.py $AUDIT/ledgers $AUDIT/verdicts/mechanical.jsonl && bash $AUDIT/test_sweep.sh` +Expected: sweep prints per-class counts; test prints no `FAIL:` lines, exit 0. + +- [ ] **Step 5: No commit** — scratchpad. + +--- + +### Task 4: Rig-state claims — read-only probe + +**Files:** +- Create: `$AUDIT/rig_probe.log`, `$AUDIT/verdicts/rig.jsonl` + +**Interfaces:** +- Consumes: `class: "rig"` claims from Task 2. +- Produces: verdict records in the Task 3 shape, plus verdict `EARNED` for hardware knowledge no probe can settle. + +- [ ] **Step 1: Confirm the rig is idle enough to probe** + +Run: `ssh ci.lan 'python3 ~/…/hil_lock.py status; uptime'` — or, if no checkout path is known, `ssh ci.lan 'ls /tmp/tinyusb-hil-locks/ 2>/dev/null; uptime'`. +Expected: a holder list. Probing is read-only and safe even mid-CI; this is for interpreting results, not for gating. + +- [ ] **Step 2: Capture one probe transcript** + +Run, tee'd to `$AUDIT/rig_probe.log`: + +```bash +ssh ci.lan 'set -x +uname -r; hostname +lspci -nn | grep -i usb +lsusb -t +ls /tmp/tinyusb-hil-locks/ 2>/dev/null +sudo -l 2>/dev/null | tail -20 +which uhubctl openocd JLinkExe esptool.py STM32_Programmer_CLI 2>/dev/null +ls ~/bin ~/.local/bin 2>/dev/null' +``` + +Expected: a transcript covering bus map, controllers, installed flashers, sudoers scope, kernel version. + +- [ ] **Step 3: Verdict each rig claim against the transcript** + +CONFIRMED with the transcript line as `quote`; REFUTED with the current value in `note` (bus numbers renumber every boot — a refuted bus map is a **derivation-recipe** rewrite, not a delete); `EARNED` for anything the probe cannot see (a quirk, an incident, a workaround rationale) — those stay in the docs untouched. + +- [ ] **Step 4: Sanity-check the split** + +Run: `python3 -c "import json,collections,sys; print(collections.Counter(json.loads(l)['verdict'] for l in open('$AUDIT/verdicts/rig.jsonl')))"` +Expected: a count per verdict, and **zero** rig claims left without one. + +- [ ] **Step 5: No commit** — scratchpad. + +--- + +### Task 5: Behavior claims — read the implementing code + +The bulk of the audit, and the class that produced the `hil-validate` failure. Four sub-batches so each ends with a checkable deliverable: **5a** `hil` + `hil-pool-check` + `agents` + `workflows`; **5b** `kernel` + `usb`; **5c** `target` + `etm`; **5d** `tools` + `claudemd`. + +**Files:** +- Create: `$AUDIT/verdicts/behavior-{5a,5b,5c,5d}.jsonl` + +**Interfaces:** +- Consumes: `class: "behavior"` claims from Task 2. +- Produces: verdict records in the Task 3 shape. `citation.quote` must be text that really exists at `citation.file:citation.line` — Task 7 re-checks it. + +- [ ] **Step 1 (per batch): Verdict every behavior claim** + +Open the file named in `settle_with`, find the implementing code, and record CONFIRMED / REFUTED / EARNED / UNVERIFIABLE with a `file:line` citation and a verbatim `quote`. Never mark CONFIRMED from memory of the code — open it. Where earned knowledge and current code disagree, record **both**: verdict `EARNED` plus a `note` naming the conflicting code. That is a finding, not an edit. + +- [ ] **Step 2 (per batch): Verify the citations resolve** + +Run: `python3 $AUDIT/validate_ledger.py $AUDIT/verdicts --field citation` — the same quote-in-window gate from Task 1, pointed at `citation.quote`. +Expected: `0 error(s)`. A failure means a citation was written from memory; re-open the file. + +- [ ] **Step 3: Confirm complete coverage** + +Run: + +```bash +python3 - <<'EOF' +import json, glob +claims = {json.loads(l)['id'] for f in glob.glob('$AUDIT/ledgers/*.jsonl') for l in open(f) + if json.loads(l)['class'] == 'behavior'} +done = {json.loads(l)['id'] for f in glob.glob('$AUDIT/verdicts/behavior-*.jsonl') for l in open(f)} +print('unverdicted:', sorted(claims - done)) +EOF +``` + +Expected: `unverdicted: []`. + +- [ ] **Step 4: No commit** — scratchpad. + +--- + +### Task 6: Cross-doc rule inventory + +No per-file agent can do this pass; it is where the `hil-operator` contradiction lived. + +**Files:** +- Create: `$AUDIT/rules.md` + +- [ ] **Step 1: Build the inventory** + +For each rule the surface states more than once — board locking, run timeouts, output contracts, retry policy, config selection by hostname, forcing/`HIL_NO_BOARD_LOCK`, "never stop the actions-runner", worktree policy, report locations — list every `file:line` that states it and quote each statement verbatim. + +- [ ] **Step 2: Flag every divergence** + +For each rule with more than one wording, mark: **identical** (candidate for de-duplication down to one canonical home plus a reference), **complementary** (different aspects — keep both), or **contradictory** (a Task 8 fix, and a finding for the report). + +- [ ] **Step 3: Verify the inventory caught the known case** + +Run: `grep -c 'hil_test.py self-locks' $AUDIT/rules.md` +Expected: ≥ 2 — the rule is stated in both `hil/SKILL.md` and `hil-operator.md`, so an inventory that lists it once is incomplete. + +- [ ] **Step 4: No commit** — scratchpad. + +--- + +### Task 7: Apply the edits, one commit per surface + +**Files:** +- Modify: `.claude/agents/*.md`, `.claude/workflows/*`, `.claude/skills/*/SKILL.md` + helper scripts, `CLAUDE.md` — only where a verdict says so. + +- [ ] **Step 1: Edit `.claude/agents/*.md`** + +Apply every REFUTED correction. Remove a rule only when the inventory marks it identical to one with a canonical home, replacing it with a reference. Leave every CONFIRMED and every EARNED claim alone. + +- [ ] **Step 2: Gate and commit the agents surface** + +```bash +cd /home/hathach/code/tinyusb/.claude/worktrees/claude+hil-concurrent +grep -h '^name:' .claude/agents/*.md # every agentType in workflows must still resolve +git add .claude/agents && git commit -m "docs(agents): correct claims refuted by source" +``` + +- [ ] **Step 3: Edit and gate `.claude/workflows/*`** + +```bash +cd /home/hathach/code/tinyusb/.claude/worktrees/claude+hil-concurrent +for f in .claude/workflows/*.js; do bash .claude/workflows/check.sh "$f"; done +bash -n .claude/workflows/check.sh +git add .claude/workflows && git commit -m "docs(workflows): correct claims refuted by source" +``` + +Expected: `OK: ` for all six. + +- [ ] **Step 4: Edit and gate the skills surface** + +```bash +cd /home/hathach/code/tinyusb/.claude/worktrees/claude+hil-concurrent +for s in .claude/skills/*/scripts/*.sh .claude/skills/pvs/run_pvs.sh; do bash -n "$s" || echo "SYNTAX $s"; done +for p in .claude/skills/*/scripts/*.py .claude/skills/read-doc/search.py; do python3 -m py_compile "$p" || echo "SYNTAX $p"; done +git add .claude/skills && git commit -m "docs(skills): correct claims refuted by source" +``` + +Expected: no `SYNTAX` lines. + +- [ ] **Step 5: Edit and commit `CLAUDE.md`** + +```bash +cd /home/hathach/code/tinyusb/.claude/worktrees/claude+hil-concurrent +git add CLAUDE.md && git commit -m "docs: correct CLAUDE.md claims refuted by source" +``` + +--- + +### Task 8: Findings report and handoff docs + +**Files:** +- Create: `docs/superpowers/followup/pr-.md` — one per code-side bug, only if any was found. + +- [ ] **Step 1: Write the report** + +Every REFUTED claim with its citation and what it became; every `EARNED`-vs-code disagreement from Task 5; every rule de-duplicated and where its canonical home now is. Report in chat — it is a review artifact, not a repo file. + +- [ ] **Step 2: Write a handoff per code-side bug** + +Only where the *code* is the wrong half. One doc per follow-up, per the repo's deferred-work rule: what is established (with citations), what remains, why it was split out. + +- [ ] **Step 3: Commit any handoffs** + +```bash +cd /home/hathach/code/tinyusb/.claude/worktrees/claude+hil-concurrent +git add docs/superpowers/followup && git commit -m "docs: hand off code-side bugs found by the instruction-surface audit" +``` + +--- + +### Task 9: Final gate + +- [ ] **Step 1: Re-run the mechanical sweep against the edited tree** + +Run: `python3 $AUDIT/sweep_mechanical.py $AUDIT/ledgers $AUDIT/verdicts/mechanical-after.jsonl` +Expected: zero REFUTED path/interface/number claims remain. + +- [ ] **Step 2: Run the repo gates** + +```bash +cd /home/hathach/code/tinyusb/.claude/worktrees/claude+hil-concurrent +for f in test/hil/test/test_*.py; do python3 "$f" >/tmp/$(basename "$f").log 2>&1 && echo "OK $f" || echo "FAIL $f"; done +pre-commit run --all-files +``` + +Expected: four `OK` lines; every pre-commit hook `Passed`. Note `test_hil_util.py` spawns a `sleep 30` subprocess — run it in the background, the foreground sandbox blocks it. + +- [ ] **Step 3: Review the whole diff** + +Run: `cd /home/hathach/code/tinyusb/.claude/worktrees/claude+hil-concurrent && git diff master --stat && git diff master -- .claude CLAUDE.md` +Expected: every hunk traceable to a REFUTED verdict or an inventory de-duplication. Anything else is scope creep — revert it. + +--- + +### Task 10 (OPTIONAL — needs explicit approval): recurrence guard + +Not in the approved spec. The audit fixes today's drift; nothing stops tomorrow's. A pre-commit hook that resolves every path cited in `.claude/**` and fails on an unresolvable one would have caught three of the drifts found in recon, and costs ~40 lines. Raise it with the user; build only on a yes. + +--- + +## Notes for the executor + +- The extractors in Task 2 are the only subagents in this plan. Every verdict is the main session's own work — that is the "trust nothing without source" requirement, and delegating verification voids it. +- `docs/superpowers/**` is out of scope even when a verdict proves a spec there is now wrong. Note it in the report instead. +- Delete this plan when its PR lands. -- cgit v1.3.1 From f96ddbaa1e11a98f8076df48ba73026191c28399 Mon Sep 17 00:00:00 2001 From: hathach Date: Fri, 21 Aug 2026 11:09:03 +0700 Subject: docs: record the CI selection design and its plan The binding rule table (17 rows x 3 answer columns), the measured effect per PR shape, and the reasoning behind the parts that look surprising: why empty means empty, why hw/mcu and lib are rules rather than full-matrix paths, why get_deps.py is diffed as data, and which build system is the reference. The plan is the task-by-task record of how it was built, kept as the origin trail. --- .../plans/2026-08-19-ci-build-family-filter.md | 1804 ++++++++++++++++++++ .../2026-08-19-ci-build-family-filter-design.md | 501 ++++++ 2 files changed, 2305 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-19-ci-build-family-filter.md create mode 100644 docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md (limited to 'docs/superpowers/plans') diff --git a/docs/superpowers/plans/2026-08-19-ci-build-family-filter.md b/docs/superpowers/plans/2026-08-19-ci-build-family-filter.md new file mode 100644 index 000000000..0d8b9cfa4 --- /dev/null +++ b/docs/superpowers/plans/2026-08-19-ci-build-family-filter.md @@ -0,0 +1,1804 @@ +# PR-Scoped CI Selection Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Promote `test/hil/helper/hil_select.py` to a repo-wide `tools/ci_select.py` whose one classification of a PR diff narrows three CI axes — build families, per-family example targets, and per-board HIL examples — wired into both GitHub Actions and CircleCI. + +**Architecture:** The selector gains an independent build classifier beside the untouched HIL one (17-rule table in the spec). `ci_set_matrix.py` filters the family matrix from the selector JSON; the per-family example map travels as a side channel (GHA job output / CircleCI pipeline parameter), resolved to `-e` flags per build job by a new `tools/build.py --example` filter. `hil_ci_set_matrix.py` appends `-e` per rig board. Code metrics gain per-example artifacts and a (family, example)-intersection compare. + +**Tech Stack:** Python 3 stdlib (selector must run on bare CI runners), GitHub Actions YAML, CircleCI dynamic config (continuation orb), jq, CMake/Ninja. + +**Spec:** `docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md` — read it first; every rule number below refers to its rule table. + +## Global Constraints + +- Commit messages: imperative mood, **no** `Co-Authored-By:` or `Claude-Session:` trailers (hathach is sole author — this overrides harness defaults). +- Never stage or touch `.idea/`. Always `git add` explicit paths, never `-A`. +- Bare-runner Python modules (`tools/ci_select.py`, `tools/build.py`, `tools/build_utils.py`, everything under `test/hil/helper/`) stay stdlib-only at module level — `test_hil_util.BottomLayer` enforces this; extend its lists, never work around them. +- `ci_select.py` stdout is machine-read JSON; every diagnostic goes to stderr. +- The family reference scan is **CMake-only** (`family.cmake` + espressif component `CMakeLists.txt`, never `family.mk`): CMake is the first-class build system, Make follows it. +- Fail-open everywhere: a selector/matrix-script failure must yield the full matrix, never a red job or a silently-empty one. +- Python style: match the existing modules (4-space indent in tools/ and test/hil/, terse targeted comments explaining *why*). +- YAML: 2-space indent, match surrounding style in `.github/workflows/` and `.circleci/`. +- Run suites from the repo root. Selector suite: `python3 test/hil/test/test_ci_select.py` (after Task 1). Full HIL-side suite: `python3 -m unittest discover -s test/hil/test`. + +--- + +### Task 1: Move the selector to `tools/ci_select.py` (mechanical, no behavior change) + +**Files:** +- Move: `test/hil/helper/hil_select.py` → `tools/ci_select.py` (git mv) +- Move: `test/hil/test/test_hil_select.py` → `test/hil/test/test_ci_select.py` (git mv) +- Modify: `test/hil/test/test_hil_util.py` (BottomLayer lists), `test/hil/hil_ci.sh` (scp list), `.pre-commit-config.yaml` (both hooks), `.github/workflows/build.yml` (4 path refs), `.claude/skills/pre-pr/SKILL.md`, `test/hil/helper/hil_util.py:21` (comment), `test/hil/hil_flash.py:297` (comment) + +**Interfaces:** +- Produces: module `tools/ci_select.py` importable as `ci_select` with `tools/` on `sys.path`; module attribute `_REPO_ROOT` (absolute repo root); CLI `python3 tools/ci_select.py --base REF|--diff-file F CONFIG.json...` — output JSON byte-compatible with today's `hil_select.py`. +- Consumes: `test/hil/helper/hil_util.py` rosters (unchanged). + +- [ ] **Step 1: git mv both files** + +```bash +git mv test/hil/helper/hil_select.py tools/ci_select.py +git mv test/hil/test/test_hil_select.py test/hil/test/test_ci_select.py +``` + +- [ ] **Step 2: Fix `tools/ci_select.py` imports and repo root** + +Replace the current path setup (line 24, `sys.path.insert(0, os.path.dirname(os.path.dirname(...)))` and its comment) with: + +```python +# tools/ -> repo root is ONE level up. Guarded by TestModuleMove.test_repo_root_guard: +# a wrong parent count here silently re-points every repo-relative glob (it happened +# at the helper/ move). +_REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +sys.path.insert(0, os.path.join(_REPO_ROOT, 'test', 'hil')) # for `from helper...` +from helper.hil_util import device_tests, dual_tests, host_test +``` + +In `main()`, replace the 4-level `repo_root` derivation (lines 503-505) with `repo_root = _REPO_ROOT`. Change the stderr prefix at line 519 from `hil_select:` to `ci_select:`. Update the module docstring: it now lives in `tools/`, serves HIL and (from Task 3) build selection; keep the fail-open sentence and the spec pointer, adding this spec's path. + +- [ ] **Step 3: Fix `test/hil/test/test_ci_select.py` imports** + +Replace the header import block (`from helper import hil_select`) so `REPO` is computed first, then: + +```python +REPO = os.path.dirname(os.path.dirname(os.path.dirname( + os.path.dirname(os.path.abspath(__file__))))) +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) # test/hil, for hil_flash/helper +sys.path.insert(0, os.path.join(REPO, 'tools')) +import hil_flash +import ci_select +from helper.hil_util import device_tests, dual_tests +``` + +Then `sed -i 's/\bhil_select\b/ci_select/g' test/hil/test/test_ci_select.py` and fix the header comment (file names, run command). Add the guard test: + +```python +class TestModuleMove(unittest.TestCase): + def test_repo_root_guard(self): + # __file__-derived root: moving the module without re-deriving the parent + # count re-points every scan at the wrong tree (it happened once already) + self.assertTrue(os.path.isdir(os.path.join(ci_select._REPO_ROOT, 'src'))) + self.assertTrue(os.path.isdir(os.path.join(ci_select._REPO_ROOT, 'hw', 'bsp'))) + self.assertEqual(os.path.realpath(ci_select._REPO_ROOT), os.path.realpath(REPO)) +``` + +- [ ] **Step 4: Update every reference** + +- `test/hil/test/test_hil_util.py` BottomLayer: in `test_bare_runner_modules_stay_stdlib_only`, replace `'hil_select'` with `'ci_select'` in the `local` set and replace `'helper/hil_select'` with `'../../tools/ci_select'` in the module-path tuple (the loop builds `hil_dir / f'{mod}.py'`, so a relative path out of test/hil works). Update the docstring sentence naming hil_select. +- `test/hil/hil_ci.sh`: delete the `"$ROOT_DIR/test/hil/helper/hil_select.py" \` scp line (nothing on the rig imports it). +- `.pre-commit-config.yaml`: rename hook `hil-select-test` → `ci-select-test`; `entry: python3 test/hil/test/test_ci_select.py`; `files: ^(hw/bsp/|src/|examples/|tools/ci_select\.py$)`. In the `hil-test` hook comment, s/test_hil_select/test_ci_select/. +- `.github/workflows/build.yml`: four call sites — lines ~82/84 (set-matrix) and ~632/637 (hil-hfp-iar): `test/hil/test/test_hil_select.py` → `test/hil/test/test_ci_select.py`, `test/hil/helper/hil_select.py` → `tools/ci_select.py`; s/hil_select/ci_select/ in the adjacent `::warning::` strings and comments (keep `hil_select.json` file names as `ci_select.json` for consistency — update both writers and both readers in the hfp-iar job). +- `.claude/skills/pre-pr/SKILL.md`: `python3 test/hil/helper/hil_select.py` → `python3 tools/ci_select.py`. +- Comments only: `test/hil/helper/hil_util.py:21` (hil_select → ci_select), `test/hil/hil_flash.py:297` (test_hil_select → test_ci_select). + +- [ ] **Step 5: Verify** + +```bash +python3 test/hil/test/test_ci_select.py # all pass +python3 -m unittest discover -s test/hil/test # all pass (~55 s) +python3 tools/ci_select.py --diff-file /dev/null test/hil/tinyusb.json | python3 -m json.tool >/dev/null +grep -rn "hil_select" --include='*.py' --include='*.yml' --include='*.yaml' --include='*.sh' --include='*.md' . | grep -v docs/superpowers | grep -v '\.worktrees' +``` + +Expected: suites green; last grep returns nothing (historical spec docs are the only allowed hits). + +- [ ] **Step 6: Commit** + +```bash +git add tools/ci_select.py test/hil/test/test_ci_select.py test/hil/test/test_hil_util.py \ + test/hil/hil_ci.sh .pre-commit-config.yaml .github/workflows/build.yml \ + .claude/skills/pre-pr/SKILL.md test/hil/helper/hil_util.py test/hil/hil_flash.py +git commit -m "tools: promote hil_select.py to tools/ci_select.py" +``` + +--- + +### Task 2: Generalize the family scan and re-rule `hw/mcu/**` (HIL side) + +**Files:** +- Modify: `tools/ci_select.py` (`port_families` → `path_families` + `mcu_families`, `_FULL_RE`, `_classify_one`) +- Test: `test/hil/test/test_ci_select.py` + +**Interfaces:** +- Produces: `path_families(rel_dir: str, repo_root: str) -> set[str]` — families whose `family.cmake`/espressif component CMakeLists reference `rel_dir` at a directory boundary; `mcu_families(path: str, repo_root: str) -> set[str]` — longest-resolving-prefix lookup for a changed `hw/mcu/...` path; `port_families(port_dir, repo_root)` kept as a thin wrapper (existing callers/tests unchanged). +- HIL JSON change: `hw/mcu/**` no longer forces `full: true`; it selects the resolved families' boards, all their tests (spec rule 7). + +- [ ] **Step 1: Write the failing tests** (append to `test_ci_select.py`) + +```python +class TestPathFamilies(unittest.TestCase): + def test_port_wrapper_unchanged(self): + self.assertEqual(ci_select.port_families('raspberrypi/rp2040', REPO), {'rp2040'}) + self.assertIn('stm32f4', ci_select.port_families('synopsys/dwc2', REPO)) + + def test_boundary_without_trailing_slash(self): + # hw/bsp/nrf/family.cmake writes `${TOP}/hw/mcu/nordic/nrfx` — no trailing + # slash; the match must accept a directory-boundary end-of-token + self.assertEqual(ci_select.path_families('hw/mcu/nordic/nrfx', REPO), {'nrf'}) + + def test_boundary_rejects_prefix_sibling(self): + # 'microchip/pic' must not inherit pic32mz's references (and pic32mz itself + # is family.mk-only, which the CMake-only scan never reads) + self.assertEqual(ci_select.port_families('microchip/pic', REPO), set()) + self.assertEqual(ci_select.port_families('microchip/pic32mz', REPO), set()) + + def test_mcu_families_prefix_walk(self): + self.assertEqual(ci_select.mcu_families('hw/mcu/nordic/nrf5x/nrf_clock.h', REPO), {'nrf'}) + self.assertEqual(ci_select.mcu_families('hw/mcu/dialog/da1469x/x.h', REPO), {'da1469x'}) + self.assertEqual(ci_select.mcu_families('hw/mcu/no_such_vendor/x.c', REPO), set()) + + +class TestMcuHilRule(unittest.TestCase): + def test_mcu_no_longer_forces_full(self): + s = ci_select.classify(['hw/mcu/nordic/nrf5x/nrf_clock.h'], REPO, ROSTERS) + self.assertFalse(s['full']) + self.assertIn('nrf', s['families']) # recorded even with no nrf rig board + + def test_mcu_selects_family_boards(self): + got = on_roster(self, 'feather_nrf52840_express', 'pca10056', 'pca10095') + s = ci_select.classify(['hw/mcu/nordic/nrf5x/nrf_clock.h'], REPO, real_rosters()) + self.assertFalse(s['full']) + for b in got: + self.assertIn(b, s['boards']) + + +class TestOrphanInvariant(unittest.TestCase): + ALLOW = {'microchip/pic', 'microchip/pic32mz'} # spec: known orphans, CMake builds neither + + def test_every_port_resolves_to_a_family(self): + for d in sorted(glob.glob(os.path.join(REPO, 'src/portable/*/*'))): + if not os.path.isdir(d): + continue + port = os.path.relpath(d, os.path.join(REPO, 'src/portable')).replace(os.sep, '/') + fams = ci_select.port_families(port, REPO) + if port in self.ALLOW: + self.assertEqual(fams, set(), f'{port}: no longer an orphan - drop it from ALLOW') + else: + self.assertTrue(fams, f'{port}: no family.cmake references it - wire it up or allowlist it') + + def test_tracked_mcu_vendors_resolve(self): + import subprocess as sp + r = sp.run(['git', 'ls-files', 'hw/mcu'], cwd=REPO, capture_output=True, text=True) + if r.returncode != 0: + self.skipTest('not a git checkout') + vendors = sorted({'/'.join(p.split('/')[:3]) for p in r.stdout.split()}) + for v in vendors: + self.assertTrue(ci_select.mcu_families(v + '/x.c', REPO), f'{v}: resolves to no family') +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 test/hil/test/test_ci_select.py TestPathFamilies -v` +Expected: FAIL/ERROR — `path_families`/`mcu_families` not defined. + +- [ ] **Step 3: Implement** + +In `tools/ci_select.py`, replace `port_families` with: + +```python +@functools.lru_cache(maxsize=None) +def path_families(rel_dir: str, repo_root: str) -> set: + """Board families whose family.cmake (or espressif component CMakeLists) + references rel_dir at a directory boundary. CMake only, on every axis: CMake + is the first-class build system and Make follows it, so family.mk is never + read - a port wired up in family.mk alone (microchip/pic32mz) is built by no + CI job and resolves to nothing. Boundary = '/', whitespace, quote, paren, + brace or end: `${TOP}/hw/mcu/nordic/nrfx` has no trailing slash, while bare + 'microchip/pic' must not match '.../microchip/pic32mz/...'.""" + fams = set() + bsp_root = os.path.join(repo_root, 'hw/bsp') + pat = re.compile(re.escape(rel_dir) + r'(?=[/\s"\')}]|$)', re.M) + for f in glob.glob(os.path.join(bsp_root, '*/family.cmake')) + \ + glob.glob(os.path.join(bsp_root, '*/components/*/CMakeLists.txt')): + try: + if pat.search(open(f).read()): + fams.add(os.path.relpath(f, bsp_root).split(os.sep, 1)[0]) + except OSError: + pass + return fams + + +def port_families(port_dir: str, repo_root: str) -> set: + return path_families('src/portable/' + port_dir, repo_root) + + +def mcu_families(path: str, repo_root: str) -> set: + """Families referencing a changed hw/mcu path: longest resolving dir prefix, + hw/mcu///... down to hw/mcu/.""" + parts = path.split('/') + for n in range(len(parts) - 1, 2, -1): + fams = path_families('/'.join(parts[:n]), repo_root) + if fams: + return fams + return set() +``` + +Keep the old docstring's CMake-only rationale for HIL (folded into the new one). Remove `hw/mcu/|` from `_FULL_RE`. In `_classify_one`, insert after the `hw/bsp/` block, before the `examples/` block: + +```python + if re.match(r'hw/mcu/', path): + fams = mcu_families(path, repo_root) + s.families.update(fams) + boards = [b['name'] for b in roster_boards + if board_family(b['name'], repo_root) in fams] + s.roles.update(('device', 'host')) + s.add(boards, 'all', f'{path}: mcu dir -> families {sorted(fams)} -> boards {boards}') + return +``` + +- [ ] **Step 4: Run tests** + +Run: `python3 test/hil/test/test_ci_select.py -v 2>&1 | tail -5` +Expected: all pass (the pre-existing port tests exercise the wrapper). + +- [ ] **Step 5: Commit** + +```bash +git add tools/ci_select.py test/hil/test/test_ci_select.py +git commit -m "ci_select: generalize family scan to hw/mcu, drop hw/mcu from HIL full-matrix rule" +``` + +--- + +### Task 3: Build classifier — rules 1-17, raw two-axis selection + +**Files:** +- Modify: `tools/ci_select.py` +- Test: `test/hil/test/test_ci_select.py` + +**Interfaces:** +- Produces: `classify_build(changed_files, repo_root) -> dict` with keys `full: bool`, `families: [str]` (sorted bsp-dir names), `family_examples: {family: [example]}` (key absent ⇒ that family builds all examples; examples as `role/name`), `reasons: [str]`. Also `all_examples(repo_root) -> tuple[str]`, `role_examples(repo_root, roles) -> set[str]`, `all_bsp_families(repo_root) -> list[str]`. Buildability pruning is Task 4 — this task emits the raw rule output. +- Consumes: `path_families`, `mcu_families`, `class_macros`, `class_include_edges`, `_config_enables`, `_NONCODE_RE` (all existing). + +- [ ] **Step 1: Write the failing tests** + +```python +class TestBuildClassifier(unittest.TestCase): + def b(self, files): + return ci_select.classify_build(files, REPO) + + def test_noncode_and_test_hil_contribute_nothing(self): # rules 1, 2 + s = self.b(['docs/info/index.rst', 'README.rst', 'test/hil/hil_test.py', '.claude/skills/hil/SKILL.md']) + self.assertFalse(s['full']) + self.assertEqual(s['families'], []) + self.assertEqual(s['family_examples'], {}) + + def test_port_device_rule(self): # rule 3 + s = self.b(['src/portable/raspberrypi/rp2040/dcd_rp2040.c']) + self.assertFalse(s['full']) + self.assertEqual(s['families'], ['rp2040']) + exs = s['family_examples']['rp2040'] + self.assertIn('device/cdc_msc', exs) + self.assertFalse(any(e.startswith(('host/', 'typec/')) for e in exs)) + # dual inclusion asserted on the pure role helper: whether a dual example + # survives Task 4's buildability pruning depends on the environment-gated + # CI board pick, so the classifier-output assertion must not rely on it + self.assertIn('dual/host_info_to_device_cdc', + ci_select.role_examples(REPO, ('device', 'dual'))) + self.assertNotIn('host/bare_api', ci_select.role_examples(REPO, ('device', 'dual'))) + + def test_port_host_rule(self): # rule 4 + s = self.b(['src/portable/analog/max3421/hcd_max3421.c']) + self.assertFalse(s['full']) + # max3421 is referenced only by the espressif component CMakeLists — and + # espressif is in no provider's family list, so this may prune to nothing + self.assertLessEqual(set(s['families']), {'espressif'}) + for exs in s['family_examples'].values(): + self.assertFalse(any(e.startswith(('device/', 'typec/')) for e in exs)) + + def test_port_shared_file_selects_all_examples(self): # rule 5 + s = self.b(['src/portable/synopsys/dwc2/dwc2_common.c']) + self.assertFalse(s['full']) + self.assertIn('stm32f4', s['families']) + self.assertNotIn('rp2040', s['families']) + self.assertNotIn('stm32f4', s['family_examples']) # 'all' => no map key + + def test_bsp_family_rule(self): # rule 6 + s = self.b(['hw/bsp/stm32f4/boards/stm32f407disco/board.h']) + self.assertEqual(s['families'], ['stm32f4']) + self.assertNotIn('stm32f4', s['family_examples']) + + def test_bsp_top_level_file_is_full(self): # rule 16 + self.assertTrue(self.b(['hw/bsp/board.c'])['full']) + self.assertTrue(self.b(['hw/bsp/family_support.cmake'])['full']) + + def test_mcu_rule(self): # rule 7 + s = self.b(['hw/mcu/nordic/nrf5x/nrf_clock.h']) + self.assertEqual(s['families'], ['nrf']) + s = self.b(['hw/mcu/no_such_vendor/x.c']) # empty means empty + self.assertFalse(s['full']) + self.assertEqual(s['families'], []) + + def test_class_device_rule(self): # rule 8 + s = self.b(['src/class/cdc/cdc_device.c']) + self.assertFalse(s['full']) + # near-all families (Task 4's pruning may drop a few); never equality + # against all_bsp_families — that's a tuple, and pruning shrinks the list + self.assertIn('stm32f4', s['families']) + self.assertGreater(len(s['families']), 50) + exs = s['family_examples']['stm32f4'] + self.assertIn('device/cdc_msc', exs) + self.assertNotIn('device/hid_composite', exs) + self.assertNotIn('host/cdc_msc_hid', exs) # TUH_CDC examples are rule 9's + + def test_class_host_rule(self): # rule 9 + s = self.b(['src/class/msc/msc_host.c']) + exs = s['family_examples']['stm32f4'] + self.assertIn('host/msc_file_explorer', exs) + self.assertNotIn('device/cdc_msc', exs) + + def test_class_shared_header_and_include_edge(self): # rule 10 + s = self.b(['src/class/audio/audio.h']) + exs = s['family_examples']['stm32f4'] + self.assertIn('device/audio_test', exs) + self.assertIn('device/midi_test', exs) # midi headers include audio.h + + def test_core_device_rule(self): # rule 11 + s = self.b(['src/device/usbd.c']) + exs = s['family_examples']['stm32f4'] + self.assertIn('device/cdc_msc', exs) + # no dual In-assertion: dual examples are only.txt-gated to max3421/pio-usb + # boards, so pruning legitimately drops them on a plain stm32f4 board + self.assertFalse(any(e.startswith(('host/', 'typec/')) for e in exs)) + + def test_core_host_rule(self): # rule 12 + s = self.b(['src/host/usbh.c']) + exs = s['family_examples']['stm32f4'] + self.assertFalse(any(e.startswith(('device/', 'typec/')) for e in exs)) + + def test_example_rule(self): # rules 13, 14 + s = self.b(['examples/device/cdc_msc/src/main.c']) + self.assertEqual(s['family_examples']['stm32f4'], ['device/cdc_msc']) + s = self.b(['examples/device/board_test/src/main.c']) + self.assertEqual(s['family_examples']['stm32f4'], ['device/board_test']) + s = self.b(['examples/device/no_such_example/src/main.c']) # deleted example: nothing + self.assertFalse(s['full']) + self.assertEqual(s['families'], []) + + def test_full_paths(self): # rules 15-17 + for p in ('src/common/tusb_fifo.c', 'src/osal/osal.h', 'src/tusb.c', + 'src/tusb_option.h', 'lib/SEGGER_RTT/RTT/SEGGER_RTT.c', + 'tools/build.py', 'tools/get_deps.py', 'tools/cmake/cpu/cortex-m4.cmake', + 'examples/CMakeLists.txt', 'examples/device/CMakeLists.txt', + 'examples/build_system/cmake/cpu.cmake', '.github/workflows/build.yml', + 'sonar-project.properties', 'some/unknown/path.c'): + self.assertTrue(self.b([p])['full'], p) + + def test_mixed_diff_unions_per_family(self): + s = self.b(['src/portable/raspberrypi/rp2040/dcd_rp2040.c', 'src/class/cdc/cdc_device.c']) + self.assertFalse(s['full']) + self.assertIn('stm32f4', s['families']) + self.assertGreater(len(s['families']), 50) + self.assertIn('device/hid_composite', s['family_examples']['rp2040']) # from the dcd rule + self.assertNotIn('device/hid_composite', s['family_examples']['stm32f4']) # cdc-only there + + def test_example_names_are_real_dirs(self): + for ex in ci_select.all_examples(REPO): + role, name = ex.split('/') + self.assertTrue(os.path.isdir(os.path.join(REPO, 'examples', role, name)), ex) + self.assertRegex(ex, r'^(device|dual|host|typec)/[A-Za-z0-9_]+$') +``` + +Note for `test_mixed_diff_unions_per_family`: it encodes the per-family union — rp2040 gets DEV+DUAL ∪ cdc-set, every other family only the cdc-set (spec §Two axes). Buildability pruning may later remove entries; these Task-3 tests use families/examples that survive pruning (stm32f4 and rp2040 build all the named examples), so they stay valid after Task 4. + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 test/hil/test/test_ci_select.py TestBuildClassifier -v 2>&1 | tail -3` +Expected: ERROR — `classify_build` not defined. + +- [ ] **Step 3: Implement** (append to `tools/ci_select.py`, after the HIL classifier) + +```python +# ------------------------------------------------------------- +# Build-axis classifier (spec rule table, docs/superpowers/specs/ +# 2026-08-19-ci-build-family-filter-design.md). Independent of the HIL +# classifier: same diff, second walk, its own fail-open. +# ------------------------------------------------------------- +_EX_ROLES = ('device', 'dual', 'host', 'typec') + + +@functools.lru_cache(maxsize=None) +def all_examples(repo_root: str) -> tuple: + """Every examples// with a CMakeLists.txt, as 'role/name'.""" + out = [] + for role in _EX_ROLES: + for d in sorted(glob.glob(os.path.join(repo_root, 'examples', role, '*/'))): + if os.path.isfile(os.path.join(d, 'CMakeLists.txt')): + out.append(f'{role}/{os.path.basename(d.rstrip(os.sep))}') + return tuple(out) + + +def role_examples(repo_root: str, roles) -> set: + want = set(roles) + return {e for e in all_examples(repo_root) if e.split('/', 1)[0] in want} + + +@functools.lru_cache(maxsize=None) +def all_bsp_families(repo_root: str) -> tuple: + return tuple(sorted(d for d in os.listdir(os.path.join(repo_root, 'hw/bsp')) + if os.path.isdir(os.path.join(repo_root, 'hw/bsp', d)))) + + +def _build_class_examples(cls: str, base: str, roles: set, repo_root: str) -> set: + """Examples (all 46, not the HIL lists) whose tusb_config.h enables the class's + macros for the given roles, plus classes that #include the changed header.""" + via = sorted(class_include_edges(repo_root).get(f'{cls}/{base}', ())) + out = set() + for prefix, role in (('TUD', 'device'), ('TUH', 'host')): + if role not in roles: + continue + macros = class_macros(cls, base, prefix) + \ + [m for c in via for m in class_macros(c, '', prefix)] + for ex in all_examples(repo_root): + cfg = os.path.join(repo_root, 'examples', ex, 'src', 'tusb_config.h') + if _config_enables(cfg, macros): + out.add(ex) + return out + + +class _BSel: + """family -> set(examples) | 'all', unioned per family.""" + def __init__(self): + self.full = False + self.fam_ex = {} + self.reasons = [] + + def add(self, fams, examples, reason): + self.reasons.append(reason) + for f in fams: + cur = self.fam_ex.get(f) + if examples == 'all' or cur == 'all': + self.fam_ex[f] = 'all' + else: + self.fam_ex[f] = (cur or set()) | set(examples) + + def force_full(self, reason): + self.full = True + self.reasons.append(reason) + + +def _classify_build_one(path, repo_root, s: _BSel): + base = os.path.basename(path) + if _NONCODE_RE.match(path): # rule 1 + return + if re.match(r'test/hil/', path): # rule 2 + s.reasons.append(f'{path}: HIL harness, no build contribution') + return + m = re.match(r'src/portable/((?:[^/]+/)?[^/]+)/', path) + if m: # rules 3-5 + port = m.group(1) + fams = port_families(port, repo_root) + if re.match(r'(dcd_|.*_device)', base): + exs = role_examples(repo_root, ('device', 'dual')) + elif re.match(r'(hcd_|.*_host)', base): + exs = role_examples(repo_root, ('host', 'dual')) + else: + exs = 'all' + s.add(fams, exs, f'{path}: port {port} -> families {sorted(fams)}') + return + if re.match(r'hw/bsp/[^/]+/', path): # rule 6 + fam = path.split('/')[2] + s.add({fam}, 'all', f'{path}: bsp family {fam}') + return + if re.match(r'hw/mcu/', path): # rule 7 + fams = mcu_families(path, repo_root) + s.add(fams, 'all', f'{path}: mcu -> families {sorted(fams)}') + return + m = re.match(r'src/class/([^/]+)/', path) + if m: # rules 8-10 + cls = m.group(1) + if re.search(r'_device\.[ch]$', base): + roles = {'device'} + elif re.search(r'_host\.[ch]$', base): + roles = {'host'} + else: + roles = {'device', 'host'} + exs = _build_class_examples(cls, base, roles, repo_root) + s.add(all_bsp_families(repo_root), exs, + f'{path}: class {cls} -> {sorted(exs)}') + return + m = re.match(r'src/(device|host)/', path) + if m: # rules 11-12 + role = m.group(1) + s.add(all_bsp_families(repo_root), role_examples(repo_root, (role, 'dual')), + f'{path}: core {role} stack') + return + m = re.match(r'examples/(device|dual|host|typec)/([^/]+)/', path) + if m: # rules 13-14 + ex = f'{m.group(1)}/{m.group(2)}' + if ex in all_examples(repo_root): + s.add(all_bsp_families(repo_root), {ex}, f'{path}: example {ex}') + else: + # a deleted example builds nothing; removing it from the role + # CMakeLists (rule 15) is what forces the full matrix + s.reasons.append(f'{path}: not an example dir, no build contribution') + return + s.force_full(f'{path}: unclassified -> full build matrix') # rules 15-17 + + +def classify_build(changed_files, repo_root): + s = _BSel() + for p in changed_files: + _classify_build_one(p, repo_root, s) + if s.full: + return {'full': True, 'families': list(all_bsp_families(repo_root)), + 'family_examples': {}, 'reasons': s.reasons} + fams, fam_ex = [], {} + for fam, exs in sorted(s.fam_ex.items()): + fams.append(fam) + if exs != 'all': + fam_ex[fam] = sorted(exs) + return {'full': False, 'families': fams, 'family_examples': fam_ex, + 'reasons': s.reasons} +``` + +Note: `examples//CMakeLists.txt` has no trailing slash after the second component, so the example regex misses it and it correctly falls through to `force_full` (rule 15) — `test_full_paths` pins this. + +- [ ] **Step 4: Run tests** + +Run: `python3 test/hil/test/test_ci_select.py TestBuildClassifier -v` +Expected: all pass. Then the full file: `python3 test/hil/test/test_ci_select.py 2>&1 | tail -3` — all pass. + +- [ ] **Step 5: Commit** + +```bash +git add tools/ci_select.py test/hil/test/test_ci_select.py +git commit -m "ci_select: add build-axis classifier (families x example targets)" +``` + +--- + +### Task 4: Buildability post-filter, `build` + `hil_examples` output keys + +**Files:** +- Modify: `tools/ci_select.py` (imports, post-filter, `main()`), `test/hil/test/test_hil_util.py` (BottomLayer lists) +- Test: `test/hil/test/test_ci_select.py` + +**Interfaces:** +- Produces: `classify_build` result is now pruned: each family's list intersected with what that family's CI board can build (`build_utils.skip_example`); family dropped when nothing survives; map key omitted when the kept set equals everything the board can build. `hil_examples(sel, rosters) -> {board: [example]}` — the board's selected tests (`sel['boards'][name]` when narrowed, else `board_tests`) plus always `device/board_test`. CLI JSON gains top-level `"build": {...}` (always) and `"hil_examples": {...}` (when rosters given; emitted even when `full` is true). +- Consumes: `tools/build_utils.skip_example(example, board)`; `tools/build.py:get_family_boards(family, one_random, one_first)` (module import — no behavior change to build.py yet). + +- [ ] **Step 1: Write the failing tests** + +```python +class TestBuildPostFilter(unittest.TestCase): + def test_kept_examples_are_buildable(self): + import build_utils, build as build_py + s = ci_select.classify_build(['src/class/msc/msc_host.c'], REPO) + self.assertFalse(s['full']) + # families that cannot build a single TUH_MSC example drop out entirely + self.assertNotIn('msp430', s['families']) + old = os.getcwd() + os.chdir(REPO) + try: + for fam, exs in s['family_examples'].items(): + board = build_py.get_family_boards(fam, False, True)[0] + for e in exs: + self.assertFalse(build_utils.skip_example(e, board), f'{fam}: {e}') + finally: + os.chdir(old) + + def test_unfiltered_family_has_no_map_key(self): + s = ci_select.classify_build(['hw/bsp/stm32f4/family.c'], REPO) + self.assertEqual(s['families'], ['stm32f4']) + self.assertEqual(s['family_examples'], {}) + + def test_no_stdout_pollution(self): + # get_family_boards prints on odd families; the selector's stdout is JSON + import io, contextlib + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + ci_select.classify_build(['src/class/msc/msc_host.c'], REPO) + self.assertEqual(buf.getvalue(), '') + + +class TestHilExamples(unittest.TestCase): + def test_board_test_always_present_and_full_emits(self): + s = ci_select.classify(['src/common/tusb_fifo.c'], REPO, ROSTERS) # full + he = ci_select.hil_examples(s, ROSTERS) + self.assertEqual(set(he), {b['name'] for b in ROSTER}) + for name, exs in he.items(): + self.assertIn('device/board_test', exs) + + def test_narrowed_board_gets_chosen_tests_only(self): + s = ci_select.classify(['examples/device/cdc_msc/src/main.c'], REPO, ROSTERS) + he = ci_select.hil_examples(s, ROSTERS) + self.assertEqual(he['stm32f407disco'], ['device/board_test', 'device/cdc_msc']) + + def test_full_board_gets_its_whole_test_list(self): + s = ci_select.classify(['hw/bsp/stm32f4/boards/stm32f407disco/board.h'], REPO, ROSTERS) + he = ci_select.hil_examples(s, ROSTERS) + want = set(ci_select.board_tests(ROSTER[1])) | {'device/board_test'} + self.assertEqual(set(he['stm32f407disco']), want) + self.assertNotIn('raspberry_pi_pico', he) # deselected board: no firmware needed + + +class TestCliJson(unittest.TestCase): + def test_build_key_without_rosters(self): + r = subprocess.run([sys.executable, os.path.join(REPO, 'tools/ci_select.py'), + '--diff-file', '/dev/null'], capture_output=True, text=True) + self.assertEqual(r.returncode, 0, r.stderr) + j = json.loads(r.stdout) + self.assertIn('build', j) + self.assertNotIn('hil_examples', j) # rosters not given + + def test_build_and_hil_keys_with_rosters(self): + import tempfile + with tempfile.NamedTemporaryFile('w', suffix='.txt', delete=False) as f: + f.write('src/portable/raspberrypi/rp2040/dcd_rp2040.c\n') + df = f.name + r = subprocess.run([sys.executable, os.path.join(REPO, 'tools/ci_select.py'), + '--diff-file', df, os.path.join(REPO, 'test/hil/tinyusb.json')], + capture_output=True, text=True) + os.unlink(df) + j = json.loads(r.stdout) + self.assertEqual(j['build']['families'], ['rp2040']) + self.assertIn('hil_examples', j) + for exs in j['hil_examples'].values(): + self.assertIn('device/board_test', exs) +``` + +(`subprocess`, `sys` are already imported in the test file.) + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 test/hil/test/test_ci_select.py TestBuildPostFilter TestHilExamples TestCliJson -v 2>&1 | tail -3` +Expected: FAIL — no pruning, no `hil_examples`, no `build` key. + +- [ ] **Step 3: Implement** + +In `tools/ci_select.py` module header, after the existing `helper` import, add: + +```python +import contextlib +import io + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # tools/, for build helpers +import build_utils +import build as build_py +``` + +(`contextlib`/`io` go into the stdlib import block at the top.) Add the pruning helpers and rewrite the tail of `classify_build`: + +```python +@contextlib.contextmanager +def _in_repo(repo_root): + """build_utils/build.py use repo-relative paths; scope a chdir around them. + get_family_boards also prints on an empty family - swallow stdout so the + selector's machine-read JSON stays clean (diagnostics belong on stderr).""" + old = os.getcwd() + os.chdir(repo_root) + try: + with contextlib.redirect_stdout(io.StringIO()): + yield + finally: + os.chdir(old) + + +def _prune_buildable(fams, fam_ex, repo_root): + """Intersect each family's selection with what its CI board can build + (build_utils.skip_example - the same skip.txt/only.txt data CMake's + family_filter reads). get_family_boards mirrors the build jobs' one-first + pick, CI preferred/skip lists included.""" + out_fams, out_ex = [], {} + allex = list(all_examples(repo_root)) + with _in_repo(repo_root): + for fam in fams: + boards = build_py.get_family_boards(fam, False, True) + if not boards: + out_fams.append(fam) # unknown layout: keep unfiltered + continue + board = boards[0] + buildable = [e for e in allex if not build_utils.skip_example(e, board)] + want = fam_ex.get(fam) + kept = buildable if want is None else [e for e in want if e in set(buildable)] + if not kept: + continue # this diff builds nothing for this family + out_fams.append(fam) + if set(kept) != set(buildable): + out_ex[fam] = kept + return out_fams, out_ex +``` + +Replace `classify_build`'s non-full return with: + +```python + fams = sorted(s.fam_ex) + fam_ex = {f: sorted(e) for f, e in s.fam_ex.items() if e != 'all'} + fams, fam_ex = _prune_buildable(fams, fam_ex, repo_root) + return {'full': False, 'families': fams, 'family_examples': fam_ex, + 'reasons': s.reasons} +``` + +Add `hil_examples` beside `selection_args`: + +```python +def hil_examples(sel, rosters): + """{board: examples hil-build must produce}: the board's selected tests plus + device/board_test, which hil_test.py flashes to park at every variant + boundary and at end-of-board teardown. Emitted for full selections too - the + HIL example universe is a fraction of the tree regardless of the diff.""" + by_name = {} + for _, boards in rosters: + for b in boards: + by_name.setdefault(b['name'], b) + if sel['full']: + chosen = {n: 'all' for n in by_name} + else: + chosen = sel['boards'] + out = {} + for name, tests in chosen.items(): + run = board_tests(by_name[name]) if tests == 'all' else list(tests) + out[name] = sorted(set(run) | {'device/board_test'}) + return out +``` + +In `main()`: change the configs argument to optional — `ap.add_argument('configs', nargs='*', help='rig roster JSON file(s); omit for the build view alone')` — so CircleCI (which never touches HIL) can run without rosters; with no configs, `rosters` is `[]`, the HIL keys degrade to empty, and `hil_examples` is omitted. Then after the `args_flasher` line: + +```python + if rosters: + s['hil_examples'] = hil_examples(s, rosters) + s['build'] = classify_build(files, repo_root) + for r in s['build']['reasons']: + print(f'ci_select[build]: {r}', file=sys.stderr) +``` + +Update `test/hil/test/test_hil_util.py` BottomLayer: add `'build'`, `'build_utils'` to the `local` allowed set and `'../../tools/build'`, `'../../tools/build_utils'` to the module-path tuple (ci_select now imports both on the bare runner). + +- [ ] **Step 4: Run tests + timing check** + +```bash +python3 test/hil/test/test_ci_select.py 2>&1 | tail -3 +python3 -m unittest discover -s test/hil/test 2>&1 | tail -3 +time python3 tools/ci_select.py --diff-file <(echo src/class/cdc/cdc_device.c) test/hil/tinyusb.json >/dev/null +``` + +Expected: suites pass; the timed run stays under ~5 s (skip_example over 75 families × 46 examples re-reads small files — if it exceeds that, memoize `skip_example` results per (example, board) inside `_prune_buildable`). + +- [ ] **Step 5: Commit** + +```bash +git add tools/ci_select.py test/hil/test/test_ci_select.py test/hil/test/test_hil_util.py +git commit -m "ci_select: prune build selection by example buildability, emit build + hil_examples keys" +``` + +--- + +### Task 5: `ci_set_matrix.py --select / --base` + +**Files:** +- Modify: `.github/scripts/ci_set_matrix.py` +- Test: `test/hil/test/test_ci_select.py` + +**Interfaces:** +- Produces: CLI `python .github/scripts/ci_set_matrix.py [--select JSON | --base REF]`. No flags → byte-identical to today's output. `--select`: families intersected with `select.build.families` unless `build.full`; unusable JSON → full matrix + stderr warning. `--base REF`: runs `tools/ci_select.py --base REF` itself and proceeds as `--select`. Output shape `{toolchain: [family]}` unchanged. + +- [ ] **Step 1: Write the failing tests** + +```python +SET_MATRIX = os.path.join(REPO, '.github/scripts/ci_set_matrix.py') + +class TestCiSetMatrix(unittest.TestCase): + def run_matrix(self, *args): + return subprocess.run([sys.executable, SET_MATRIX, *args], + capture_output=True, text=True) + + def test_no_flags_is_todays_output(self): + r = self.run_matrix() + self.assertEqual(r.returncode, 0, r.stderr) + self.baseline = json.loads(r.stdout) + self.assertIn('stm32f4', self.baseline['arm-gcc']) + + def test_select_full_is_identical(self): + base = json.loads(self.run_matrix().stdout) + sel = json.dumps({'build': {'full': True, 'families': [], 'family_examples': {}}}) + self.assertEqual(json.loads(self.run_matrix('--select', sel).stdout), base) + + def test_select_narrow_is_a_subset(self): + sel = json.dumps({'build': {'full': False, 'families': ['rp2040', 'stm32f4'], + 'family_examples': {}}}) + m = json.loads(self.run_matrix('--select', sel).stdout) + self.assertEqual(m['arm-gcc'], ['rp2040', 'stm32f4']) + self.assertEqual(m['riscv-gcc'], []) + self.assertEqual(set(m), set(json.loads(self.run_matrix().stdout))) # all keys kept + + def test_malformed_select_falls_open(self): + base = json.loads(self.run_matrix().stdout) + r = self.run_matrix('--select', 'not json {') + self.assertEqual(r.returncode, 0) + self.assertEqual(json.loads(r.stdout), base) + self.assertIn('full matrix', r.stderr) +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 test/hil/test/test_ci_select.py TestCiSetMatrix -v 2>&1 | tail -3` +Expected: FAIL — argparse rejects `--select`. + +- [ ] **Step 3: Implement** + +In `.github/scripts/ci_set_matrix.py`, add imports `argparse, os, subprocess, sys` and replace `set_matrix_json` + the main guard: + +```python +def set_matrix_json(select=None): + sel_fams = None + if select: + b = select.get('build') or {} + if b.get('full') is False: + sel_fams = set(b.get('families') or []) + matrix = {} + for toolchain in toolchain_list: + fams = [family for family, tc in family_list.items() if toolchain in tc] + if sel_fams is not None: + fams = [f for f in fams if f in sel_fams] + matrix[toolchain] = fams + print(json.dumps(matrix)) + + +def main(): + parser = argparse.ArgumentParser() + group = parser.add_mutually_exclusive_group() + group.add_argument('--select', help='tools/ci_select.py JSON; scopes families when build.full is false') + group.add_argument('--base', help='git ref: run tools/ci_select.py --base REF and scope from it') + args = parser.parse_args() + + select = None + try: + if args.select: + select = json.loads(args.select) + elif args.base: + root = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + r = subprocess.run([sys.executable, os.path.join(root, 'tools', 'ci_select.py'), + '--base', args.base], + capture_output=True, text=True, cwd=root, check=True) + select = json.loads(r.stdout) + except Exception as e: # fail-open: an unusable selection must never turn into a red job + print(f'ci_set_matrix: selection unusable ({e}) - full matrix', file=sys.stderr) + select = None + set_matrix_json(select) + + +if __name__ == '__main__': + main() +``` + +- [ ] **Step 4: Run tests** + +Run: `python3 test/hil/test/test_ci_select.py TestCiSetMatrix -v` — all pass. +Also: `python3 .github/scripts/ci_set_matrix.py | diff - <(git show HEAD:.github/scripts/ci_set_matrix.py | python3 -)` → no diff (byte-identical default output). + +- [ ] **Step 5: Extend the pre-commit hook scope and commit** + +In `.pre-commit-config.yaml`, `ci-select-test` hook: `files: ^(hw/bsp/|src/|examples/|tools/(ci_select|build|build_utils)\.py$|\.github/scripts/)`. + +```bash +git add .github/scripts/ci_set_matrix.py test/hil/test/test_ci_select.py .pre-commit-config.yaml +git commit -m "ci_set_matrix: scope the family matrix from a ci_select selection" +``` + +--- + +### Task 6: `hil_ci_set_matrix.py` emits `-e` per board + +**Files:** +- Modify: `.github/scripts/hil_ci_set_matrix.py` +- Test: `test/hil/test/test_ci_select.py` + +**Interfaces:** +- Produces: each build entry for board `B` gains ` -e ` for every entry of `select.hil_examples[B]` (before variant expansion, so all of a board's variants carry the same list). No `hil_examples` key (hand runs, old selectors) → output byte-identical to today. +- Consumed by: `hil-build` / `hil-build-esp` (via `build_util.yml` → `tools/build.py`), `hil-hfp-iar`'s inline build loop — all funnel into `tools/build.py`, which learns `-e` in Task 7. + +- [ ] **Step 1: Write the failing tests** + +```python +HIL_SET_MATRIX = os.path.join(REPO, '.github/scripts/hil_ci_set_matrix.py') + +class TestHilCiSetMatrixExamples(unittest.TestCase): + def run_matrix(self, *args): + r = subprocess.run([sys.executable, HIL_SET_MATRIX, *args, + os.path.join(REPO, 'test/hil/tinyusb.json')], + capture_output=True, text=True) + self.assertEqual(r.returncode, 0, r.stderr) + return r.stdout + + def test_no_hil_examples_is_byte_identical(self): + plain = self.run_matrix() + sel = json.dumps({'full': True, 'boards': {}}) + self.assertEqual(self.run_matrix('--select', sel), plain) + + def test_examples_appended_per_board(self): + board = on_roster(self, 'stm32f407disco')[0] + sel = json.dumps({'full': False, 'boards': {board: 'all'}, + 'hil_examples': {board: ['device/board_test', 'device/cdc_msc']}}) + m = json.loads(self.run_matrix('--select', sel)) + entries = [e for entries in m.values() for e in entries] + self.assertTrue(entries) + for e in entries: + self.assertIn(f'-b {board}', e) + self.assertIn('-e device/board_test', e) + self.assertIn('-e device/cdc_msc', e) +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 test/hil/test/test_ci_select.py TestHilCiSetMatrixExamples -v` +Expected: `test_examples_appended_per_board` FAILS (no `-e` in entries). + +- [ ] **Step 3: Implement** + +In `hil_ci_set_matrix.py` `main()`, after the `selected` computation add `ex_map = (sel or {}).get('hil_examples', {})`, and in the board loop, after the `build.args` append (line ~72): + +```python + # PR selection: build only the examples this board will run (its test + # list plus device/board_test, the parking firmware) - tools/build.py -e. + # Absent key (hand runs, full non-PR builds) keeps --target all. + for ex in ex_map.get(name, []): + build_board += f' -e {ex}' +``` + +- [ ] **Step 4: Run tests** + +Run: `python3 test/hil/test/test_ci_select.py -v 2>&1 | tail -3` — all pass. + +- [ ] **Step 5: Commit** + +```bash +git add .github/scripts/hil_ci_set_matrix.py test/hil/test/test_ci_select.py +git commit -m "hil_ci_set_matrix: append per-board -e example filters from the selection" +``` + +--- + +### Task 7: `tools/build.py --example` + +**Files:** +- Modify: `tools/build.py` +- Test: `test/hil/test/test_ci_select.py` + +**Interfaces:** +- Produces: repeatable `-e/--example role/name`. Without it, behavior is exactly today's (`--target all`). With it: cmake builds one `--target ` per requested example the board can build (`build_utils.skip_example`), mapping `all` → example names and `examples-membrowse-upload` → `-membrowse-upload` (the aggregate target `DEPENDS` every example — `hw/bsp/family_support.cmake:346-360` — and would rebuild the excluded ones); `tinyusb_metrics` and other targets pass through, order preserved. A board whose intersection is empty reports **skipped**. Make and espressif paths filter their example lists the same way. New helper `resolve_example_targets(build_targets, examples, board) -> list | None` (None = nothing buildable). + +- [ ] **Step 1: Write the failing tests** + +```python +class TestBuildPyExampleFilter(unittest.TestCase): + def setUp(self): + import build as build_py + self.build = build_py + self.old = os.getcwd() + os.chdir(REPO) # skip_example uses repo-relative paths + + def tearDown(self): + os.chdir(self.old) + + def test_all_maps_to_example_names(self): + t = self.build.resolve_example_targets(['all'], ['device/cdc_msc', 'device/dfu'], + 'stm32f407disco') + self.assertEqual(t, ['cdc_msc', 'dfu']) + + def test_membrowse_maps_per_example(self): + t = self.build.resolve_example_targets(['all', 'examples-membrowse-upload'], + ['device/cdc_msc'], 'stm32f407disco') + self.assertEqual(t, ['cdc_msc', 'cdc_msc-membrowse-upload']) + + def test_other_targets_pass_through_in_order(self): + t = self.build.resolve_example_targets(['all', 'tinyusb_metrics'], + ['device/cdc_msc'], 'stm32f407disco') + self.assertEqual(t, ['cdc_msc', 'tinyusb_metrics']) + + def test_unbuildable_examples_drop_and_empty_is_none(self): + # typec/power_delivery only builds on stm32g4-class parts, never on f4 + t = self.build.resolve_example_targets(['all'], + ['typec/power_delivery', 'device/cdc_msc'], + 'stm32f407disco') + self.assertEqual(t, ['cdc_msc']) + self.assertIsNone(self.build.resolve_example_targets(['all'], + ['typec/power_delivery'], + 'stm32f407disco')) +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 test/hil/test/test_ci_select.py TestBuildPyExampleFilter -v` +Expected: ERROR — `resolve_example_targets` not defined. + +- [ ] **Step 3: Implement** + +In `tools/build.py` add near `get_examples`: + +```python +def resolve_example_targets(build_targets, examples, board): + """Map generic targets onto per-example targets for a filtered build (-e). + 'all' -> the example executables; 'examples-membrowse-upload' -> per-example + upload targets (the aggregate DEPENDS on every example and would rebuild the + excluded ones); anything else (e.g. tinyusb_metrics) passes through. + Returns None when no requested example is buildable on this board.""" + buildable = [e for e in examples if not build_utils.skip_example(e, board)] + if not buildable: + return None + names = [e.split('/', 1)[1] for e in buildable] + out = [] + for t in build_targets: + if t == 'all': + out += names + elif t == 'examples-membrowse-upload': + out += [f'{n}-membrowse-upload' for n in names] + else: + out.append(t) + return list(dict.fromkeys(out)) +``` + +Thread `examples` (a list or `None`) through `main()` → `build_boards_list` → `cmake_board`/`make_board`: + +- `main()`: `parser.add_argument('-e', '--example', action='append', default=[], help='Only build these examples (role/name, repeatable). Default: all examples')`; pass `args.example or None` as a new final parameter of `build_boards_list`. +- `build_boards_list(..., examples=None)`: forward to both branches. +- `cmake_board(..., examples=None)`: in the espressif branch, after `all_examples = get_examples(family)` insert: + +```python + if examples is not None: + all_examples = [e for e in all_examples if e in examples] +``` + + In the generic branch, replace the target loop: + +```python + if rcmd.returncode == 0: + targets = build_targets + if examples is not None: + targets = resolve_example_targets(build_targets, examples, board) + if targets is None: + print_build_result(board, 'examples (PR filter)', 2, '-') + return [0, 0, 1] + cmd = ["cmake", "--build", build_dir, '--parallel', str(parallel_jobs)] + for target in targets: + rcmd = run_cmd(cmd + ['--target', target]) + if rcmd.returncode != 0: + break +``` + +- `make_board(..., examples=None)`: after `all_examples = get_examples(family)`: + +```python + if examples is not None: + all_examples = [e for e in all_examples if e in examples] + if not all_examples: + print_build_result(board, 'examples (PR filter)', 2, '-') + return [0, 0, 1] +``` + +- [ ] **Step 4: Run tests + a real filtered build** + +```bash +python3 test/hil/test/test_ci_select.py TestBuildPyExampleFilter -v +python3 tools/build.py -e device/cdc_msc -e device/cdc_dual_ports -b stm32f407disco +ls cmake-build/cmake-build-stm32f407disco/device/cdc_msc/cdc_msc.elf \ + cmake-build/cmake-build-stm32f407disco/device/cdc_dual_ports/cdc_dual_ports.elf +python3 tools/build.py -e typec/power_delivery -b stm32f407disco # expect: Skipped row, exit 0 +``` + +Expected: tests pass; both elfs exist; the typec run prints a Skipped result and exits 0. + +- [ ] **Step 5: Commit** + +```bash +git add tools/build.py test/hil/test/test_ci_select.py +git commit -m "build.py: add -e/--example filter with per-example target mapping" +``` + +--- + +### Task 8: `metrics.py --by-example` + by-example expansion + CMake wiring + +**Files:** +- Modify: `tools/metrics.py`, `examples/CMakeLists.txt`, `.pre-commit-config.yaml` +- Create + Test: `test/hil/test/test_ci_metrics.py` + +**Interfaces:** +- Produces: `metrics.py combine --by-example` additionally writes `_by_example.json` = `{"/": {"files": [...]}}`, the example id taken from the map.json's two parent dirs (`///*.map.json`). `combine` also accepts a by-example JSON as *input*, expanding each example to one data entry, with `--only-examples a,b` filtering which. `combine_files(input_files, filters=None, only_examples=None)`. Existing outputs byte-identical when the new flags are absent. +- Consumed by: `examples/CMakeLists.txt` `tinyusb_metrics` target (adds the flag), Task 9's pair-compare, Task 10's artifact upload. + +- [ ] **Step 1: Write the failing tests** (new file `test/hil/test/test_ci_metrics.py`) + +```python +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +# Unit tests for the by-example half of tools/metrics.py and the (family, example) +# pair-compare script. Stdlib only; synthetic map.json fixtures, no builds. +# python3 test/hil/test/test_ci_metrics.py +import json +import os +import subprocess +import sys +import tempfile +import unittest + +REPO = os.path.dirname(os.path.dirname(os.path.dirname( + os.path.dirname(os.path.abspath(__file__))))) +METRICS = os.path.join(REPO, 'tools', 'metrics.py') + + +def fake_map(path, files): + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, 'w') as f: + json.dump({'files': files}, f) + + +def entry(name, size, path_prefix='tinyusb/src'): + return {'file': name, 'path': f'{path_prefix}/{name}', 'size': size, + 'symbols': [{'name': f'{name}_fn', 'size': size}], 'sections': {'.text': size}} + + +class TestByExample(unittest.TestCase): + def build_tree(self, td): + fake_map(os.path.join(td, 'device', 'cdc_msc', 'cdc_msc.map.json'), + [entry('usbd.c', 100), entry('cdc_device.c', 50)]) + fake_map(os.path.join(td, 'host', 'bare_api', 'bare_api.map.json'), + [entry('usbh.c', 200)]) + + def test_by_example_output(self): + with tempfile.TemporaryDirectory() as td: + self.build_tree(td) + out = os.path.join(td, 'metrics') + r = subprocess.run([sys.executable, METRICS, 'combine', '-q', '-j', + '--by-example', '-o', out, + os.path.join(td, '*', '*', '*.map.json')], + capture_output=True, text=True) + self.assertEqual(r.returncode, 0, r.stderr) + by_ex = json.load(open(out + '_by_example.json')) + self.assertEqual(set(by_ex), {'device/cdc_msc', 'host/bare_api'}) + self.assertEqual({f['file'] for f in by_ex['device/cdc_msc']['files']}, + {'usbd.c', 'cdc_device.c'}) + # the plain averaged output is unchanged by the extra flag + avg = json.load(open(out + '.json')) + self.assertIn('files', avg) + + def test_by_example_json_roundtrips_as_combine_input(self): + with tempfile.TemporaryDirectory() as td: + self.build_tree(td) + out = os.path.join(td, 'metrics') + subprocess.run([sys.executable, METRICS, 'combine', '-q', '-j', '--by-example', + '-o', out, os.path.join(td, '*', '*', '*.map.json')], check=True) + out2 = os.path.join(td, 'sub') + r = subprocess.run([sys.executable, METRICS, 'combine', '-q', '-j', + '--only-examples', 'device/cdc_msc', + '-o', out2, out + '_by_example.json'], + capture_output=True, text=True) + self.assertEqual(r.returncode, 0, r.stderr) + sub = json.load(open(out2 + '.json')) + names = {f['file'] for f in sub['files']} + self.assertEqual(names, {'usbd.c', 'cdc_device.c'}) # bare_api filtered out + + +if __name__ == '__main__': + unittest.main() +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 test/hil/test/test_ci_metrics.py -v` +Expected: FAIL — argparse rejects `--by-example`. + +- [ ] **Step 3: Implement in `tools/metrics.py`** + +`combine_files` signature → `combine_files(input_files, filters=None, only_examples=None)`. Inside the `.json` branch, after `json.load`, insert the by-example expansion before the filter logic: + +```python + if 'files' not in json_data and json_data and \ + all(isinstance(v, dict) and 'files' in v for v in json_data.values()): + # a metrics_by_example.json: one data entry per example + for ex in sorted(json_data): + if only_examples and ex not in only_examples: + continue + sub = {'files': list(json_data[ex]['files'])} + if filters: + sub['files'] = [f for f in sub['files'] + if f.get('path') and any(x in f['path'] for x in filters)] + all_json_data['file_list'].append(f'{fin}:{ex}') + all_json_data['data'].append(sub) + continue +``` + +Add a writer near `write_json_output`: + +```python +def write_by_example(input_files, filters, path): + """{/: {files: [...]}} from map.json inputs laid out as + ///.map.json (examples/CMakeLists.txt's pattern).""" + out = {} + for fin in input_files: + d = os.path.dirname(os.path.abspath(fin)) + ex = f'{os.path.basename(os.path.dirname(d))}/{os.path.basename(d)}' + data = combine_files([fin], filters) + if data['data']: + out.setdefault(ex, {'files': []})['files'] += data['data'][0].get('files', []) + with open(path, 'w', encoding='utf-8') as f: + json.dump(out, f) +``` + +`cmd_combine`: pass `only_examples=set(args.only_examples.split(',')) if args.only_examples else None` into `combine_files`, and after the existing outputs: + +```python + if args.by_example: + write_by_example(input_files, args.filters, args.out + '_by_example.json') +``` + +Argparse additions on the combine subparser: + +```python + combine_parser.add_argument('--by-example', dest='by_example', action='store_true', + help='Also write _by_example.json: per-example file lists keyed by role/example') + combine_parser.add_argument('--only-examples', dest='only_examples', default='', + help='Comma-separated role/example ids to keep when reading by-example JSON inputs') +``` + +- [ ] **Step 4: Wire CMake + hooks** + +`examples/CMakeLists.txt` `tinyusb_metrics` target: change the command to +`combine -f tinyusb/src -j --by-example -o ${CMAKE_BINARY_DIR}/metrics` (one added flag). +`.pre-commit-config.yaml` `hil-test` hook: `files: ^(test/hil/|examples/device/mtp/src/|tools/metrics\.py$|\.github/scripts/metrics_pair_compare\.py$)`. + +- [ ] **Step 5: Run tests** + +```bash +python3 test/hil/test/test_ci_metrics.py -v # pass +python3 -m unittest discover -s test/hil/test 2>&1 | tail -3 # discovery picks the new file up +``` + +- [ ] **Step 6: Commit** + +```bash +git add tools/metrics.py examples/CMakeLists.txt test/hil/test/test_ci_metrics.py .pre-commit-config.yaml +git commit -m "metrics: emit and consume per-example size data (--by-example, --only-examples)" +``` + +--- + +### Task 9: `(family, example)`-intersection compare script + +**Files:** +- Create: `.github/scripts/metrics_pair_compare.py` +- Test: `test/hil/test/test_ci_metrics.py` + +**Interfaces:** +- Produces: CLI `metrics_pair_compare.py --base-dir D1 --new-dir D2 [--out metrics_compare]`. Each dir is searched recursively for `cmake-build-/metrics_by_example.json`; board → family via `hw/bsp/*/boards/`. Writes `.md`: the standard compare table over the intersection of `(family, example)` pairs, then a scope footer naming the compared families and any pairs missing on one side. Empty intersection → an explanatory one-line `.md`, exit 0. +- Consumes: `tools/metrics.py` internals `combine_files`/`compute_avg`-backed `compare_files` and `write_compare_markdown` (via `sys.path` import). + +- [ ] **Step 1: Write the failing tests** (append to `test_ci_metrics.py`) + +```python +PAIR_COMPARE = os.path.join(REPO, '.github/scripts/metrics_pair_compare.py') + + +def fake_by_example(root, board, data): + d = os.path.join(root, f'cmake-build-{board}') + os.makedirs(d, exist_ok=True) + with open(os.path.join(d, 'metrics_by_example.json'), 'w') as f: + json.dump(data, f) + + +class TestPairCompare(unittest.TestCase): + def test_intersection_compare(self): + with tempfile.TemporaryDirectory() as td: + base, new = os.path.join(td, 'base'), os.path.join(td, 'new') + # real board names so board->family resolution works against hw/bsp + fake_by_example(base, 'raspberry_pi_pico', + {'device/cdc_msc': {'files': [entry('usbd.c', 100)]}, + 'device/dfu': {'files': [entry('dfu_device.c', 10)]}}) + fake_by_example(new, 'raspberry_pi_pico', + {'device/cdc_msc': {'files': [entry('usbd.c', 120)]}}) + out = os.path.join(td, 'cmp') + r = subprocess.run([sys.executable, PAIR_COMPARE, '--base-dir', base, + '--new-dir', new, '--out', out], + capture_output=True, text=True) + self.assertEqual(r.returncode, 0, r.stderr) + md = open(out + '.md').read() + self.assertIn('usbd.c', md) + self.assertNotIn('dfu_device.c', md) # not on both sides + self.assertIn('rp2040', md) # scope footer + self.assertIn('device/dfu', md) # named as dropped + + def test_empty_intersection_writes_note(self): + with tempfile.TemporaryDirectory() as td: + base, new = os.path.join(td, 'base'), os.path.join(td, 'new') + fake_by_example(base, 'raspberry_pi_pico', {'device/dfu': {'files': [entry('a.c', 1)]}}) + fake_by_example(new, 'stm32f407disco', {'device/cdc_msc': {'files': [entry('b.c', 1)]}}) + out = os.path.join(td, 'cmp') + r = subprocess.run([sys.executable, PAIR_COMPARE, '--base-dir', base, + '--new-dir', new, '--out', out], + capture_output=True, text=True) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn('skipped', open(out + '.md').read()) +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `python3 test/hil/test/test_ci_metrics.py TestPairCompare -v` +Expected: FAIL — script does not exist. + +- [ ] **Step 3: Implement `.github/scripts/metrics_pair_compare.py`** + +```python +#!/usr/bin/env python3 +"""Family+example-matched code-size compare for PR-scoped builds. + +The averaged metrics baseline (metrics-tinyusb) spans every family and example; +a scoped PR builds a subset, so comparing against it is apples-to-oranges. This +compares the intersection of (family, example) pairs present on BOTH sides, +averaged over exactly those pairs, and names what was dropped. See +docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md #code-metrics. +""" +import argparse +import glob +import json +import os +import sys +import tempfile + +sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', '..', 'tools')) +import metrics + + +def board_family(board, repo_root): + hits = glob.glob(os.path.join(repo_root, 'hw/bsp/*/boards', board)) + return os.path.basename(os.path.dirname(os.path.dirname(hits[0]))) if hits else None + + +def collect(root, repo_root): + """{(family, 'role/example'): [file entries]} from every + **/cmake-build-/metrics_by_example.json under root.""" + pairs = {} + pat = os.path.join(root, '**', 'metrics_by_example.json') + for f in sorted(glob.glob(pat, recursive=True)): + board = os.path.basename(os.path.dirname(f)) + if not board.startswith('cmake-build-'): + continue + fam = board_family(board[len('cmake-build-'):], repo_root) + if not fam: + print(f'pair_compare: no family for {board}, skipping', file=sys.stderr) + continue + try: + data = json.load(open(f)) + except (OSError, ValueError) as e: + print(f'pair_compare: unreadable {f} ({e}), skipping', file=sys.stderr) + continue + for ex, ent in data.items(): + pairs.setdefault((fam, ex), []).extend(ent.get('files', [])) + return pairs + + +def main(): + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument('--base-dir', required=True) + ap.add_argument('--new-dir', required=True) + ap.add_argument('--out', default='metrics_compare') + a = ap.parse_args() + repo_root = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + + base = collect(a.base_dir, repo_root) + new = collect(a.new_dir, repo_root) + common = sorted(set(base) & set(new)) + dropped = sorted(set(base) ^ set(new)) + + if not common: + with open(a.out + '.md', 'w') as f: + f.write('_Code-size comparison skipped: no (family, example) pair was built ' + 'on both the base branch and this PR._\n') + return + + def synth(pairs, path): + with open(path, 'w') as f: + json.dump({'files': [e for k in common for e in pairs[k]]}, f) + + with tempfile.TemporaryDirectory() as td: + b, n = os.path.join(td, 'base.json'), os.path.join(td, 'new.json') + synth(base, b) + synth(new, n) + comparison = metrics.compare_files(b, n, ['tinyusb/src']) + if comparison is None: + with open(a.out + '.md', 'w') as f: + f.write('_Code-size comparison failed to produce data._\n') + return + metrics.write_compare_markdown(comparison, a.out + '.md', 'name+') + + with open(a.out + '.md', 'a') as f: + fams = sorted({k[0] for k in common}) + f.write(f'\n_Scoped compare: {len(common)} (family, example) pairs across ' + f'{", ".join(fams)}._\n') + if dropped: + f.write('_Not compared (missing on one side): ' + + ', '.join(f'{fam}:{ex}' for fam, ex in dropped) + '._\n') + + +if __name__ == '__main__': + main() +``` + +- [ ] **Step 4: Run tests** + +Run: `python3 test/hil/test/test_ci_metrics.py -v` — all pass. + +- [ ] **Step 5: Commit** + +```bash +git add .github/scripts/metrics_pair_compare.py test/hil/test/test_ci_metrics.py +git commit -m "ci: add (family, example)-intersection code-size compare for scoped PRs" +``` + +--- + +### Task 10: GitHub Actions wiring (`build.yml` + `build_util.yml`) + +**Files:** +- Modify: `.github/workflows/build.yml`, `.github/workflows/build_util.yml` + +**Interfaces:** +- `set-matrix` new outputs: `example_map` (JSON `{family: [example]}`), `build_filtered` (`'true'`/`'false'`), `build_families_regex` (`fam1|fam2`, only when filtered). +- `build_util.yml` new input `example-map` (string, default `''`); when set, each leg resolves `-e` flags for its `matrix.arg` family and appends them (via env `$EX_ARGS`) to the Build and Membrowse invocations; metrics upload also grabs `metrics_by_example.json`. +- `code-metrics` gains `needs: set-matrix` and a scoped-baseline path. + +- [ ] **Step 1: Rename + thread the selection in `set-matrix`** + +Rename the step `HIL selection (PR only)` → `CI selection (PR only)` (id stays `hil-select`; renaming the id would touch every `steps.hil-select` reference — leave it). In the **Generate matrix json** step, replace the first three lines of the script (`MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py)` and the two echo lines) with: + +```bash + # Build matrix, scoped by the PR selection when one exists. Best-effort: + # ci_set_matrix falls back to the full matrix itself on unusable JSON, + # and an empty $SELECT (non-PR event, selector fallback) means no flags. + if [ -n "$SELECT" ]; then + MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py --select "$SELECT") || MATRIX_JSON='' + else + MATRIX_JSON='' + fi + [ -z "$MATRIX_JSON" ] && MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py) + echo "matrix=$MATRIX_JSON" + echo "matrix=$MATRIX_JSON" >> $GITHUB_OUTPUT + + # Build-axis extras: the per-family example map rides as a side channel + # (a value inside matrix entries would break CircleCI's family parameter + # and multiply GHA matrix legs). NOTE jq's // treats false like null, so + # .build.full is compared explicitly. + EXAMPLE_MAP=$(printf '%s' "${SELECT:-null}" | jq -c '.build.family_examples // {}') || EXAMPLE_MAP='{}' + BUILD_FILTERED=$(printf '%s' "${SELECT:-null}" | jq -r 'if (.build? | type) == "object" and .build.full == false then "true" else "false" end') || BUILD_FILTERED='false' + FAM_REGEX='' + if [ "$BUILD_FILTERED" = "true" ]; then + FAM_REGEX=$(printf '%s' "$SELECT" | jq -r '.build.families | join("|")') || FAM_REGEX='' + [ -z "$FAM_REGEX" ] && BUILD_FILTERED='false' + fi + echo "example_map=$EXAMPLE_MAP" >> $GITHUB_OUTPUT + echo "build_filtered=$BUILD_FILTERED" >> $GITHUB_OUTPUT + echo "build_families_regex=$FAM_REGEX" >> $GITHUB_OUTPUT +``` + +Add to the `set-matrix` job `outputs:` block: + +```yaml + example_map: ${{ steps.set-matrix-json.outputs.example_map }} + build_filtered: ${{ steps.set-matrix-json.outputs.build_filtered }} + build_families_regex: ${{ steps.set-matrix-json.outputs.build_families_regex }} +``` + +- [ ] **Step 2: `build_util.yml` — example-map input** + +Add the input: + +```yaml + example-map: + required: false + default: '' + type: string +``` + +Insert between **Get Dependencies** and **Build**: + +```yaml + - name: Resolve PR example filter + if: inputs.example-map != '' && inputs.example-map != '{}' + env: + # values are PR-derived - keep them out of ${{ }} script interpolation + # (env expansion word-splits but never re-parses shell metacharacters) + EXAMPLE_MAP: ${{ inputs.example-map }} + FAMILY: ${{ matrix.arg }} + run: | + # -e flags for this family; a family absent from the map builds everything + EX_ARGS=$(printf '%s' "$EXAMPLE_MAP" | jq -r --arg fam "$FAMILY" '(.[$fam] // []) | map("-e " + .) | join(" ")') || EX_ARGS='' + echo "EX_ARGS=$EX_ARGS" + echo "EX_ARGS=$EX_ARGS" >> $GITHUB_ENV +``` + +Append `$EX_ARGS` to all three `tools/build.py` invocations (the esp-idf docker line, the generic Build line, and the Membrowse line — build.py maps `examples-membrowse-upload` per example when `-e` is active, because the aggregate target rebuilds everything). Extend the metrics upload: + +```yaml + path: | + cmake-build/cmake-build-*/metrics.json + cmake-build/cmake-build-*/metrics_by_example.json +``` + +- [ ] **Step 3: `cmake` job passes the map** + +In the `cmake` job's `with:` block add `example-map: ${{ needs.set-matrix.outputs.example_map }}`. Do **not** add it to `hil-build`/`hil-build-esp`/`build-os` — hil legs carry `-e` inside their matrix entries; build-os keeps the full example set. + +- [ ] **Step 4: `code-metrics` scoped baseline** + +Verify the download action supports regexp names: +`curl -fsSL https://raw.githubusercontent.com/dawidd6/action-download-artifact/v11/action.yml | grep -n name_is_regexp` — expect a hit. (Fallback if absent: replace the download step below with a `gh run download`-based loop over `build_families_regex` split on `|`, using `gh api` to find the newest master run per artifact; keep the same directory layout.) + +Change `needs: [ check-paths, cmake ]` → `needs: [ check-paths, cmake, set-matrix ]`. Guard the two unscoped steps with the filtered flag: on **Download Base Branch Metrics** change the `if:` to + +```yaml + if: (github.event_name == 'pull_request' || github.event_name == 'workflow_dispatch') && needs.set-matrix.outputs.build_filtered != 'true' +``` + +and on **Compare with Base Branch** change `if: github.event_name != 'push'` to + +```yaml + if: github.event_name != 'push' && needs.set-matrix.outputs.build_filtered != 'true' +``` + +Insert after **Download Base Branch Metrics**: + +```yaml + - name: Download base per-family metrics (scoped PR) + if: github.event_name == 'pull_request' && needs.set-matrix.outputs.build_filtered == 'true' + uses: dawidd6/action-download-artifact@v11 + with: + workflow: build.yml + workflow_conclusion: '' + search_artifacts: true # a docs-only master push uploads no per-family artifacts + branch: ${{ github.base_ref }} + name: ^metrics-(${{ needs.set-matrix.outputs.build_families_regex }})$ + name_is_regexp: true + path: base-family-metrics + continue-on-error: true + + - name: Compare with Base Branch (scoped) + if: github.event_name == 'pull_request' && needs.set-matrix.outputs.build_filtered == 'true' + run: | + # never fall back to the averaged metrics-tinyusb here: a scoped PR vs the + # 64-family/46-example average is exactly the mismatch this path prevents + python .github/scripts/metrics_pair_compare.py \ + --base-dir base-family-metrics --new-dir cmake-build --out metrics_compare + cat metrics_compare.md +``` + +(The PR-side `cmake-build/` dir already holds this run's `metrics_by_example.json` files from the artifact download at the top of the job.) + +- [ ] **Step 5: Validate and commit** + +```bash +python3 -c "import yaml,sys; yaml.safe_load(open('.github/workflows/build.yml')); yaml.safe_load(open('.github/workflows/build_util.yml')); print('yaml ok')" +command -v actionlint >/dev/null && actionlint .github/workflows/build.yml .github/workflows/build_util.yml || true +git add .github/workflows/build.yml .github/workflows/build_util.yml +git commit -m "ci: scope the GHA build matrix and code-metrics baseline by PR selection" +``` + +--- + +### Task 11: CircleCI wiring + +**Files:** +- Modify: `.circleci/config.yml`, `.circleci/config2.yml` + +**Interfaces:** +- `config.yml` set-matrix: on PRs, runs the selector (gated on its own unit suite), scopes `MATRIX_JSON` via `--select`, skips empty toolchains, and forwards `example-map` + `build-filtered` to the continued workflow as pipeline parameters. +- `config2.yml`: declares those parameters; the `build` command resolves `-e` flags per family; `code-metrics` compare is bypassed with a note when filtered; a `no-op` job keeps the workflow valid when nothing is selected. + +- [ ] **Step 1: Verify the continuation orb accepts parameters** + +`curl -fsSL "https://circleci.com/developer/orbs/orb/circleci/continuation" | grep -io 'parameters' | head -1` — the `continuation/continue` command takes a `parameters` input (inline JSON or a file path). If the page is unreachable, proceed — the orb has carried this input since 0.2; the fallback is `parameters: '{"example-map": ...}'` inline via an env-composed string. + +- [ ] **Step 2: `config.yml` — selector + scoping + parameters** + +In the `Set matrix` run command, replace the first two lines (`MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py)` and its echo) with: + +```bash + # PR-scoped selection (best-effort: any failure falls back to the full + # matrix). CircleCI has no base-branch var; tinyusb PRs target master. + SELECT_JSON='' + if [ -n "${CIRCLE_PULL_REQUEST:-}" ]; then + git fetch --no-tags origin master || true + if python3 test/hil/test/test_ci_select.py >/dev/null 2>&1; then + SELECT_JSON=$(python3 tools/ci_select.py --base origin/master) || SELECT_JSON='' + else + echo "ci_select unit suite failed - using the full matrix" + fi + fi + MATRIX_JSON='' + if [ -n "$SELECT_JSON" ]; then + MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py --select "$SELECT_JSON") || MATRIX_JSON='' + fi + [ -z "$MATRIX_JSON" ] && MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py) + echo "MATRIX_JSON=$MATRIX_JSON" + + EXAMPLE_MAP=$(printf '%s' "${SELECT_JSON:-null}" | jq -c '.build.family_examples // {}') || EXAMPLE_MAP='{}' + BUILD_FILTERED=$(printf '%s' "${SELECT_JSON:-null}" | jq -r 'if (.build? | type) == "object" and .build.full == false then "true" else "false" end') || BUILD_FILTERED='false' + jq -n --arg map "$EXAMPLE_MAP" --arg filt "$BUILD_FILTERED" \ + '{"example-map": $map, "build-filtered": $filt}' > /tmp/continue_params.json +``` + +In the toolchain loop, after `FAMILY=$(echo $MATRIX_JSON | jq -r ".\"$toolchain\"")` add: + +```bash + if [ "$(echo "$FAMILY" | jq 'length')" = "0" ]; then + # an empty matrix parameter is a hard CircleCI config error, not a skip + echo "skip build-${build_system}-${toolchain}: no families selected" + continue + fi +``` + +(the `continue` also keeps the alias out of `BUILD_ALIASES`, so `code-metrics` never requires a job that was not generated). Guard the code-metrics emission and keep the workflow non-empty: + +```bash + if [ ${#BUILD_ALIASES[@]} -gt 0 ]; then + echo " - code-metrics:" >> .circleci/config2.yml + echo " requires:" >> .circleci/config2.yml + for alias in "${BUILD_ALIASES[@]}"; do + echo " - $alias" >> .circleci/config2.yml + done + else + # a workflow with zero jobs is invalid config + echo " - no-op" >> .circleci/config2.yml + fi +``` + +(replacing the current unconditional code-metrics block). Change the continuation call to: + +```yaml + - continuation/continue: + configuration_path: .circleci/config2.yml + parameters: /tmp/continue_params.json +``` + +- [ ] **Step 3: `config2.yml` — parameters, `-e` resolution, scoped-compare note, no-op job** + +At the top, after `version: 2.1`: + +```yaml +parameters: + example-map: + type: string + default: "{}" + build-filtered: + type: string + default: "false" +``` + +In the `build` command's **Build** step, before the toolchain if/else, insert: + +```bash + # PR example filter for this family ('{}' or a missing key = build all). + # The parameter is a JSON string composed by set-matrix from ci_select. + EX_ARGS=$(printf '%s' '<< pipeline.parameters.example-map >>' | jq -r --arg fam "<< parameters.family >>" '(.[$fam] // []) | map("-e " + .) | join(" ")' 2>/dev/null) || EX_ARGS='' +``` + +and append `$EX_ARGS` to both `tools/build.py` invocations (docker esp-idf and the generic one). In `code-metrics`, wrap the existing compare `when:` condition with the filter guard and add the note branch: + +```yaml + - when: + condition: + and: + - not: + equal: [ master, << pipeline.git.branch >> ] + - equal: [ "false", << pipeline.parameters.build-filtered >> ] + steps: + # ... the existing Download Base Branch Metrics + Compare + store_artifacts steps, unchanged ... + - when: + condition: + and: + - not: + equal: [ master, << pipeline.git.branch >> ] + - equal: [ "true", << pipeline.parameters.build-filtered >> ] + steps: + - run: + name: Scoped build - comparison unavailable + command: | + # CircleCI stores only the averaged metrics.json; the per-example + # baseline lives on GHA. See the GHA code-metrics PR comment. + echo "_Code-size comparison skipped on CircleCI: this PR built a scoped example set._" > metrics_compare.md + - store_artifacts: + path: metrics_compare.md + destination: metrics_compare.md +``` + +Add the no-op job beside the other job definitions: + +```yaml + no-op: + docker: + - image: cimg/base:current + resource_class: small + steps: + - run: + name: No families selected + command: echo "PR selection - no families to build on CircleCI" +``` + +- [ ] **Step 4: Validate and commit** + +```bash +python3 -c "import yaml; yaml.safe_load(open('.circleci/config.yml')); yaml.safe_load(open('.circleci/config2.yml')); print('yaml ok')" +command -v circleci >/dev/null && circleci config validate .circleci/config.yml || true +git add .circleci/config.yml .circleci/config2.yml +git commit -m "ci: scope the CircleCI build matrix and example set by PR selection" +``` + +--- + +### Task 12: End-to-end validation, review, hand-off + +**Files:** none new — verification only (fix-ups amend the relevant earlier area). + +- [ ] **Step 1: Full hooks + suites** + +```bash +pre-commit run --all-files # ~55 s; HIL hooks exercise real timeouts deliberately +``` + +Expected: all hooks pass (`ci-select-test` and `hil-test` among them). + +- [ ] **Step 2: Selector scenario table** + +```bash +for f in src/portable/raspberrypi/rp2040/dcd_rp2040.c src/class/cdc/cdc_device.c \ + src/host/usbh.c examples/device/cdc_msc/src/main.c test/hil/hil_test.py \ + src/common/tusb_fifo.c hw/mcu/nordic/nrf5x/x.h; do + echo "== $f" + python3 tools/ci_select.py --diff-file <(echo "$f") test/hil/tinyusb.json 2>/dev/null | \ + python3 -c "import json,sys; s=json.load(sys.stdin); b=s['build']; print('hil_full:', s['full'], ' build_full:', b['full'], ' fams:', len(b['families']), ' mapped:', len(b['family_examples']))" +done +``` + +Expected (spot-check against the spec's measured table): rp2040 → 1 family; cdc_device → all families, mapped lists; usbh → ~25 families; example → all families, 1-example lists; test/hil → 0 families, hil_full true; common → build_full true; hw/mcu → 1 family (`nrf`). + +- [ ] **Step 3: Matrix + build smoke** + +```bash +SEL=$(python3 tools/ci_select.py --diff-file <(echo src/portable/raspberrypi/rp2040/dcd_rp2040.c) test/hil/tinyusb.json 2>/dev/null) +python3 .github/scripts/ci_set_matrix.py --select "$SEL" | python3 -m json.tool | head +python3 .github/scripts/hil_ci_set_matrix.py --select "$SEL" test/hil/tinyusb.json | python3 -m json.tool | head +python3 tools/build.py -e device/cdc_msc -b stm32f407disco --target all --target tinyusb_metrics +python3 -c "import json; d=json.load(open('cmake-build/cmake-build-stm32f407disco/metrics_by_example.json')); print(sorted(d))" +``` + +Expected: matrix shows only rp2040 under arm-gcc; hil matrix entries carry `-e ... -e device/board_test`; the by-example JSON lists exactly `['device/cdc_msc']`. + +- [ ] **Step 4: Full example set for one board** (repo validation rule after tool changes) + +```bash +cd examples && cmake -B cmake-build-stm32f407disco -DBOARD=stm32f407disco -G Ninja -DCMAKE_BUILD_TYPE=MinSizeRel . \ + && cmake --build cmake-build-stm32f407disco && cd .. +``` + +Expected: builds green (objcopy warnings non-critical per CLAUDE.md). + +- [ ] **Step 5: Local review, then stop** + +Run the `/code-review` skill on the branch diff (user policy: every push carrying local changes gets a local review pass first) and fix what holds up, amending into the appropriate task commits. Then **stop and hand back to the user** — pushing `build-filter` and opening the PR is their call; note for the PR description that the workflow changes only fully prove out on a real PR run (first PR after merge-to-branch should be watched with `gh pr checks --watch`, and the `hil-select` step's warnings checked for silent fallbacks). + +--- + +## Self-Review Notes + +- Spec coverage: rule table (Tasks 2-4), CMake-only scan (Task 2), orphan invariant (Task 2), build/hil_examples JSON contract (Task 4), `ci_set_matrix` flags (Task 5), `hil_ci_set_matrix -e` (Task 6), `build.py -e` incl. membrowse aggregate-dependency workaround (Task 7), metrics by-example + intersection compare + never-fall-back rule (Tasks 8-10), GHA side channel + injection-safe env passing (Task 10), CircleCI empty-toolchain/alias/no-op fixes + parameters (Task 11), move fallout table (Task 1). +- Known deviation from the spec text, both directions justified inline: `hil_examples` uses the *narrowed* chosen test list when a board is narrowed (the spec's JSON example implies this; its prose says `board_tests` — the narrowed form is a strict subset and matches what the rig runs, and re-run specs are subsets of it). +- Spec's measured "hcd_max3421.c → 1 leg" is really 1 *bsp* family (`espressif`) that neither provider's family list builds → 0 CI legs; Task 3's rule-4 test therefore asserts shape, not that specific count. diff --git a/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md b/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md new file mode 100644 index 000000000..9fa358bee --- /dev/null +++ b/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md @@ -0,0 +1,501 @@ +# PR-scoped CI selection: promoting hil_select to tools/ci_select.py + +**Date:** 2026-08-19 +**Branch:** `build-filter` + +## Motivation + +Every PR builds every example on one board per family, on both CI providers: **74 legs / +2494 example-builds** on the GitHub Actions `cmake` job, and 129 family-legs per build system +on CircleCI (which runs cmake *and* make, plus clang/IAR). Most PRs touch one port, one class, +or one example, and a `hid_host.c` change cannot break an MSC device example on msp430. + +`hil-build` is worse in a different way: it builds **1702 example-builds** (37 board-builds × +46 examples, `--target all`) to run a test suite that needs at most **515**. The HIL example +universe is only 21 of the 46 examples in tree, and the median board needs 15 of them. + +`test/hil/helper/hil_select.py` already maps a PR diff to affected boards and per-board test +lists for HIL, and already owns both mappings the build matrix needs: port-to-family, and +class-macro-to-example +(`docs/superpowers/specs/2026-07-29-hil-pr-scoped-selection-design.md`). That design listed +"scoping the non-HIL build jobs" as an explicit non-goal; this is that follow-up. + +## Goal / non-goals + +**Goal:** promote the selector to a repo-wide `tools/ci_select.py` whose single classification +of a diff drives **all three** CI axes from one rule table — which families to build, which +example targets to build on each, and which rig boards run which tests — wired into +`ci_set_matrix.py` and `hil_ci_set_matrix.py` so both providers and the rig filter from one +source. Scoping applies to `pull_request` events only; push, release and `workflow_dispatch` +keep the full matrix. + +**Non-goals:** +- Variant-level or board-level selection below one-board-per-family on the build axis (all + variants of a selected HIL board still build and run). +- Changing `hil_test.py` behaviour. The selector only *composes* existing `-b` / `-bt` args. +- Changing which tests HIL decides to run. The HIL board/test decision is preserved except for + the single rule-7 change called out below. + +## The rule table + +One classification, three outputs. Every rule yields build families, build examples, and HIL +boards/tests. Pairs are unioned **per family** (build) and **per board** (HIL), so a mixed diff +never inflates one axis with another's breadth. + +`DEV` = 33 `examples/device/*`, `HOST` = 9, `DUAL` = 3, `TYPEC` = 1, `ALL` = 46. +`FAM` = the families whose `family.cmake` references the changed path (CMake only — see below). +"roster boards" = boards on `test/hil/{tinyusb,hfp}.json`. + +| # | Changed path | Build families | Build examples | HIL boards → tests | +| --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | +| 1 | `docs/`, `.claude/`, `*.md`, `*.rst`, `LICENSE` | — | — | — | +| 2 | `test/hil/**` | — | — | all boards → all tests | +| 2b | `tools/metrics.py`, `.github/scripts/metrics_*.py` | `ALL` (unchanged — `tinyusb_metrics` runs `metrics.py` as a build target) | `ALL` | — (nothing on the rig runs it) | +| 3 | `src/portable//dcd_*`, `*_device.[ch]` | `FAM` | `DEV`+`DUAL` | `FAM`'s device-role boards → device+dual tests | +| 4 | `src/portable//hcd_*`, `*_host.[ch]` | `FAM` | `HOST`+`DUAL` | `FAM`'s host-role boards → host+dual tests | +| 5 | `src/portable//**` (anything else) | `FAM` | `ALL` | `FAM`'s boards → all their tests | +| 5b | `src/portable//**` where `FAM` is empty | — | — | — (empty resolves to nothing on BOTH axes) | +| 6 | `hw/bsp//**` | that family | `ALL` | that family's boards → all tests (a `boards//` path narrows to that board) | +| 7 | `hw/mcu//**` | `FAM` — empty resolves to nothing (maintainer ruling) | `ALL` | `FAM`'s boards → all tests; empty resolves to nothing (maintainer ruling) ⚠ *see below* | +| 8 | `src/class//*_device.[ch]` | `ALL` | examples enabling `CFG_TUD_` | device-role boards → HIL tests enabling `CFG_TUD_` | +| 9 | `src/class//*_host.[ch]` | `ALL` | examples enabling `CFG_TUH_` | host-role boards → HIL tests enabling `CFG_TUH_` | +| 10 | `src/class//**` (shared header) | `ALL` | either, **plus include-edge classes** | both roles → same, plus include-edge classes | +| 11 | `src/device/**` | `ALL` | `DEV`+`DUAL` | device-role boards → device+dual tests | +| 12 | `src/host/**` | `ALL` | `HOST`+`DUAL` | host-role boards → host+dual tests | +| 13 | `examples///**` | `ALL` | just `` | if `` is a HIL test: all boards → that test; else nothing | +| 14 | `examples/device/board_test/**` | `ALL` | just `board_test` | all boards → all tests (HIL parking firmware) | +| 15 | `examples/build_system/**`, `examples/CMakeLists.txt`, `examples//CMakeLists.txt` | `ALL` | `ALL` | all boards → all tests | +| 16 | `src/common/`, `src/osal/`, `src/tusb.[ch]`, `src/tusb_option.h`, `tools/build*.py`, `tools/cmake/**`, `hw/bsp/{family_support.cmake,board.c,board_api.h,ansi_escape.h}`, `.github/**` | `ALL` | `ALL` | all boards → all tests | +| 16a | `lib//**` | `ALL` | examples whose own `CMakeLists.txt`/`Makefile` names `lib/` | those examples that are HIL tests, on all boards; empty resolves to nothing | +| 16b | `tools/get_deps.py` | families whose `deps_mandatory`/`deps_optional` entries changed | `ALL` | those families' boards → all tests; a logic change, an `'all'` entry, no base content or a changed token naming no family → full | +| 17 | anything unclassified | `ALL` | `ALL` | all boards → all tests (fail-open) | + +**Rule 2 is deliberately asymmetric.** A `test/hil/**` change is invisible to the family matrix +but is exactly what the rig exercises, so it builds nothing and runs everything. + +**Rule 7 is the one HIL-side behaviour change in this design.** Today `hw/mcu/` sits in +`hil_select`'s `_FULL_RE` and forces the full HIL matrix. Since the build axis now resolves +those paths to a family through the same scan, forcing full on the rig is inconsistent. The +path fires rarely — 4 commits in 3 years — so this is low-risk either way; if you would rather +keep the HIL view untouched, rule 7's HIL column becomes "all boards → all tests" and nothing +else in this design changes. + +Rules 8–10 reuse machinery `hil_select` already has — `class_macros`, `_config_enables`, +`class_include_edges` — applied over all 46 examples' `src/tusb_config.h` for the build axis +and over the HIL test list for the HIL axis. The include edges are why an `audio.h` change also +selects the MIDI examples (`midi{,2}_{device,host}.h` include `class/audio/audio.h`) and a +`cdc.h` change the net one. + +### Buildability post-filter (build axis) + +After the pairs are unioned, every `(family, examples)` pair is pruned with +`build_utils.skip_example(example, )` — the same `skip.txt` / `only.txt` +data CMake's `family_filter` uses (40 `skip.txt`, 13 `only.txt` in tree). Examples the family +cannot build are dropped; a family left with none is dropped entirely. + +This is where most of the host-side saving comes from: only 23 of 75 CI families can build +`host/bare_api` at all, and 2 can build `typec/power_delivery`. + +### Measured effect + +GHA `cmake` job, baseline **74 legs / 2494 example-builds**; `hil-build`, baseline **1702 +example-builds** across 37 board-builds. + +| PR shape | Build legs | Build ex-builds | HIL boards | hil-build ex-builds | +| ---------------------------- | ---------: | --------------: | ---------: | ------------------: | +| `dcd_rp2040.c` | 1 | 35 | 2 | 32 | +| `hcd_max3421.c` | 1 | 10 | 7 | 36 | +| `hw/bsp/stm32f4/**` | 1 | 45 | 1 | 15 | +| `dcd_dwc2.c` | 20 | 646 | 10 | 184 | +| `hid_host.c` | 24 | 68 | — | — | +| `msc_host.c` | — | — | 9 | 40 | +| `usbh.c` | 25 | 217 | 10 | 57 | +| `msc_device.c` | 74 | 350 | — | — | +| `cdc_device.c` | 74 | 588 | 27 | 192 | +| `examples/device/cdc_msc/**` | 73 | 73 | 25 | 60 | +| `usbd.c` | 74 | 2297 | 27 | 472 | +| `src/common/**` (full) | 74 | 2494 | 30 | 515 | +| `test/hil/**` only | 0 | 0 | 30 | 515 | + +The full-matrix row is the headline for `hil-build`: even with **no** PR narrowing, per-board +example selection takes it from 1702 to 515. + +### Why "empty means empty" + +Both views answer an empty `FAM` the same way (rule 5b): nothing. The HIL view used to force +the full 30-board rig there, on the theory that an empty result might be a scan miss — but the +build view answered the identical condition with zero families for the same path, so the rig +ran every board to validate a file that nothing compiled. In the build view that theory costs +74 legs, and the +evidence does not support it: of the 28 `src/portable/*/*` directories, **26 resolve to at +least one family**. The two that do not are both real orphans as far as CI is concerned: +`microchip/pic` (only `dcd_pic.c` and a README, with no `hw/bsp/pic` family at all) and +`microchip/pic32mz` (`hw/bsp/pic32mz` has only a `family.mk`, and `pic32mz` is in neither +provider's family list, so no CI job builds it today). A file no CI job compiles cannot be +validated by building anything. + +The safety this gives up is recovered structurally: a unit test asserts every +`src/portable/*/*` and every tracked `hw/mcu/` resolves to ≥1 family, with an explicit +allowlist of known orphans (`microchip/pic`, `microchip/pic32mz`). Adding a port without +wiring a family then fails +pre-commit instead of silently building nothing on every later PR. Same enforcement style as +the existing `test_hil_util.BottomLayer` structural tests. + +Fail-open survives where it belongs: an *unclassified* path or any exception widens to `ALL` on +every axis. + +### Why `hw/mcu/**` is rule 7 and not "full" + +`hw/mcu` is overwhelmingly dependency territory — `tools/get_deps.py` has 87 entries under it, +and those paths are gitignored, so they can never appear in a diff. Only 51 files survive +in-tree, touched 4 times in 3 years, and they resolve through the same scan the ports use: + +| Tracked directory | In `get_deps`? | Resolves to | +| ----------------------------- | ------------------------------------------- | ----------- | +| `hw/mcu/dialog/` (`da1469x`) | **no** — real in-repo MCU support, 21 files | `da1469x` | +| `hw/mcu/nordic/` (`nrf5x`) | beside the `nrfx` dep | `nrf` | +| `hw/mcu/sony/` (`cxd56`) | beside the `spresense-exported-sdk` dep | `cxd56` | +| `hw/mcu/bridgetek/` (`ft9xx`) | beside the `ft90x-sdk` dep | `ft9xx` | + +Rule 7 is therefore not a mechanism of its own — it is rules 3–5's scan pointed at a second +tree, because `src/portable/` and `hw/mcu/` ask the same question. + +Unlike the port rule, an `hw/mcu` path that resolves to no family contributes *nothing* on +either axis (maintainer ruling): if no family's build references it, no build compiles it. The +table above is kept honest by `test_tracked_mcu_vendors_resolve`, which fails pre-commit if a +tracked vendor directory stops resolving. + +### Why `lib/**` is rule 16a and scanned per example + +Its tracked contents (`SEGGER_RTT`, `networking`, `rt-thread`, `embedded-cli`, 22 commits in +3 years) are wired in at `examples/build_system` and per-example `CMakeLists.txt`, not per +family — so the family scan the ports use is the wrong instrument here: it would *wrongly* +narrow `SEGGER_RTT` to the three families that name the path in their `family.cmake`, while +the path is not compiled by any of them by default (it is reached only through `LOGGER=rtt`). +That scan stays applied to `src/portable/` and `hw/mcu/` only. + +Rule 16a asks the per-example question instead (maintainer ruling: only the examples that use +the lib need building): `lib_examples()` reads each example's own `CMakeLists.txt` and +`Makefile` and keeps the ones naming `lib/` at a directory boundary. Every family stays +in play — any of them can build those examples — while the example list collapses: + +| Tracked lib | Examples that build it | HIL tests among them | +| -------------- | ----------------------------------------------------------- | -------------------- | +| `embedded-cli` | `host/msc_file_explorer`, `host/msc_file_explorer_freertos` | both | +| `networking` | `device/net_lwip_webserver` | none (test disabled) | +| `SEGGER_RTT` | — | — | +| `rt-thread` | — | — | + +`SEGGER_RTT` and `rt-thread` resolve to nothing, and "empty means empty" applies: no CI build +compiles them, so there is nothing to validate by building. + +### Why `tools/get_deps.py` is rule 16b + +`deps_mandatory` / `deps_optional` are data: `path -> [url, commit, 'fam1 fam2 ...']`. A commit +bump therefore affects exactly the families listed in that entry, and building the other 70+ is +pure waste. `get_deps_changed_families()` parses both sides of the file with `ast` (never +`exec` — this is PR content), diffs the two dict literals **separately**, and unions the family +tokens of every added, removed or edited entry, from **both** sides (a removed entry has only a +base side; an edited family list must cover the families that lose the dep as well as the ones +that gain it). Separately, because merging the dicts before diffing hides a *move* between +`deps_mandatory` and `deps_optional` — the value is untouched, but mandatory deps are fetched +for every family, so demoting one stops families fetching it. + +It falls open to the full matrix whenever the entries are not the whole answer: + +* anything outside the two dict assignments differs — a logic change to `get_deps` can change + what every family fetches (compared as `ast.dump(..., annotate_fields=False)` of the module + with those two assignments removed, so comments and reformatting alone are not a logic + change); +* an `'all'` entry (every mandatory dep) changed; +* the file will not parse; +* there is no base content: `--diff-file` mode has no git, so no merge-base blob; +* a changed entry carries a family token that names no `hw/bsp/` and is not one of the + eight known aliases. "Changed but unmappable" is not "nothing changed": reading it as the + latter empties the whole build matrix for a dep bump. + +The eight known aliases (`sam3x`, `samd21`, `samd51`, `same5x`, `fc100s`, `spresense`, +`stm32l1`, `stm32l5`) are pinned in `_DEPS_ALIAS_TOKENS` and select nothing. `get_deps` matches +a token against a requested family name verbatim (`f in entry[2].split()`), so these tokens +match nothing there either — four are pre-rename spellings listed beside the current name in +the same entry, two point at a differently-named family dir (`fc100s`→`f1c100s`, +`spresense`→`cxd56`, both unreachable in `get_deps` itself), and two name no family in the tree. +A ninth appearing fails `TestOrphanInvariant`. + +## Component: `tools/ci_select.py` + +`git mv test/hil/helper/hil_select.py tools/ci_select.py` (history preserved). The HIL +classifier is unchanged apart from rule 7; a second, independent build classifier is added +beside it. One diff read, two classifiers, one unit suite. + +``` +python3 tools/ci_select.py --base [--diff-file ] [CONFIG.json ...] +``` + +`configs` becomes `nargs='*'`. With rosters it emits everything it emits today plus the new +keys; with none it emits only the build view, so CircleCI never needs to know HIL exists. + +```json +{ + "full": false, + "boards": {"raspberry_pi_pico": "all"}, + "families": ["rp2040"], + "args": {"tinyusb.json": "-b raspberry_pi_pico"}, + "args_flasher": {"tinyusb.json": {"openocd": "-b raspberry_pi_pico"}}, + "hil_examples": {"raspberry_pi_pico": ["device/cdc_msc", "device/board_test"]}, + "build": { + "full": false, + "families": ["rp2040"], + "family_examples": { + "rp2040": ["device/cdc_msc", "device/hid_composite", "dual/dynamic_switch"] + } + }, + "reasons": ["src/portable/raspberrypi/rp2040/dcd_rp2040.c: port rp2040 -> ..."] +} +``` + +`build.families` is the build family axis. `build.family_examples` maps a family to its example +list; **a family absent from the map builds all its examples**, so the common "narrow families, +all examples" case carries no payload. `build.full` true means no build narrowing at all. + +`hil_examples` is the new HIL build axis: per roster board, the examples `hil-build` must +produce. It is `board_tests(board)` — which the selector already computes — **plus +`device/board_test`**, which `hil_test.py` flashes to park every board at each variant boundary +and at end-of-board teardown (`hil_test.py:1798`, `:1866`). It is emitted even when +`full: true`, because the HIL example universe is 21 of 46 examples regardless of any diff. + +All pre-existing keys keep their exact meaning, so `.github/scripts/hil_ci_set_matrix.py`, the +HIL legs in `build.yml` and `.claude/skills/pre-pr/SKILL.md` need only a path update. + +### Shared helper change + +`port_families(port_dir, repo_root)` generalizes to a path-to-families reference scan over a +second tree (`hw/mcu/`). Its existing **CMake-only** behaviour is kept unchanged and is now the +rule for every axis: it scans `hw/bsp/*/family.cmake` plus the espressif component +`CMakeLists.txt`, and never `family.mk`. + +CMake is the first-class build system; Make follows whatever CMake decides. A family that +CMake does not wire up to a port is not a consumer of that port, and the Make legs on CircleCI +build the same families CMake does. Scanning `family.mk` as well would only ever *widen* the +selection to families CMake never builds, which is coverage nobody asked for — and it would +resolve `microchip/pic32mz` to a family that appears in no CI family list. + +One consequence to keep in view: because HIL and build now share one scan, there is no +per-caller flag, no second cache key, and no way for the two axes to disagree about which +families own a port. + +**Boundary matching.** A directory reference must match at a directory boundary — a trailing +`/` *or* end-of-token — not as a bare substring. Both traps are live: `hw/bsp/nrf/family.cmake` +writes `${TOP}/hw/mcu/nordic/nrf5x` with no trailing slash, while the existing port scan +requires a trailing `/` precisely to stop `microchip/pic` matching `microchip/pic32mz`. + +### Move fallout + +All mechanical, all one-line: + +| File | Change | +| ---------------------------------- | ----------------------------------------------------------------- | +| `tools/ci_select.py` | `sys.path` walk 4 levels → 2 | +| `test/hil/hil_ci.sh` | drop from the scp list (nothing on the rig imports it) | +| `test/hil/test/test_hil_select.py` | rename to `test_ci_select.py`, import path | +| `test/hil/test/test_hil_util.py` | `BottomLayer` stdlib-closure allowlist + module list | +| `.pre-commit-config.yaml` | both hooks (`hil-select-test` → `ci-select-test`, `files:` globs) | +| `.github/workflows/build.yml` | selector path, step name | +| `.claude/skills/pre-pr/SKILL.md` | selector path | + +The test file stays in `test/hil/test/` — it still consumes the rig rosters and `hil_util`. + +The selector gains one non-stdlib-but-local import: `tools/build_utils.skip_example` for the +buildability post-filter. `build_utils` imports only `subprocess`, `pathlib` and `re`, so the +stdlib closure the bare GitHub runner depends on is preserved; `BottomLayer` must be extended +to cover it. + +Because a wrong parents-count already broke this module once (there is a comment in the source +recording it), the moved module gets a guard test asserting its derived repo root contains +`src/` and `hw/bsp/`. + +## Component: `tools/build.py --example` + +`build.py` has no example filter today. `-T/--target` exists and maps to +`cmake --build --target `, but it hard-fails on a target that does not exist, and absent +targets are routine (40 `skip.txt`, 13 `only.txt`). + +New repeatable `-e/--example /`: + +- Default (none given) keeps today's behaviour exactly: `--target all`. +- Given, each board's list is intersected with `build_utils.skip_example(example, board)`, then + passed as one `--target ` per example. Example target names are the directory names and + are unique across all four roles (verified: 46 examples, zero collisions). +- A board whose intersection is empty is reported **skipped**, not failed. +- `--target tinyusb_metrics` must stay last so metrics run after the examples that feed them. +- The espressif path already builds per example via `get_examples` + `skip_example`; it takes + the same filter. + +Both the family matrix and `hil-build` use this one flag. + +## CI wiring + +### `.github/scripts/ci_set_matrix.py` + +Two mutually exclusive optional flags. **Output shape is unchanged** — `{toolchain: [family]}`, +just fewer families. With no flags the output is byte-for-byte today's, so push, release and +`workflow_dispatch` are untouched. + +| Flag | Caller | Behaviour | +| --------------- | -------- | -------------------------------------------------------- | +| `--select JSON` | GHA | consumes the selector JSON the workflow already computes | +| `--base REF` | CircleCI | runs `tools/ci_select.py` itself | + +`build.full` true, or any exception, prints the full matrix with a warning on stderr. + +### `.github/scripts/hil_ci_set_matrix.py` + +Already takes `--select` and already scopes boards. It additionally appends `-e ` per +board from `hil_examples`, so each `hil-build` entry builds only what its board will run plus +`board_test`. When `hil_examples` is absent (hand-runs), it falls back to today's `--target all`. + +### The example map is a side channel, not a matrix entry + +The build example list deliberately does **not** ride inside the family matrix entry string. On +CircleCI the `family` parameter is also passed to `python tools/get_deps.py +<< parameters.family >>` and tested with `if [ << parameters.family >> == "rp2040" ]` — a value +carrying `-e` flags breaks both — and CircleCI matrix parameters form a cartesian product, so a +parallel `example-args` parameter would multiply the jobs rather than zip with them. + +So `build.family_examples` travels as one JSON blob and each build job resolves its own entry: + +- **GHA:** `set-matrix` exposes it as an output; `build_util.yml` gains an optional + `example-map` input (default `''`); a step resolves `-e` flags for `matrix.arg` with `jq`. +- **CircleCI:** `set-matrix` writes `example_map.json` and `persist_to_workspace`s it; the + `build` job gains `attach_workspace` and resolves the same way. + +Consequences of keeping the matrix shape: the metrics artifact name stays `metrics-`, +and CircleCI's generated `config2.yml` does not inflate to one entry per family. `hil-build` +needs none of this — its matrix entries are already compound per-board strings from +`hil_ci_set_matrix.py`, so `-e` flags go straight in. + +### `.github/workflows/build.yml` + +The existing `HIL selection (PR only)` step in `set-matrix` is already gated on +`pull_request` — exactly the gate wanted. It is renamed, repointed at `tools/ci_select.py`, and +its `select` output is threaded into `ci_set_matrix.py --select`, so the filter costs zero extra +selector invocations. + +`build_util.yml`'s `if: inputs.build-args != '[]'` already skips a toolchain leg whose list is +empty, and a partially-skipped matrix aggregating to success is the documented pattern +`hil-build` already relies on. When every leg is empty (a `test/hil`-only PR), the `cmake` job +has nothing to build. Accepted: GitHub treats a skipped job as satisfying a required status +check, and HIL is unaffected because `hil-build` is a separate matrix. `code-metrics` still +runs (`!cancelled()` plus `cmake` success-or-skipped) and posts a "built no families on this +push" marker, so the sticky size comment never shows a stale table from an earlier push. + +### `.circleci/config.yml` + +The `set-matrix` job passes `--base origin/master` when `CIRCLE_PULL_REQUEST` is set, after +`git fetch --no-tags origin master || true`; unfiltered otherwise. CircleCI does not expose the +PR base branch, so `master` is assumed — true for essentially every tinyusb PR, and any ref or +clone problem falls back to the full matrix. + +Two fixes the GHA side does not need: + +- `gen_build_entry` must **skip** a toolchain whose family list is `[]`. An empty matrix + parameter is a hard CircleCI config error, not a skipped job. +- `BUILD_ALIASES` must collect only aliases that were actually generated, or `code-metrics`' + `requires:` names a job that does not exist. + +## Code metrics + +`tools/metrics.py` averages per-file sizes across every build, and the per-family +`metrics-` artifact stores only that average — over whichever examples were built. Both +build axes therefore break the comparison: a 3-family PR against master's 64-family average, +and an 11-example average against master's 46-example one. + +The fix is to make the artifact carry per-example detail and compare the intersection. + +1. **`metrics.py combine --by-example`** additionally writes `metrics_by_example.json`, + `{example: {files: [...]}}`. The example name is the map.json's parent directory + (`///*.map.json`). +2. `examples/CMakeLists.txt`'s `tinyusb_metrics` target emits both files; `build_util.yml` + uploads both under the existing `metrics-` artifact name. +3. `combine` learns to expand a by-example JSON into one data entry per example and an + `--only-examples` filter, so a subset can be averaged on demand. +4. `code-metrics` computes the **intersection of `(family, example)` pairs present on both + sides**, averages each side over exactly those pairs, and compares. Dropped pairs are named + in the PR comment. An empty intersection skips the compare with an explicit note. + +`search_artifacts: true` is required on the base-side download: a docs-only master push +produces no per-family artifacts — which is why `metrics-carry-forward` exists for the +aggregate — so per-family baselines may come from different master runs. That is still a valid +per-family baseline. + +Today's `metrics-tinyusb` aggregate keeps being produced for the unfiltered path, releases and +`metrics-carry-forward`. The filtered path never falls back to it — that is precisely the +mismatched compare this section exists to prevent. `hil-build` uploads no metrics, so its +narrowing does not touch any of this. + +For narrow PRs this is sharper than today: a `dcd_rp2040` PR's size delta stops being diluted +by a 64-family, 46-example average. + +**Size check the plan must run first:** the by-example JSON is ~46× the entries of today's +average. If it proves too large as an artifact, drop per-symbol detail from the by-example file +(sizes only) — symbols are only needed in the aggregate. The plan must also verify that +`dawidd6/action-download-artifact@v11` supports `name_is_regexp`; the fallback is a +`gh run download` loop. + +## Testing + +Extended in `test/hil/test/test_ci_select.py` (stdlib-only, ~0.1 s, already a pre-commit hook +and already gating CI's selector step): + +- One case per rule 1–17, asserting all three outputs. +- Per-family union: a mixed diff (`dcd_rp2040.c` + `cdc_device.c`) gives `rp2040` the device + list and every other family the CDC list — not the cross product of both. +- Include edges: an `audio.h` change selects the MIDI examples; a `cdc.h` change the net one. +- `hil_examples` always contains `device/board_test` for every selected board, including when + `full: true`, and is otherwise exactly `board_tests(board)`. +- `hil_examples` never exceeds the 21-example HIL universe. +- The scan is CMake-only: a port referenced solely from a `family.mk` (`microchip/pic32mz`) + resolves to no family, and no `family.mk` is ever read. +- Boundary matching: `microchip/pic` does not inherit `microchip/pic32mz`'s families, and + `hw/mcu/nordic/nrf5x` resolves despite having no trailing slash at its reference site. +- Buildability post-filter: `typec/power_delivery` prunes to 2 families, `host/bare_api` to 23. +- Structural invariant: every `src/portable/*/*` and every tracked `hw/mcu/` resolves + to ≥1 family, allowlist `{microchip/pic, microchip/pic32mz}`. +- Every name in `build.families` is a real `hw/bsp/`; every example name on either axis is + a real `examples//` directory. +- Repo-root guard for the moved module. +- `ci_set_matrix.py`: no flags → byte-identical to today; `--select` with `build.full` → + identical; `--select` narrow → a subset; malformed `--select` → full plus a warning. +- `hil_ci_set_matrix.py`: no `hil_examples` → today's args byte-for-byte; with it → `-e` flags + appended per board, `board_test` always present. +- `build.py`: `-e` with an example the board skips builds nothing and reports skipped, not + failed; no `-e` still passes `--target all`. + +## Known gaps + +- **CircleCI size comparison.** CircleCI stores only the combined `metrics.json`, so the + intersection compare is unavailable there; when filtered it prints a note and copies + `metrics.md`. Its `metrics_compare.md` is a stored artifact that nothing reads in review — the + PR comment comes from GHA. Making CircleCI store per-example metrics is a follow-up. +- **HIL re-run attempts.** A re-run spec is a subset of the original selection, so the + firmware `hil-build` produced already covers it. This holds only while re-run specs stay + subsets; a future "re-run with extra tests" feature would need `hil-build` re-run too. +- **Membrowse** receives rows for fewer families and fewer examples on filtered PRs. If that + service misbehaves, the escape hatch is keeping the membrowse upload leg unfiltered. +- **`typec/power_delivery`** is reached only through rules 5 and 13 (`src/portable/st/typec` + has neither a `dcd_` nor an `hcd_` prefix, so it selects `ALL` examples on its 5 families, + which the post-filter then prunes to 2). A dedicated typec rule is possible later; the + post-filter already makes it cheap. +- **A `test/hil`-only PR reports `cmake` as skipped** rather than passing. Accepted; + revertible with a one-family floor if branch protection turns out to disagree. + `code-metrics` still runs in that case: with no `cmake-build/*/metrics.json` to + aggregate it writes `_Code-size comparison skipped: PR selection built no families + on this push._` and posts that as the sticky comment, so the size section reflects + THIS push instead of keeping the previous one's table. +- **`microchip/pic32mz` builds nothing.** The scan is CMake-only and `hw/bsp/pic32mz` ships + only a `family.mk`, so a change there selects no family. That matches reality — `pic32mz` is + in neither provider's family list — but it means the port is unbuilt by CI whether or not + this design lands. Giving it a `family.cmake` is the fix, and is out of scope here. +- **Seven bsp families are in no CI toolchain today** (`espressif`, `efm32`, `same7x`, + `cxd56`, `f1c100s`, `pic32mz`, `py32f0`); the intersection drops them, matching current + behaviour. This change does not alter that. -- cgit v1.3.1 From 375e20090a0a60fd2d47ba6553d189e05482dfea Mon Sep 17 00:00:00 2001 From: Ha Thach Date: Tue, 25 Aug 2026 15:04:42 +0700 Subject: hil: make hil_report.md a rendering of hil_report.json (#3840) hil_report.json and hil_report.md were written independently. Four writers produced the markdown and three wrote no JSON at all -- and those three are the paths where a run died, so a JSON consumer saw nothing exactly when it mattered: the per-board verdicts an agent hands back reported the whole fleet as "no report row" while a human read the real story from the markdown. Every writer now goes through render_report(), so a table can never contain something the JSON does not. The document gains `scope` (a three-board PR run and a full run that lost 24 boards were indistinguishable) and `caveat` (how the run ended). `banner` carries rig health across an --accumulate retry; `caveat` records how a run ended and must not -- conflating them made a clean retry publish an abandonment that never happened. helper/hil_report.py owns the document end to end, dissolving the import cycle that forced write_timeout_report to compose its own markdown and removing a duplicate cell classifier kept in sync by hand. hil_summary.py is deleted; its CLI moves there. hil_ci.sh uploads the sidecar so a remote --accumulate has a merge bas --- .claude/agents/hil-operator.md | 6 +- .claude/workflows/hil-validate.js | 37 +- .claude/workflows/test-hil-validate.mjs | 29 +- .../followup/pr3803-hil-blindness-reporting.md | 3 +- .../followup/pr3836-report-single-source.md | 470 -------- .../followup/pr3840-mret-board-result.md | 90 ++ .../followup/pr3840-skill-md-no-boards-drift.md | 38 + .../followup/pr3840-write-report-atomicity.md | 30 + .../plans/2026-08-21-hil-report-module.md | 658 +++++++++++ .../specs/2026-08-21-hil-report-module-design.md | 144 +++ test/hil/helper/hil_health.py | 39 +- test/hil/helper/hil_report.py | 544 +++++++++ test/hil/helper/hil_summary.py | 115 -- test/hil/hil_ci.sh | 113 +- test/hil/hil_test.py | 222 +--- test/hil/test/test_ci_select.py | 1 + test/hil/test/test_hil_bounded.py | 149 +-- test/hil/test/test_hil_health.py | 43 - test/hil/test/test_hil_report.py | 1162 ++++++++++++++++++++ 19 files changed, 2880 insertions(+), 1013 deletions(-) delete mode 100644 docs/superpowers/followup/pr3836-report-single-source.md create mode 100644 docs/superpowers/followup/pr3840-mret-board-result.md create mode 100644 docs/superpowers/followup/pr3840-skill-md-no-boards-drift.md create mode 100644 docs/superpowers/followup/pr3840-write-report-atomicity.md create mode 100644 docs/superpowers/plans/2026-08-21-hil-report-module.md create mode 100644 docs/superpowers/specs/2026-08-21-hil-report-module-design.md create mode 100644 test/hil/helper/hil_report.py delete mode 100644 test/hil/helper/hil_summary.py create mode 100644 test/hil/test/test_hil_report.py (limited to 'docs/superpowers/plans') diff --git a/.claude/agents/hil-operator.md b/.claude/agents/hil-operator.md index a37501211..81fdc08e9 100644 --- a/.claude/agents/hil-operator.md +++ b/.claude/agents/hil-operator.md @@ -68,12 +68,12 @@ For a board run, do NOT transcribe the report table. Run the tests, then hand ba output verbatim: ```bash -python3 test/hil/helper/hil_summary.py -b BOARD [-b BOARD...] # from the report dir +python3 test/hil/helper/hil_report.py -b BOARD [-b BOARD...] # from the report dir ``` -`{"results": , "banner": , "wedged": ["board", ...]}` +`{"results": , "banner": , "caveat": , "wedged": ["board", ...]}` -`results` and `banner` are copied, never retyped, reworded or re-ordered: report rows are named +`results`, `banner` and `caveat` are copied, never retyped, reworded or re-ordered (`caveat` is the run-level notice — abandoned, aborted, no-boards — and it can say the run failed while every row says pass): report rows are named per variant, a variant name need not start with the board name, and lock contention is a cell rather than a phrase, so re-deriving any of it by hand is how this contract broke before. `wedged` is yours — the boards your run left unresponsive, usually none — and the only field you diff --git a/.claude/workflows/hil-validate.js b/.claude/workflows/hil-validate.js index bc0bda8b9..741ba481d 100644 --- a/.claude/workflows/hil-validate.js +++ b/.claude/workflows/hil-validate.js @@ -11,10 +11,10 @@ if (!args || !Array.isArray(args.boards) || args.boards.length === 0) { throw new Error('args must be { boards: string[], force? } with the boards already built') } -// The operator returns hil_summary.py's JSON verbatim plus its own observations. It does NOT +// The operator returns hil_report.py's JSON verbatim plus its own observations. It does NOT // retype the report table: rows are named per variant, a variant need not start with the board // name, and lock contention is a cell rather than a phrase — rebuilding board identity from -// prose produced a defect in each of four review rounds. hil_summary.py does that join against +// prose produced a defect in each of four review rounds. hil_report.py does that join against // the roster, so `locked` and `ran` arrive as fields and nothing here parses a detail string. const BOARD = { type: 'object', additionalProperties: false, @@ -26,12 +26,16 @@ const BOARD = { } const HIL = { type: 'object', additionalProperties: false, - required: ['results', 'wedged'], + required: ['results', 'wedged', 'caveat'], properties: { results: { type: 'array', items: BOARD }, // the operator's own observation — not derivable from the report wedged: { type: 'array', items: { type: 'string' } }, banner: { type: 'string' }, + // the run-level caveat: abandoned / aborted / selected-no-boards. `banner` carries rig + // HEALTH across an --accumulate retry; `caveat` carries how THIS run ended, and every + // row can still say pass while it failed — so it gates `pass` in summarize() below. + caveat: { type: 'string' }, }, } @@ -51,12 +55,12 @@ const runBoards = (boards, isRetry = false) => agent( (args.force ? 'THE USER HAS EXPLICITLY AUTHORIZED FORCING: run hil_test.py with HIL_NO_BOARD_LOCK=1 in the environment (bypasses the board lock check; do NOT release or kill the existing holder). ' : 'A board whose lock is held (a dev session or concurrent CI job) fails fast inside the run without blocking the others — never force the lock. ') + - 'If hil_test.py refuses the run with "board(s) not in ", re-run it WITHOUT the unknown names but keep the FULL board list on the hil_summary call below — it emits a ran:false entry for every board you name, so the unknown ones surface as "no report row" instead of costing the whole batch. ' + + 'If hil_test.py refuses the run with "board(s) not in ", re-run it WITHOUT the unknown names but keep the FULL board list on the hil_report call below — it emits a ran:false entry for every board you name, so the unknown ones surface as "no report row" instead of costing the whole batch. ' + 'Use the config for this host (hostname first). Run hil_test.py as a BACKGROUND Bash task and wait for it (a stuck fleet runs to its pool guard, 60 min by default — beyond any foreground timeout); never cancel it early. ' + 'On non-lock failures retry ONCE from the re-run spec hil_test.py just wrote — `.failed`, which already begins with --accumulate — adding -v. A usbtest battery that produced per-case verdicts is NOT auto-retried, so its result already stands. ' + 'THEN, from the directory the run wrote its report to, produce the results with:\n' + - ` python3 test/hil/helper/hil_summary.py ${boards.map((b) => `-b ${b}`).join(' ')}\n` + - 'Return its `results` array and `banner` EXACTLY as printed — do not retype, reword, re-order or "correct" them, and never transcribe the markdown table instead. ' + + ` python3 test/hil/helper/hil_report.py ${boards.map((b) => `-b ${b}`).join(' ')}\n` + + 'Return its `results` array, `banner` and `caveat` EXACTLY as printed — do not retype, reword, re-order or "correct" them, and never transcribe the markdown table instead. ' + 'Add `wedged`: the board names whose board or fixture your run left unresponsive (usually none). That is your own observation and the one field you author; put `dmesg | tail -50` in your reply text for any board you list.', { label: boards.length === 1 ? `hil:${boards[0]}` : `hil:${boards.length} boards`, @@ -64,7 +68,7 @@ const runBoards = (boards, isRetry = false) => agent( }, ) -// A lookup, not a reconciliation: hil_summary.py emits exactly one entry per requested board, +// A lookup, not a reconciliation: hil_report.py emits exactly one entry per requested board, // so a missing entry means the operator dropped it rather than that the names disagree. const byBoard = (out) => new Map((out?.results || []) .filter((r) => r && typeof r.board === 'string') @@ -91,7 +95,9 @@ const results = args.boards.map((b) => { }) for (const r of results) log(`${r.board}: ${r.pass ? 'PASS' : r.locked ? 'LOCKED' : 'FAIL'}`) if (first?.banner) log(`report banner: ${first.banner.trim().split('\n')[0]}`) +if (first?.caveat) log(`report caveat: ${first.caveat.trim().split('\n')[0]}`) +let runCaveat = first?.caveat || '' // A concurrent CI job may have held some boards (its hil_test.py flock). // CI finishes a board in minutes — retry locked boards once, at the end. if (!args.force) { @@ -116,19 +122,26 @@ if (!args.force) { } log(`${b}: retry ${results[i].pass ? 'PASS' : 'FAIL'}`) } + // the retry's own run-level verdict, not the first attempt's: a retry that abandoned + // or aborted must sink the run even though its rows may all say pass. + if (again?.caveat) runCaveat = again.caveat } } -// pass/wedged/locked in one place so it can be exercised without running an agent -const summarize = (rs, force) => ({ - pass: rs.every((r) => r.pass), +// pass/wedged/locked in one place so it can be exercised without running an agent. +// `caveat` is a RUN-level verdict and must gate `pass`: on the abandon and no-boards +// paths every row can legitimately say pass while the run itself failed (hil_test.py +// os._exit(1) -- a red job), so per-row agreement alone published those runs as green. +const summarize = (rs, force, caveat) => ({ + pass: rs.every((r) => r.pass) && !/^\*\*HIL run (abandoned|aborted|selected no boards)/m + .test(caveat || ''), wedged: rs.filter((r) => r.wedged).map((r) => r.board), locked: force ? [] : rs.filter((r) => !r.pass && r.locked).map((r) => r.board), }) -const { pass, wedged, locked } = summarize(results, args.force) +const { pass, wedged, locked } = summarize(results, args.force, runCaveat) if (wedged.length) log(`WEDGED boards needing usb-kernel-recover: ${wedged.join(', ')}`) // Workers cannot prompt the user — surface still-locked boards for the main // session to ask: force (re-invoke with force: true), wait, or accept. if (locked.length) log(`still locked after retry: ${locked.join(', ')} — ask the user: force / keep waiting / accept`) -return { pass, results, wedged, locked } +return { pass, results, wedged, locked, caveat: runCaveat } diff --git a/.claude/workflows/test-hil-validate.mjs b/.claude/workflows/test-hil-validate.mjs index db73095f6..e8be57cfc 100644 --- a/.claude/workflows/test-hil-validate.mjs +++ b/.claude/workflows/test-hil-validate.mjs @@ -4,7 +4,7 @@ // `board locked` out of a prose detail, folding rows, keeping a wedged flag alive -- produced // a defect in each of four review rounds, including a test that asserted an invariant using // the one input shape that could not break it. That logic now lives in -// test/hil/helper/hil_summary.py, where the roster is, and arrives here as fields. What is +// test/hil/helper/hil_report.py, where the roster is, and arrives here as fields. What is // left is a lookup and a verdict, and this pins both. // // Run: node .claude/workflows/test-hil-validate.mjs @@ -23,6 +23,11 @@ const cut = (start, end) => { } const body = cut('const byBoard =', 'const first = await runBoards') + cut('const summarize =', 'const { pass, wedged, locked } =') +// more than one schema declares `required:`; pick the HIL one by its contents +const HIL_REQUIRED = (src.match(/required: \[[^\]]*\]/g) || []) + .map((m) => m.replace('required: ', '').replace(/'/g, '"')) + .map((m) => JSON.parse(m)) + .find((a) => a.includes('wedged')) || [] const { byBoard, summarize, wedgedFor } = new Function(`${body}; return { byBoard, summarize, wedgedFor }`)() let failed = 0 @@ -65,5 +70,27 @@ check('wedged surfaces', summarize([R('a', false, false, true)], false).wedged, check('a wedged board that passed still surfaces', summarize([R('a', true, false, true)], false).wedged, ['a']) +// A run-level caveat outranks per-row agreement: on the abandon and no-boards paths every +// row can legitimately pass while hil_test.py exits non-zero. Row agreement alone published +// those runs green. +check('all rows pass and no caveat is a pass', + summarize([R('a', true), R('b', true)], false, '').pass, true) +check('an abandoned run is not a pass', + summarize([R('a', true)], false, + '**HIL run abandoned: the worker pool would not shut down.** x').pass, false) +check('an aborted run is not a pass', + summarize([R('a', true)], false, '**HIL run aborted: a worker raised RuntimeError**').pass, + false) +check('a no-boards run is not a pass', + summarize([R('a', true)], false, '**HIL run selected no boards.** filters emptied').pass, + false) +check('a rig-health note is NOT a caveat and does not fail the run', + summarize([R('a', true)], false, '> **Rig note.** 2 process(es) in D state').pass, true) +check('a retry that abandoned sinks the run even with all rows passing', + summarize([R('a', true)], false, + '**HIL run abandoned: the worker pool would not shut down.** retry').pass, false) +check('an omitted caveat cannot silently disable the gate (schema requires it)', + HIL_REQUIRED.includes('caveat'), true) + console.log(failed ? `\n${failed} FAILED` : '\nall checks passed') process.exit(failed ? 1 : 0) diff --git a/docs/superpowers/followup/pr3803-hil-blindness-reporting.md b/docs/superpowers/followup/pr3803-hil-blindness-reporting.md index 69ff939b0..374ee62c7 100644 --- a/docs/superpowers/followup/pr3803-hil-blindness-reporting.md +++ b/docs/superpowers/followup/pr3803-hil-blindness-reporting.md @@ -158,7 +158,8 @@ def _board_result_on_error(name, exc): """A row for a board that died by exception. err_count 1, no per-test detail, but the blindness and stray fields are still accurate -- they explain the failure more often than the exception text does.""" - rows = [(name, {BOUNDARY_CELL: f'{REPORT_CELL["fail"]} {type(exc).__name__}'}, None)] + rows = [(name, {hil_report.BOUNDARY_CELL: + f'{hil_report.REPORT_CELL["fail"]} {type(exc).__name__}'}, None)] return _board_result(name, 1, [], rows, 0.0, True) ``` diff --git a/docs/superpowers/followup/pr3836-report-single-source.md b/docs/superpowers/followup/pr3836-report-single-source.md deleted file mode 100644 index f4ccc77c2..000000000 --- a/docs/superpowers/followup/pr3836-report-single-source.md +++ /dev/null @@ -1,470 +0,0 @@ -# One Source of Truth for the HIL Report Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make `hil_report.md` a rendering of `hil_report.json` rather than a second, independently written artifact, so no run can produce a table whose contents are not in the JSON. - -**Architecture:** `hil_report.json` gains the two fields the markdown carries but the JSON does not (`scope`, and a `caveat` for text prepended after the fact). A single `render_report(doc) -> str` turns that document into the markdown, and every writer — the normal path, the pool-guard fallback, the no-boards exit, and `_abandon_exit` — goes through `write_report(report_dir, doc)`, which writes both files from the same dict. `_abandon_exit` stops doing a text-prepend on a file it did not write and instead sets `doc['caveat']`. - -**Tech Stack:** Python 3.13 stdlib only (`json`, `pathlib`); existing unit suites under `test/hil/test/` run with plain `unittest`. - -**Spec:** none — this is a follow-up split out of the `claude/hil-doc-audit` branch. The evidence it argues from is inline below. - -**Origin:** split out of PR #3836 (the HIL one-run rework + `.claude` instruction audit). Delete this file when its own PR lands. - -## Global Constraints - -- **No behaviour change to the containment paths' ordering or exit codes.** `_abandon_exit` runs while the interpreter is being torn down; its own comments record that anything raising between the pool's `finally` and `os._exit` hangs the process in multiprocessing's unbounded `join()` (reproduced at rc=124/25s with SIGTERM-ignoring workers). Serialisation added there must stay inside the existing `try`/`except` and must never raise past it. -- **The markdown stays the human artifact.** `.github/workflows/build.yml:487` uploads `hil_report.md`, `test/hil/hil_ci.sh:293` copies only it back, and `.claude/skills/hil/SKILL.md` tells the operator to paste that table verbatim. It becomes generated output, not a dropped file. -- **Banner outranks the scope note outranks the table.** Preserve the existing order (`hil_test.py:2153-2161`): the caveat is outermost because that is where `hil/SKILL.md` tells an agent to look. -- **`--accumulate` merges from the JSON** (`hil_test.py:2102-2115`), including carrying the prior banner forward. Adding fields must not break that merge for a sidecar written by an older version. -- Run `python3 -m unittest discover -s test/hil/test` (115 tests, ~78 s) before each commit; `pre-commit run --files ` before pushing. - -## Why this is worth doing - -Four writers produce `hil_report.md`, and three of them write no JSON at all: - -| Writer | JSON? | Line | -|---|---|---| -| `accumulate_report` — the normal path | yes | `hil_test.py:2149`, `:2162` | -| `**HIL run selected no boards.**` | **no** | `hil_test.py:2317` | -| pool-guard fallback → `hil_health.write_timeout_report(...)` | **no** | `hil_test.py:2469`, `hil_health.py:346` | -| `_abandon_exit` — prepends to whatever `.md` exists | **no** | `hil_test.py:2603` | - -Those three are exactly the paths where the run died, so they are the cases where the artifact matters most and where a JSON consumer sees nothing. `test/hil/helper/hil_summary.py` (added on the origin branch) reads the JSON to build the per-board verdicts an agent hands back — on any of those three paths it finds no file and reports "no report row for this board" for the whole fleet, while a human reading the markdown sees the real story. - -Separately, `scope` exists only in the markdown (`hil_test.py:2154`, from `accumulate_report`'s `scope: str = ''` parameter at `:2092`). A PR-scoped three-board table and a full-fleet run that lost 24 boards are indistinguishable in the JSON. - ---- - -### Task 1: Put `scope` in the JSON - -**Files:** -- Modify: `test/hil/hil_test.py:2092-2163` (`accumulate_report`) -- Test: `test/hil/test/test_hil_bounded.py` (new class beside `CaveatSurvivesAccumulate`) - -**Interfaces:** -- Produces: `hil_report.json` gains a top-level `"scope": str` (empty string when unscoped). Existing keys `rows` and `banner` are unchanged. - -- [ ] **Step 1: Write the failing test** - -```python -class ScopeSurvivesInTheJson(unittest.TestCase): - """A scoped run's small table is indistinguishable from a full run that lost boards. - The markdown says so; the JSON did not, so any JSON consumer could not tell.""" - - def _rows(self, board, cell): - return [(board, 0, 0, [(board, {cell: 'OK'}, '1s')], 0)] - - def test_scope_is_recorded_in_the_sidecar(self): - import json - td = TemporaryDirectory() - self.addCleanup(td.cleanup) - rd = Path(td.name) - hil_test.accumulate_report(self._rows('boardA', 'cdc_msc'), rd, True, - '-b boardA', '') - doc = json.loads((rd / 'hil_report.json').read_text()) - self.assertEqual(doc['scope'], '-b boardA') - - def test_an_unscoped_run_records_an_empty_scope(self): - import json - td = TemporaryDirectory() - self.addCleanup(td.cleanup) - rd = Path(td.name) - hil_test.accumulate_report(self._rows('boardA', 'cdc_msc'), rd, True, '', '') - self.assertEqual(json.loads((rd / 'hil_report.json').read_text())['scope'], '') -``` - -- [ ] **Step 2: Run it to verify it fails** - -Run: `python3 test/hil/test/test_hil_bounded.py ScopeSurvivesInTheJson` -Expected: FAIL — `KeyError: 'scope'` - -- [ ] **Step 3: Add the field** - -In `accumulate_report`, change the `jpath.write_text(...)` call at `hil_test.py:2149`: - -```python - jpath.write_text(json.dumps({'rows': [{'board': k, 'cells': c, 'duration': d} - for k, (c, d) in acc.items()], - 'banner': banner, - 'scope': scope}, indent=2) + '\n') -``` - -- [ ] **Step 4: Run the tests** - -Run: `python3 test/hil/test/test_hil_bounded.py ScopeSurvivesInTheJson` → PASS -Run: `python3 -m unittest discover -s test/hil/test` → 117 tests OK (the merge at `:2102` reads only `rows` and `banner`, so an older sidecar without `scope` still loads). - -- [ ] **Step 5: Commit** - -```bash -git add test/hil/hil_test.py test/hil/test/test_hil_bounded.py -git commit -m "hil_test: record the run's scope in hil_report.json - -The markdown says a scoped table is scoped; the JSON did not, so a consumer -could not tell a three-board PR run from a full run that lost 24 boards." -``` - ---- - -### Task 2: Render the markdown from the document - -**Files:** -- Modify: `test/hil/hil_test.py:1921` (`render_matrix`), `:2149-2163` (`accumulate_report`'s tail) -- Test: `test/hil/test/test_hil_bounded.py` - -**Interfaces:** -- Consumes: the `scope` key from Task 1. -- Produces: `render_report(doc: dict) -> str`, where `doc` is `{'rows': [{'board','cells','duration'}], 'banner': str, 'scope': str, 'caveat': str}`. `caveat` is optional and empty by default (Task 4 sets it). Order is caveat, banner, scope note, table. - -- [ ] **Step 1: Write the failing test** - -```python -class RenderReportIsPureFunctionOfTheDocument(unittest.TestCase): - def _doc(self, **kw): - d = {'rows': [{'board': 'boardA', 'cells': {'cdc_msc': 'pass'}, 'duration': '1s'}], - 'banner': '', 'scope': '', 'caveat': ''} - d.update(kw) - return d - - def test_table_comes_from_rows(self): - md = hil_test.render_report(self._doc()) - self.assertIn('boardA', md) - self.assertIn('cdc_msc', md) - - def test_scope_note_appears_above_the_table(self): - md = hil_test.render_report(self._doc(scope='-b boardA')) - self.assertLess(md.index('Scoped run'), md.index('boardA')) - - def test_banner_outranks_the_scope_note(self): - md = hil_test.render_report(self._doc(scope='-b boardA', - banner='> **Rig dirty.** x\n')) - self.assertLess(md.index('Rig dirty'), md.index('Scoped run')) - - def test_caveat_is_outermost(self): - md = hil_test.render_report(self._doc(banner='> **Rig dirty.** x\n', - caveat='**HIL run abandoned.**\n')) - self.assertLess(md.index('abandoned'), md.index('Rig dirty')) - - def test_a_document_with_no_rows_still_renders(self): - md = hil_test.render_report(self._doc(rows=[])) - self.assertIn('No tests were run.', md) -``` - -- [ ] **Step 2: Run it to verify it fails** - -Run: `python3 test/hil/test/test_hil_bounded.py RenderReportIsPureFunctionOfTheDocument` -Expected: FAIL — `AttributeError: module 'hil_test' has no attribute 'render_report'` - -- [ ] **Step 3: Add `render_report` and route `accumulate_report` through it** - -Add beside `render_matrix` (after `hil_test.py:1919`): - -```python -def render_report(doc: dict) -> str: - """The markdown IS a rendering of the sidecar. Every writer goes through here, so a - table can never contain something the JSON does not.""" - md = render_matrix([(r['board'], r['cells'], r.get('duration')) - for r in doc.get('rows', [])]) - if doc.get('scope'): - # a scoped run's small table is otherwise indistinguishable from a full one, and - # it replaces the previous full table in the sticky PR comment - md = f'_Scoped run: {doc["scope"]}. Boards/tests not listed were not run._\n\n' + md - # banner, then caveat: a rig-health caveat outranks the table AND the scope note, and an - # abandon notice outranks even that -- the top of the report is where hil/SKILL.md tells - # the agent to look - if doc.get('banner'): - md = doc['banner'] + '\n' + md - if doc.get('caveat'): - md = doc['caveat'] + '\n' + md - return md -``` - -Then replace `accumulate_report`'s tail (`hil_test.py:2153-2163`) with: - -```python - doc = {'rows': [{'board': k, 'cells': c, 'duration': d} for k, (c, d) in acc.items()], - 'banner': banner, 'scope': scope, 'caveat': ''} - jpath.write_text(json.dumps(doc, indent=2) + '\n') - md = render_report(doc) - (report_dir / REPORT_MD).write_text(md + '\n', encoding='utf-8') - return md -``` - -- [ ] **Step 4: Run the tests** - -Run: `python3 -m unittest discover -s test/hil/test` -Expected: 122 OK. `CaveatSurvivesAccumulate` must still pass — it asserts the banner survives a rerun, which is now the `banner` key round-tripping through the document. - -- [ ] **Step 5: Commit** - -```bash -git add test/hil/hil_test.py test/hil/test/test_hil_bounded.py -git commit -m "hil_test: render the markdown from the report document - -One function turns the sidecar into the table, so the markdown cannot carry -anything the JSON lacks. Ordering (caveat > banner > scope > table) is pinned -by tests rather than by the order of three string concatenations." -``` - ---- - -### Task 3: Give the two early-exit paths a document - -**Files:** -- Modify: `test/hil/hil_test.py:2313-2320` (no-boards exit), `test/hil/helper/hil_health.py:346` (`write_timeout_report`) -- Test: `test/hil/test/test_hil_health.py` (beside `WriteTimeoutReport`), `test/hil/test/test_hil_bounded.py` - -**Interfaces:** -- Consumes: `render_report(doc)` from Task 2. -- Produces: `write_report(report_dir: Path, doc: dict) -> None`, which writes `hil_report.json` and `hil_report.md` from one dict. Both early-exit paths call it. - -- [ ] **Step 1: Write the failing test** - -```python -class EveryExitPathLeavesBothArtifacts(unittest.TestCase): - """hil_summary.py builds an agent's verdicts from the JSON. A path that writes only - markdown reports the whole fleet as 'no report row' while a human sees the real story.""" - - def test_the_no_boards_exit_writes_json_too(self): - import json - td = TemporaryDirectory() - self.addCleanup(td.cleanup) - rd = Path(td.name) - hil_test.write_report(rd, {'rows': [], 'banner': '', 'scope': '', - 'caveat': '**HIL run selected no boards.** why\n'}) - self.assertIn('selected no boards', (rd / 'hil_report.md').read_text()) - doc = json.loads((rd / 'hil_report.json').read_text()) - self.assertEqual(doc['rows'], []) - self.assertIn('selected no boards', doc['caveat']) -``` - -and, in `test_hil_health.py`: - -```python - def test_timeout_report_writes_the_sidecar(self): - import json - td = TemporaryDirectory() - self.addCleanup(td.cleanup) - rd = Path(td.name) - hil_health.write_timeout_report(rd, [{'name': 'boardA'}], 3600, 'hil_report.md') - self.assertTrue((rd / 'hil_report.json').is_file()) - self.assertIn('boardA', (rd / 'hil_report.json').read_text()) -``` - -- [ ] **Step 2: Run them to verify they fail** - -Run: `python3 test/hil/test/test_hil_bounded.py EveryExitPathLeavesBothArtifacts` -Expected: FAIL — `AttributeError: module 'hil_test' has no attribute 'write_report'` -Run: `python3 test/hil/test/test_hil_health.py WriteTimeoutReport` -Expected: FAIL — `hil_report.json` is not a file - -- [ ] **Step 3: Add `write_report` and use it in both paths** - -Beside `render_report`: - -```python -def write_report(report_dir: Path, doc: dict) -> None: - """Write both artifacts from one document. Best-effort by design: every caller is on a - failure path where an OSError must not replace the failure being reported.""" - try: - report_dir.mkdir(parents=True, exist_ok=True) - (report_dir / REPORT_JSON).write_text(json.dumps(doc, indent=2) + '\n') - (report_dir / REPORT_MD).write_text(render_report(doc) + '\n', encoding='utf-8') - except OSError: - pass -``` - -Replace the no-boards block at `hil_test.py:2315-2320` with: - -```python - rd = Path(os.environ.get('HIL_REPORT_DIR', '.')) - write_report(rd, {'rows': [], 'banner': '', 'scope': '', - 'caveat': f'**HIL run selected no boards.** {msg}\n'}) -``` - -In `hil_health.write_timeout_report`, after the markdown is composed, write the sidecar next to it with a row per stuck board: - -```python - json_path = report_dir / 'hil_report.json' - json_path.write_text(json.dumps( - {'rows': [{'board': b['name'], 'cells': {'pool-timeout': 'fail'}, - 'duration': None} for b in boards], - 'banner': banner, 'scope': '', 'caveat': prefix}, indent=2) + '\n') -``` - -Keep it inside the function's existing broad `try` — a roster entry without `name` must not escape, which is what that handler exists to prevent. - -- [ ] **Step 4: Run the tests** - -Run: `python3 -m unittest discover -s test/hil/test` -Expected: 124 OK. - -- [ ] **Step 5: Commit** - -```bash -git add test/hil/hil_test.py test/hil/helper/hil_health.py test/hil/test/ -git commit -m "hil_test, hil_health: write the sidecar on the early-exit paths too - -The no-boards exit and the pool-guard fallback wrote markdown only, so a JSON -consumer saw nothing on exactly the runs that failed. hil_summary.py reported -the whole fleet as 'no report row' while the markdown told the real story." -``` - ---- - -### Task 4: Make `_abandon_exit` set a field instead of prepending text - -**Files:** -- Modify: `test/hil/hil_test.py` (`_abandon_exit`, the `if report is not None:` block near `:2622`), and its call site at `:2603` -- Test: `test/hil/test/test_hil_bounded.py` - -**Interfaces:** -- Consumes: `write_report`/`render_report` from Tasks 2–3. -- Produces: `_abandon_exit(pool, mgr, abandoned, err_count, report_dir: Path | None = None)` — the parameter becomes the **directory**, not the markdown path. - -- [ ] **Step 1: Write the failing test** - -```python -class AbandonNoticeLandsInBothArtifacts(unittest.TestCase): - def test_abandon_sets_the_caveat_not_just_the_markdown(self): - import json - td = TemporaryDirectory() - self.addCleanup(td.cleanup) - rd = Path(td.name) - hil_test.accumulate_report( - [('boardA', 0, 0, [('boardA', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') - hil_test.mark_report_abandoned(rd, 'the worker pool would not shut down.') - doc = json.loads((rd / 'hil_report.json').read_text()) - self.assertIn('abandoned', doc['caveat']) - self.assertEqual(len(doc['rows']), 1, 'the finished board must survive') - md = (rd / 'hil_report.md').read_text() - self.assertLess(md.index('abandoned'), md.index('boardA')) - - def test_marking_a_missing_report_is_a_no_op(self): - td = TemporaryDirectory() - self.addCleanup(td.cleanup) - hil_test.mark_report_abandoned(Path(td.name), 'x') # must not raise -``` - -- [ ] **Step 2: Run it to verify it fails** - -Run: `python3 test/hil/test/test_hil_bounded.py AbandonNoticeLandsInBothArtifacts` -Expected: FAIL — `AttributeError: module 'hil_test' has no attribute 'mark_report_abandoned'` - -- [ ] **Step 3: Implement it** - -```python -def mark_report_abandoned(report_dir: Path, why: str) -> None: - """Stamp an existing report as abandoned, in BOTH artifacts. - - Best-effort and silent: this runs while the interpreter is being torn down, and an - exception here hangs the process in multiprocessing's unbounded join().""" - try: - jpath = report_dir / REPORT_JSON - doc = json.loads(jpath.read_text()) if jpath.is_file() else None - if doc is None: - return - doc['caveat'] = (f'**HIL run abandoned: {why}** The table below is this run\'s ' - f'partial result.\n') - write_report(report_dir, doc) - except (OSError, ValueError, TypeError): - pass -``` - -Then in `_abandon_exit`, replace the read-modify-write of the markdown with `mark_report_abandoned(report, ...)` and change the call site at `:2603` from `report_dir / REPORT_MD` to `report_dir`. - -- [ ] **Step 4: Run the tests** - -Run: `python3 -m unittest discover -s test/hil/test` -Expected: 126 OK. - -- [ ] **Step 5: Commit** - -```bash -git add test/hil/hil_test.py test/hil/test/test_hil_bounded.py -git commit -m "hil_test: stamp abandonment into the document, not onto the markdown - -_abandon_exit did a text prepend on a file it had not written, so the caveat -never reached the JSON and an agent reading the sidecar saw a clean partial -report under a red job. Still best-effort and still silent: it runs while the -interpreter is being torn down." -``` - ---- - -### Task 5: Prove the two artifacts cannot disagree - -**Files:** -- Test: `test/hil/test/test_hil_bounded.py` - -- [ ] **Step 1: Write the test** - -```python -class MarkdownIsAlwaysARenderingOfTheJson(unittest.TestCase): - """The property this whole change buys: whatever wrote the report, re-rendering the - sidecar reproduces the markdown byte for byte.""" - - def _check(self, rd): - import json - doc = json.loads((rd / 'hil_report.json').read_text()) - self.assertEqual((rd / 'hil_report.md').read_text(), - hil_test.render_report(doc) + '\n') - - def test_normal_path(self): - td = TemporaryDirectory(); self.addCleanup(td.cleanup); rd = Path(td.name) - hil_test.accumulate_report( - [('boardA', 0, 0, [('boardA', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, - '-b boardA', '> **Rig note.** x\n') - self._check(rd) - - def test_after_an_accumulate_rerun(self): - td = TemporaryDirectory(); self.addCleanup(td.cleanup); rd = Path(td.name) - hil_test.accumulate_report( - [('boardA', 0, 0, [('boardA', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') - hil_test.accumulate_report( - [('boardB', 0, 0, [('boardB', {'cdc_msc': 'OK'}, '1s')], 0)], rd, False, '', '') - self._check(rd) - - def test_after_abandonment(self): - td = TemporaryDirectory(); self.addCleanup(td.cleanup); rd = Path(td.name) - hil_test.accumulate_report( - [('boardA', 0, 0, [('boardA', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') - hil_test.mark_report_abandoned(rd, 'the worker pool would not shut down.') - self._check(rd) - - def test_no_boards_exit(self): - td = TemporaryDirectory(); self.addCleanup(td.cleanup); rd = Path(td.name) - hil_test.write_report(rd, {'rows': [], 'banner': '', 'scope': '', - 'caveat': '**HIL run selected no boards.** why\n'}) - self._check(rd) -``` - -- [ ] **Step 2: Run it** - -Run: `python3 test/hil/test/test_hil_bounded.py MarkdownIsAlwaysARenderingOfTheJson` -Expected: PASS on all four. A failure here means a writer still bypasses `render_report`. - -- [ ] **Step 3: Full gate and commit** - -```bash -python3 -m unittest discover -s test/hil/test # 130 OK -pre-commit run --files test/hil/hil_test.py test/hil/helper/hil_health.py \ - test/hil/test/test_hil_bounded.py test/hil/test/test_hil_health.py -git add test/hil/test/test_hil_bounded.py -git commit -m "test/hil: pin that the markdown is always a rendering of the sidecar - -Four writers, one renderer. This is the invariant the change exists to create, -so it is asserted directly rather than inferred from the writers." -``` - ---- - -## Out of scope - -Deliberately not included, each its own follow-up: - -- **The flat `HIL_POOL_TIMEOUT`.** `hil_test.py:225` is a per-process 3600 s guard that does not scale with board count. It was per board when runs were serial; the origin branch made one run cover the fleet, so a 27-board run shares one budget. Real, and a scheduling change rather than a reporting one. -- **`hil_ci.sh` accumulate in remote mode.** `hil_ci.sh:183` `rm -rf`s `REMOTE_DIR` every run and the copies are one-way, so a remote `--accumulate` retry has no merge base and its one-row report overwrites the local full-fleet one. Fixing that means uploading `hil_report.json` and `.failed` before the run, or keeping `REMOTE_DIR` when `--accumulate` is present. -- **Dropping `hil_report.md` entirely.** Not proposed. It is the PR artifact and what the `hil` skill tells operators to paste; this plan makes it generated, not redundant. diff --git a/docs/superpowers/followup/pr3840-mret-board-result.md b/docs/superpowers/followup/pr3840-mret-board-result.md new file mode 100644 index 000000000..7b8da7b9c --- /dev/null +++ b/docs/superpowers/followup/pr3840-mret-board-result.md @@ -0,0 +1,90 @@ +# Give the HIL worker result a name + +**Origin:** split out of PR #3840 (making `hil_report.md` a rendering of `hil_report.json`). +Delete this file when its own PR lands. + +## What is established + +`test_board()` returns a bare tuple that three producers build and fourteen call sites read +positionally. It has grown 5 → 6 → 7 fields, and the code already works around its own +shape: + +```python +hil_test.py:1992 dirty = [(r[0], r[6]) for r in mret if len(r) > 6 and r[6]] +hil_test.py:2014 blind = [r[0] for r in mret if len(r) > 5 and r[5]] +hil_test.py:2386 for name, _, _, _, dur, *_ in mret: +hil_report.py:306 for name, _, _, rows, *_ in mret: +``` + +Two facts make this worth closing rather than tolerating: + +- **The declared type is already wrong.** `hil_test.py:1711` says + `tuple[str, int, list[str], list, float]` — five fields — while the main return at `:1872` + yields seven (`+ sysfs_blind(), stray`). +- **A wrong slot is a wrong verdict, not a crash.** Field 5 is `blind`, which decides whether + a board's red cells are reported as broken hardware or as "could not tell". Inserting a + field mid-tuple makes `r[5]` read the wrong slot and keep running. + +It has bitten once already: `test_hil_bounded.py`'s +`test_both_row_widths_survive_the_report_writers` exists because the blindness flag widened +the tuple to 6 while the pool-timeout path still synthesised 5-field rows, and *"a +fixed-width unpack in either one raises INSIDE the containment path, which is where a raise +costs every board's results."* That is why the unpacks end in `*_`. + +## What remains + +A `NamedTuple` with defaults. Verified to pickle across the pool boundary and to stay +fully tuple-compatible — existing `r[0]`, `e[1]`, `for name, _, _, rows, *_` and `len(r)` +all keep working, so it lands without touching the fourteen consumers: + +```python +class BoardResult(NamedTuple): + """What one worker returns. Field ORDER is load-bearing: it is unpacked positionally + in a dozen places, and the pool-timeout path synthesises one by hand.""" + name: str + err_count: int + failed_tests: list[str] + rows: list | None # None from the pool-timeout synthesis, never [] + duration: float + blind: bool = False # defaults, so a synthesised result is full-width + stray: int = 0 +``` + +Then a second, smaller step removes the coupling itself: `accumulate_report` takes +`[(name, rows)]` pairs instead of `mret`, and `hil_test` does the extraction because it owns +the shape. One line at each end; the subtle merge logic — stale lock clearing, +`BOUNDARY_CELL`, `duration=None` preservation — is untouched. + +## Sizing + +| | Sites | +|---|---| +| Producers to convert | 4 (`hil_test.py:1724`, `:1872`, `:2283`, `:2327`) | +| Arity guards deleted | 2 (`:1992`, `:2014`) | +| Wrong annotation fixed | 1 (`:1711`) | +| `hil_report`'s coupled line | 1 (`:306`) | +| Positional consumers (optional migration) | 14 | +| **Test fixtures building tuples by hand** | **34** | + +Production code is roughly ten changed lines. **The work is dominated by the test +fixtures**, which is also the risk. + +## Do this first, or the refactor is unverifiable + +`test_hil_report.py` (27 sites) and `test_hil_bounded.py` (7) construct plain tuples by +hand — `('boardA', 0, [], [], 1.0, True)`. A producer that forgot to switch to +`BoardResult`, or a pickling regression, **passes the entire 310-test suite** and surfaces +only on the rig. Convert the fixtures to build `BoardResult` as task 1, before touching any +producer. This ordering is not optional. + +Second trap: `rows` is `None` on the pool-timeout path (`hil_test.py:2283`), never `[]`, and +`accumulate_report` guards with `if rows and ...`. A well-meaning `rows: list = []` default +silently changes that path. Pin it with a test before the conversion. + +## Why it was split out + +PR #3840 touches the report document. This touches `test_board`'s return and the containment +paths, where a raise costs every board's results rather than one board's — a different blast +radius, needing its own review and its own rig run. #3840 is twice-reviewed and dogfooded +ten times on hardware; folding this in would reset that surface for a latent-trap cleanup +that is not causing bugs today. diff --git a/docs/superpowers/followup/pr3840-skill-md-no-boards-drift.md b/docs/superpowers/followup/pr3840-skill-md-no-boards-drift.md new file mode 100644 index 000000000..a039a8c12 --- /dev/null +++ b/docs/superpowers/followup/pr3840-skill-md-no-boards-drift.md @@ -0,0 +1,38 @@ +# `SKILL.md` contradicts the code on no-boards tables + +**Origin:** split out of PR #3840, surfaced by its second review round. Delete this file +when its own PR lands. + +`.claude/skills/hil/SKILL.md:150-151` tells the reading agent: + +> `**HIL run selected no boards.**` — the filters intersected to nothing, so there is **no +> table at all**. Report that (and the filter shown), never `"pass": true`. + +That was true when the no-boards exit wrote a bare notice. It no longer is. An +`--accumulate` no-boards run keeps the accumulated rows — deliberately, because wiping them +destroyed real results — so the artifact now reads: + +``` +**HIL run selected no boards.** filters emptied + +**✅ 1 passed · ❌ 0 failed · ⚪ 0 skipped · blank not run** + +| Board | t | duration | +... +``` + +The behaviour is correct; the documentation is wrong, and wrong in the direction that +matters. An agent is told to expect no table, sees one, and has no rule for whether those +rows are reportable. **They are not this run's** — they are a previous attempt's, carried +forward. + +**What remains:** update that bullet to describe both cases — a fresh run has no table, an +`--accumulate` run shows the previous attempt's rows under the notice and they must not be +reported as this run's. Add a test asserting the fresh case renders no matrix, so the two +halves cannot drift again. + +## Why it was split out + +PR #3840 fixed the findings that changed a verdict. This is a documentation drift: the +behaviour is correct and the doc describing it is not, so it is better reviewed on its own +than appended to a branch already carrying a module consolidation. diff --git a/docs/superpowers/followup/pr3840-write-report-atomicity.md b/docs/superpowers/followup/pr3840-write-report-atomicity.md new file mode 100644 index 000000000..2094207bb --- /dev/null +++ b/docs/superpowers/followup/pr3840-write-report-atomicity.md @@ -0,0 +1,30 @@ +# `write_report` commits the two artifacts non-atomically + +**Origin:** split out of PR #3840, surfaced by its second review round. Delete this file +when its own PR lands. + +```python +md = render_report(doc) + '\n' +report_dir.mkdir(parents=True, exist_ok=True) +(report_dir / REPORT_JSON).write_text(json.dumps(doc, indent=2) + '\n') +(report_dir / REPORT_MD).write_text(md, encoding='utf-8') +``` + +Rendering before writing closed the *render-failure* case: a raise can no longer commit a +sidecar the markdown contradicts. It does not close the *interrupted-between-writes* case. A +kill between those two lines leaves the pair disagreeing — and this runs on the containment +path, on the way to `os._exit`, on a rig whose jobs get cancelled by the GitHub job ceiling. + +**What remains:** write both to temp files, then `os.replace` both. The window shrinks from +two full writes to two renames, and neither file is ever observed half-written. `os.replace` +is atomic per file on POSIX; the pair is still not transactional, which is acceptable and +should be said in the docstring rather than implied away. + +Worth pairing with a test that kills between the writes — or, more practically, one that +asserts no partial file is ever visible by checking the temp-then-rename shape directly. + +## Why it was split out + +A durability edge, not a wrong verdict. PR #3840 closed the render-failure half of this +(nothing is written until the markdown renders); the interrupted-between-writes half needs +a temp-then-rename and is better reviewed on its own. diff --git a/docs/superpowers/plans/2026-08-21-hil-report-module.md b/docs/superpowers/plans/2026-08-21-hil-report-module.md new file mode 100644 index 000000000..5a7c832d8 --- /dev/null +++ b/docs/superpowers/plans/2026-08-21-hil-report-module.md @@ -0,0 +1,658 @@ +# hil_report.py Module Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Fold every function that produces, renders, merges or reads `hil_report.json`/`hil_report.md` into one module, `test/hil/helper/hil_report.py`, and take the two fixes that consolidation enables. + +**Architecture:** A new leaf-ish module owns the report document. `hil_test.py` and `hil_health.py` both import it, which dissolves the circular-import constraint that forced `write_timeout_report` to compose its own markdown. The duplicated cell classifier (`cell_kind` in `hil_test`, `cell_state` in `hil_summary`) collapses into one. `hil_summary.py` is deleted and its CLI moves in. + +**Tech Stack:** Python 3.13 stdlib only (`json`, `argparse`, `pathlib`); existing unit suites under `test/hil/test/` run with plain `unittest`. + +**Spec:** `docs/superpowers/specs/2026-08-21-hil-report-module-design.md` + +## Global Constraints + +- **Behaviour-preserving motion.** `hil_test.py`'s CLI, arguments, output and report format stay byte-identical. The two intended exceptions are named in the spec: the `hil_summary.py` → `hil_report.py` CLI path, and `write_timeout_report` rendering instead of concatenating. +- **`hil_report.py` must work in two modes.** It is imported as `helper.hil_report` by `hil_test.py`, and run as a script by the operator (`python3 test/hil/helper/hil_report.py -b BOARD`). A script run puts `test/hil/helper/` on `sys.path`, *not* `test/hil/`, so `from helper import hil_health` fails in that mode. Task 1 pins both modes with tests. +- **Containment paths must never raise.** `mark_report_abandoned` and `write_timeout_report` run while the interpreter is being torn down or on the way to `os._exit`; anything escaping hangs the process in multiprocessing's unbounded `join()`. Their existing broad handlers move with them unchanged. +- **`hil_ci.sh` stages helpers by an explicit list** (`test/hil/hil_ci.sh:222-228`). A helper module missing from it reaches the rig absent, and the run dies with `ImportError` *after* `REMOTE_DIR` has been wiped. `RemoteStaging.test_import_closure_is_staged_to_the_rig` in `test_hil_bounded.py` already enforces this from the AST import closure; Task 1 only has to add the file to the list. +- Run `python3 -m unittest discover -s test/hil/test` (~82 s) before each commit; `pre-commit run --files ` before pushing. + +--- + +### Task 1: The module, the vocabulary, one classifier, and the render half + +**Files:** +- Create: `test/hil/helper/hil_report.py` +- Create: `test/hil/test/test_hil_report.py` +- Modify: `test/hil/hil_test.py:110` (`REPORT_CELL`), `:1715` (`BOUNDARY_CELL`), `:1902-1903` (`REPORT_MD`/`REPORT_JSON`), `:1921-1978` (`render_matrix`), `:1981-2003` (`render_report`), `:67` (imports) +- Modify: `test/hil/hil_ci.sh:222-228` (scp list) +- Modify: `test/hil/test/test_hil_bounded.py` (move `RenderReportIsPureFunctionOfTheDocument` out) + +**Interfaces:** +- Produces: `helper.hil_report` exposing `REPORT_MD`, `REPORT_JSON`, `REPORT_CELL`, `BOUNDARY_CELL`, `LOCKED_CELL`, `cell_state(v) -> str`, `render_matrix(rows_all) -> str`, `render_report(doc) -> str`. +- `hil_test.py` re-exports nothing: call sites become `hil_report.NAME`. + +- [ ] **Step 1: Write the failing tests** + +Create `test/hil/test/test_hil_report.py`: + +```python +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +# Unit tests for the report document: the vocabulary, the one cell classifier, rendering, +# the four writers, and the fold to per-board verdicts. Split out of test_hil_bounded.py +# and test_hil_health.py when the report code moved into helper/hil_report.py. +# Run directly: +# python3 test/hil/test/test_hil_report.py +import json +import os +import subprocess +import sys +import unittest +from pathlib import Path +from tempfile import TemporaryDirectory + +TEST_DIR = os.path.dirname(os.path.abspath(__file__)) +HIL_DIR = os.path.dirname(TEST_DIR) +sys.path.insert(0, HIL_DIR) + +from helper import hil_report + + +class OneClassifierForBothArtifacts(unittest.TestCase): + """The markdown tally and the agent's verdict used to classify cells with two separate + copies of one rule -- hil_test's cell_kind against REPORT_CELL, and hil_summary's + cell_state against its own re-typed '❌'/'⚪' literals. Change the icons and the table + and the verdict silently disagree.""" + + def test_bare_states(self): + self.assertEqual(hil_report.cell_state('fail'), 'fail') + self.assertEqual(hil_report.cell_state('skip'), 'skip') + self.assertEqual(hil_report.cell_state('pass'), 'pass') + + def test_icon_prefixed_metrics_carry_their_verdict(self): + self.assertEqual(hil_report.cell_state(f'{hil_report.REPORT_CELL["fail"]} 29/30'), 'fail') + self.assertEqual(hil_report.cell_state(f'{hil_report.REPORT_CELL["skip"]} board wedged'), + 'skip') + + def test_an_unprefixed_metric_is_a_pass(self): + """Load-bearing: a passing test may return a plain metric string. Classifying + unknown shapes as fail would publish a green table as a red verdict.""" + self.assertEqual(hil_report.cell_state('480.0 MBps'), 'pass') + self.assertEqual(hil_report.cell_state('1103 KB/s'), 'pass') + + def test_a_non_string_cell_does_not_raise(self): + """render_matrix's copy guarded with isinstance; hil_summary's did not, because its + caller str()'d first. The merged one keeps the guard -- it is the safer superset.""" + self.assertEqual(hil_report.cell_state(None), 'pass') + + def test_the_icons_come_from_REPORT_CELL(self): + """No second copy of the emoji anywhere in the module.""" + src = (Path(HIL_DIR) / 'helper' / 'hil_report.py').read_text(encoding='utf-8') + for icon in ('❌', '⚪', '✅'): + self.assertEqual(src.count(f"'{icon}'"), 1, + f'{icon} is spelled as a literal more than once') + + +class ModuleWorksImportedAndAsAScript(unittest.TestCase): + """It is imported as helper.hil_report by hil_test, and run as a script by the operator + (.claude/agents/hil-operator.md). A script run puts helper/ on sys.path, NOT test/hil, + so a plain `from helper import hil_health` breaks the CLI and only the CLI.""" + + def test_importable_as_a_package_module(self): + r = subprocess.run( + [sys.executable, '-c', + f'import sys; sys.path.insert(0, {HIL_DIR!r}); ' + f'from helper import hil_report; print(hil_report.REPORT_JSON)'], + capture_output=True, text=True, timeout=60) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn('hil_report.json', r.stdout) + + def test_runnable_as_a_script(self): + r = subprocess.run( + [sys.executable, str(Path(HIL_DIR) / 'helper' / 'hil_report.py'), '--help'], + capture_output=True, text=True, timeout=60) + self.assertEqual(r.returncode, 0, r.stderr) + + +class HilCiStagesEveryHelperTheRunImports(unittest.TestCase): + """hil_ci.sh copies helper modules by an EXPLICIT list. One missing module reaches the + rig absent and the run dies with ImportError -- after REMOTE_DIR has already been + rm -rf'd, so the previous run's report and re-run spec are gone too.""" + + def test_the_scp_list_covers_what_hil_test_imports(self): + sh = (Path(HIL_DIR) / 'hil_ci.sh').read_text(encoding='utf-8') + staged = {line.split('helper/')[1].rstrip('" \\\n') + for line in sh.splitlines() if '/test/hil/helper/' in line and '.py' in line} + imported = set() + for mod in (Path(HIL_DIR) / 'hil_test.py', Path(HIL_DIR) / 'helper' / 'hil_report.py'): + src = mod.read_text(encoding='utf-8') + for raw in src.splitlines(): + line = raw.strip() # hil_report's own import is indented in a try + if line.startswith('from helper import '): + imported |= {f'{n.strip()}.py' for n in line.split('import', 1)[1].split(',')} + elif line.startswith('from helper.'): + imported.add(line.split('.')[1].split(' ')[0] + '.py') + missing = imported - staged + self.assertEqual(missing, set(), + f'hil_ci.sh does not stage {missing}; a remote run will ImportError') + + +if __name__ == '__main__': + unittest.main() +``` + +Then **move** the class `RenderReportIsPureFunctionOfTheDocument` from `test/hil/test/test_hil_bounded.py` into this file verbatim, changing only `hil_test.render_report` → `hil_report.render_report` throughout. + +- [ ] **Step 2: Run them to verify they fail** + +Run: `python3 test/hil/test/test_hil_report.py` +Expected: FAIL — `ModuleNotFoundError: No module named 'helper.hil_report'` + +- [ ] **Step 3: Create the module** + +Create `test/hil/helper/hil_report.py`: + +```python +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +"""The HIL report document: one owner for hil_report.json and hil_report.md. + +The markdown IS a rendering of the sidecar -- every writer goes through render_report(), +so a table can never contain something the JSON does not. This module owns the whole life +of that document: the cell vocabulary, the one classifier both artifacts share, rendering, +the four writers, and the fold to one machine-readable verdict per board. + +Dual-mode by design: imported as `helper.hil_report` by hil_test.py, and run as a script by +the operator (see .claude/agents/hil-operator.md). A script run puts test/hil/helper on +sys.path rather than test/hil, hence the guarded hil_health import below. +""" +import argparse +import json +import sys +from pathlib import Path + +try: # imported as part of the helper package + from helper.hil_health import _p +except ImportError: # run as a script: helper/ is sys.path[0] + from hil_health import _p + +REPORT_MD = 'hil_report.md' +REPORT_JSON = 'hil_report.json' +# The status vocabulary, shared by the code that WRITES a cell (hil_test's test runners) and +# the code that reads one back (cell_state). One dict, so the human's table and the agent's +# verdict cannot drift apart. +REPORT_CELL = {'pass': '✅', 'fail': '❌', 'skip': '⚪'} +BOUNDARY_CELL = 'same-PID boundary' +LOCKED_CELL = 'board-locked' + + +def cell_state(v) -> str: + """'pass' | 'fail' | 'skip' for one report cell. + + THE classifier -- the markdown tally and the per-board verdict both call this, so they + cannot disagree. 'fail' or a ❌ prefix is a failure, 'skip' or a ⚪ prefix is a skip, and + EVERYTHING ELSE is a pass. That last arm is load-bearing: a passing test may return a + plain metric string ('480.0 MBps') that lands in the cell unprefixed, while failures are + guaranteed marked -- TestFail's docstring pins that its metric is icon-prefixed precisely + so render and tally treat it as a failure. Classifying unknown shapes as fail here would + publish a green table as a red verdict. + + isinstance-guarded: cells are usually str but a caller may hand over None or a number, + and .startswith on those raises inside a report writer that must not raise.""" + if v == 'fail' or (isinstance(v, str) and v.startswith(REPORT_CELL['fail'])): + return 'fail' + if v == 'skip' or (isinstance(v, str) and v.startswith(REPORT_CELL['skip'])): + return 'skip' + return 'pass' +``` + +Then move, verbatim, from `hil_test.py`: +- `render_matrix` (`hil_test.py:1921-1978`) — with one change: delete its nested `cell_kind` + definition and call the module-level `cell_state` instead. The line + `kinds = [cell_kind(v) for _, cells, _ in rows_all for v in cells.values()]` becomes + `kinds = [cell_state(v) for _, cells, _ in rows_all for v in cells.values()]`. +- `render_report` (`hil_test.py:1981-2003`) — unchanged. + +Add a placeholder CLI so `--help` works (Task 4 fills in `summarize`): + +```python +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument('config_file') + ap.add_argument('-b', '--board', action='append', default=[], + help='boards to report on; default: every board in the config') + ap.add_argument('--report-dir', default='.', help=f'where {REPORT_JSON} lives (default: cwd)') + ap.parse_args() + raise SystemExit('hil_report: summarize() lands in Task 4') + + +if __name__ == '__main__': + sys.exit(main()) +``` + +- [ ] **Step 4: Point `hil_test.py` at the module** + +In `hil_test.py:67`, extend the import: + +```python +from helper import hil_health, hil_lock, hil_report, hil_util +``` + +Delete `REPORT_CELL` (`:110`), `BOUNDARY_CELL` (`:1715`), `REPORT_MD`/`REPORT_JSON` +(`:1902-1903`), `render_matrix` and `render_report` from `hil_test.py`. Then rewrite every +reference to the moved names as `hil_report.`. Find them all with: + +```bash +grep -n "REPORT_CELL\|BOUNDARY_CELL\|REPORT_MD\|REPORT_JSON\|render_matrix\|render_report" \ + test/hil/hil_test.py +``` + +Known sites: `:876`, `:1369`, `:1459`, `:1490`, `:1492`, `:1508`, `:1818`, `:1834`, `:2162`, +`:2191-2192`, `:2209-2210`, `:2403`, `:2592`. + +- [ ] **Step 5: Stage the new module for remote runs** + +In `test/hil/hil_ci.sh:222-228`, add the module to the scp list (keep alphabetical-ish order +with the rest): + +```bash +scp -q "$ROOT_DIR/test/hil/helper/__init__.py" \ + "$ROOT_DIR/test/hil/helper/hil_util.py" \ + "$ROOT_DIR/test/hil/helper/hil_health.py" \ + "$ROOT_DIR/test/hil/helper/hil_lock.py" \ + "$ROOT_DIR/test/hil/helper/hil_report.py" \ + "$ROOT_DIR/test/hil/helper/hil_summary.py" \ + "$ROOT_DIR/test/hil/helper/hil_select.py" \ + "$REMOTE:$REMOTE_DIR/test/hil/helper/" +``` + +- [ ] **Step 6: Run the tests** + +Run: `python3 test/hil/test/test_hil_report.py` → OK +Run: `python3 -m unittest discover -s test/hil/test` → 274 OK (266 + 8 new: 5 classifier, +2 dual-mode, 1 scp guard; `RenderReport…` moves rather than adds) + +- [ ] **Step 7: Commit** + +```bash +git add test/hil/helper/hil_report.py test/hil/hil_test.py test/hil/hil_ci.sh \ + test/hil/test/test_hil_report.py test/hil/test/test_hil_bounded.py +git commit -m "hil_report: new module for the report vocabulary, classifier and rendering + +The markdown tally and the agent's verdict classified cells with two separate +copies of one rule, the second documented as 'the EXACT classifier hil_test.py's +own tally uses'. One cell_state now serves both, keyed off the one REPORT_CELL." +``` + +--- + +### Task 2: Move the three writers + +**Files:** +- Modify: `test/hil/helper/hil_report.py` (add the writers) +- Modify: `test/hil/hil_test.py:2005-2036` (`write_report`, `mark_report_abandoned`), `:2149-2212` (`accumulate_report`) +- Modify: `test/hil/test/test_hil_bounded.py` (move three classes out), `test/hil/test/test_hil_report.py` + +**Interfaces:** +- Consumes: `render_report`, `REPORT_MD`, `REPORT_JSON`, `BOUNDARY_CELL` from Task 1. +- Produces: `hil_report.write_report(report_dir, doc)`, `hil_report.mark_report_abandoned(report_dir, why)`, `hil_report.accumulate_report(mret, report_dir, fresh, scope='', banner='') -> str`. + +- [ ] **Step 1: Move the tests** + +Move these classes from `test/hil/test/test_hil_bounded.py` into `test/hil/test/test_hil_report.py`, +verbatim except `hil_test.` → `hil_report.` for the three moved functions: + +- `ScopeSurvivesInTheJson` +- `EveryExitPathLeavesBothArtifacts` +- `AbandonNoticeLandsInBothArtifacts` +- `CaveatSurvivesAccumulate` +- `MarkdownIsAlwaysARenderingOfTheJson` + +`AbandonNoticeLandsInBothArtifacts.test_an_existing_abandon_caveat_is_not_overwritten` calls +`hil_health.write_timeout_report`; leave that call as-is — Task 3 moves it. + +- [ ] **Step 2: Run them to verify they fail** + +Run: `python3 test/hil/test/test_hil_report.py` +Expected: FAIL — `AttributeError: module 'helper.hil_report' has no attribute 'write_report'` + +- [ ] **Step 3: Move the functions** + +Cut `write_report` (`hil_test.py:2005-2014`), `mark_report_abandoned` (`:2016-2036`) and +`accumulate_report` (`:2149-2212`) from `hil_test.py` and paste them into `hil_report.py` +below `render_report`, unchanged. + +Add to `accumulate_report`'s docstring, after the existing text, so the wart is recorded +where a reader meets it: + +``` + `mret` is hil_test.py's worker-result shape (name, err, fts, rows, ...), so this one + function knows something about its caller that the rest of the module does not. Folding + mret into rows could live in hil_test and only the merge here, but that would rewrite + the subtle parts -- stale board-locked clearing, BOUNDARY_CELL dropping, duration=None + preservation -- for a tidier seam. Data-shape coupling, not an import cycle. +``` + +- [ ] **Step 4: Update the call sites** + +In `hil_test.py`, the three call sites become `hil_report.*`: + +```bash +grep -n "accumulate_report(\|write_report(\|mark_report_abandoned(" test/hil/hil_test.py +``` + +Known sites: `:2260` (inside `_abandon_exit`), `:2351` (no-boards exit), `:2486`, `:2525`, +`:2618`. + +- [ ] **Step 5: Run the tests** + +Run: `python3 -m unittest discover -s test/hil/test` → 274 OK (motion only, no count change) + +- [ ] **Step 6: Commit** + +```bash +git add test/hil/helper/hil_report.py test/hil/hil_test.py \ + test/hil/test/test_hil_report.py test/hil/test/test_hil_bounded.py +git commit -m "hil_report: move the report writers off hil_test + +write_report, mark_report_abandoned and accumulate_report join the renderer they +already call. Pure motion; accumulate_report's knowledge of mret's tuple shape +moves with it and is now documented rather than implicit." +``` + +--- + +### Task 3: `write_timeout_report` renders like everyone else + +**Files:** +- Modify: `test/hil/helper/hil_report.py` (receive the function) +- Modify: `test/hil/helper/hil_health.py:347-398` (remove it), `:19` (drop `import json`) +- Modify: `test/hil/hil_test.py:2498` (call site) +- Modify: `test/hil/test/test_hil_health.py` (move `WriteTimeoutReport` out), `test/hil/test/test_hil_report.py` + +**Interfaces:** +- Consumes: `render_report`, `write_report` from Tasks 1-2. +- Produces: `hil_report.write_timeout_report(report_dir, boards, secs, banner='', prefix='')`. The `md_name` parameter is **gone** — the module owns `REPORT_MD`. + +- [ ] **Step 1: Write the failing tests** + +Move `WriteTimeoutReport` from `test/hil/test/test_hil_health.py` into +`test/hil/test/test_hil_report.py`, changing `hil_health.write_timeout_report` → +`hil_report.write_timeout_report` and dropping the `md_name` argument from every call. Two +of its tests change substantively: + +```python + def test_the_prior_attempts_rows_survive(self): + """Was: the prior MARKDOWN TEXT survives below the banner. It now re-renders from + the merged sidecar, so the guarantee is stated against rows -- one table with the + stuck boards in it, rather than a banner stapled above a duplicate table.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('done', 0, 0, [('done', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600) + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertEqual([r['board'] for r in doc['rows']], ['done', 'stuck']) + md = (rd / hil_report.REPORT_MD).read_text() + self.assertIn('done', md) + self.assertIn('stuck', md) + self.assertIn('abandoned', md) + self.assertLess(md.index('abandoned'), md.index('done')) + self.assertEqual(md.count('| Board'), 1, 'the prior table was duplicated, not merged') + + def test_prefix_carries_the_preflight_diagnosis(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.write_timeout_report(rd, [{'name': 'b1'}], 4200, + prefix='> **wedged usb_hub_wq worker.**\n') + out = (rd / hil_report.REPORT_MD).read_text() + self.assertTrue(out.startswith('> **wedged usb_hub_wq worker.**')) + self.assertIn('timed out after 4200s', out) + self.assertIn('b1', out) +``` + +And in `MarkdownIsAlwaysARenderingOfTheJson`, **delete** +`test_the_pool_guard_fallback_agrees_even_if_it_does_not_render` and add the fifth case in +its place: + +```python + def test_the_pool_guard_fallback(self): + """The last writer to join the invariant: it composed its own markdown only because + hil_health could not import the renderer.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('done', 0, 0, [('done', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600, + prefix='> **wedged usb_hub_wq worker.**\n') + self._check(rd) +``` + +- [ ] **Step 2: Run them to verify they fail** + +Run: `python3 test/hil/test/test_hil_report.py` +Expected: FAIL — `AttributeError: module 'helper.hil_report' has no attribute 'write_timeout_report'` + +- [ ] **Step 3: Move it and make it render** + +Add to `hil_report.py`, and delete `hil_health.py:347-398` plus its now-unused +`import json` at `hil_health.py:19`: + +```python +def write_timeout_report(report_dir: Path, boards, secs: int, + banner: str = '', prefix: str = '') -> None: + """Leave a report behind when the worker pool has to be abandoned. + + map_async is all-or-nothing, so a timeout loses every per-board result and the report + dir would stay empty with no reason for the failure. Any prior attempt's rows are kept + and the stuck boards are merged in beside them. + + `prefix` carries the preflight rig-health verdict: the timeout aborts before + accumulate_report, so without it the report loses the one line saying WHY the pool never + finished.""" + try: + # Built INSIDE the try: a roster entry without a 'name' key raises while assembling + # the board list, and outside the try that escaped and stranded the runner -- which + # is exactly what the broad handler below exists to prevent. + caveat = (prefix + '\n' if prefix else '') + (banner or ( + f'**HIL run abandoned: worker pool timed out after {secs}s.**\n\n' + f'No per-board results could be collected for this attempt, so any rows below ' + f'are from an earlier one. Boards dispatched:\n\n' + + '\n'.join(f'- {b.get("name", "?")}' for b in boards) + '\n')) + # Rows MERGE rather than replace: an earlier attempt's finished boards are real + # results and this attempt has none of its own. Own handler, because a torn sidecar + # must not cost the stuck rows -- losing the old table is a nicety, losing the + # caveat is the failure. + jpath = report_dir / REPORT_JSON + try: + doc = json.loads(jpath.read_text()) if jpath.is_file() else {} + rows = list(doc.get('rows', [])) + except (OSError, ValueError, TypeError, AttributeError): + doc, rows = {}, [] + done = {r.get('board') for r in rows if isinstance(r, dict)} + rows += [{'board': b.get('name', '?'), 'cells': {'pool-timeout': 'fail'}, + 'duration': None} for b in boards if b.get('name', '?') not in done] + write_report(report_dir, {'rows': rows, 'banner': doc.get('banner', ''), + 'scope': doc.get('scope', ''), 'caveat': caveat}) + except Exception as e: # noqa: BLE001 + # Deliberately broad: this is the first statement of the pool-abandon path, so ANY + # escape skips kill_pool_children and os._exit and strands the runner. + _p(f'warning: cannot write {REPORT_MD} to {report_dir}: {e}', flush=True) +``` + +Update `hil_health.py`'s module docstring: its first line reads "Shutting a wedged HIL run +down: kill what the workers spawned, then report." — drop ", then report". + +- [ ] **Step 4: Update the call site** + +`hil_test.py:2498` becomes: + +```python + hil_report.write_timeout_report( + report_dir, [b for b in config_boards + if b['name'] in stuck], POOL_TIMEOUT, + prefix=health_banner) +``` + +- [ ] **Step 5: Run the tests** + +Run: `python3 -m unittest discover -s test/hil/test` → 274 OK (one deleted, one added) + +- [ ] **Step 6: Commit** + +```bash +git add test/hil/helper/hil_report.py test/hil/helper/hil_health.py test/hil/hil_test.py \ + test/hil/test/test_hil_report.py test/hil/test/test_hil_health.py +git commit -m "hil_report: the pool-guard fallback renders like every other writer + +It composed its own markdown for one reason: hil_health cannot import hil_test +back, so it could not reach render_report. With the renderer in a module both +import, that constraint is gone and all five writers are byte-identical -- +MarkdownIsAlwaysARenderingOfTheJson covers the fifth, and the weaker +'agrees even if it does not render' promise is deleted. + +hil_health goes back to doing one thing: killing wedged processes." +``` + +--- + +### Task 4: Fold `hil_summary.py` in and delete it + +**Files:** +- Modify: `test/hil/helper/hil_report.py` (real `summarize` + CLI) +- Delete: `test/hil/helper/hil_summary.py` +- Modify: `test/hil/hil_ci.sh` (drop `hil_summary.py` from the scp list) +- Modify: `.claude/agents/hil-operator.md:71`, `.claude/workflows/hil-validate.js:14,17,54,58,67`, `.claude/workflows/test-hil-validate.mjs:7` +- Modify: `test/hil/test/test_hil_bounded.py` (move `SummaryFoldsReportToBoards` out), `test/hil/test/test_hil_report.py` + +**Interfaces:** +- Consumes: `cell_state`, `LOCKED_CELL`, `REPORT_JSON` from Task 1. +- Produces: `hil_report.variants_of(cfg, board) -> list`, `hil_report.summarize(cfg, boards, report) -> dict` returning `{'results': [...], 'banner': str, 'caveat': str}`; CLI `python3 test/hil/helper/hil_report.py [-b BOARD]... [--report-dir DIR]`. + +- [ ] **Step 1: Move the tests** + +Move `SummaryFoldsReportToBoards` from `test/hil/test/test_hil_bounded.py` into +`test/hil/test/test_hil_report.py`, changing the subprocess target from +`helper/hil_summary.py` to `helper/hil_report.py` in both places (`test_hil_bounded.py:1675` +and `:1757`). Add one test pinning that the old entry point is gone: + +```python + def test_the_old_entry_point_is_gone(self): + """hil_summary.py's CLI moved here. A leftover file would keep working while + drifting from the module that now owns the fold.""" + self.assertFalse((Path(HIL_DIR) / 'helper' / 'hil_summary.py').exists()) +``` + +- [ ] **Step 2: Run them to verify they fail** + +Run: `python3 test/hil/test/test_hil_report.py` +Expected: FAIL — the subprocess exits non-zero with `hil_report: summarize() lands in Task 4` + +- [ ] **Step 3: Move `summarize` in and delete the old file** + +Copy `variants_of` (`hil_summary.py:47-52`) and `summarize` (`:54-92`) into `hil_report.py` +verbatim, with two changes: `cell_state(str(val))` becomes `cell_state(val)` (the merged +classifier is isinstance-guarded, so the `str()` is dead), and the module's own +`FAIL_ICON`/`SKIP_ICON`/`LOCKED_CELL`/`cell_state` definitions are NOT copied — Task 1's +already serve. + +Replace the Task 1 placeholder `main()` with the real one from `hil_summary.py:94-115`, +changing `Path(a.report_dir) / 'hil_report.json'` to `Path(a.report_dir) / REPORT_JSON`. + +Then: + +```bash +git rm test/hil/helper/hil_summary.py +``` + +- [ ] **Step 4: Update the consumers** + +`test/hil/hil_ci.sh` — remove the `hil_summary.py` line from the scp list added in Task 1. + +`.claude/agents/hil-operator.md:71`: + +```bash +python3 test/hil/helper/hil_report.py -b BOARD [-b BOARD...] # from the report dir +``` + +`.claude/workflows/hil-validate.js:58`: + +```javascript + ` python3 test/hil/helper/hil_report.py ${boards.map((b) => `-b ${b}`).join(' ')}\n` + +``` + +In `.claude/workflows/hil-validate.js` lines 14, 17, 54 and 67, and +`.claude/workflows/test-hil-validate.mjs` line 7, replace the prose mentions of +`hil_summary.py` with `hil_report.py`. Change nothing else in those files — the operator's +return contract (`{results, banner, wedged}`) is untouched. + +- [ ] **Step 5: Run the tests** + +Run: `python3 test/hil/test/test_hil_report.py` → OK +Run: `python3 -m unittest discover -s test/hil/test` → 275 OK +Run: `node .claude/workflows/test-hil-validate.mjs` → OK +Run: `grep -rn "hil_summary" . --include=*.py --include=*.sh --include=*.js --include=*.mjs --include=*.md | grep -v docs/superpowers` → no hits + +- [ ] **Step 6: Commit** + +```bash +git add test/hil/helper/hil_report.py test/hil/hil_ci.sh test/hil/test/ \ + .claude/agents/hil-operator.md .claude/workflows/hil-validate.js \ + .claude/workflows/test-hil-validate.mjs +git rm --cached test/hil/helper/hil_summary.py 2>/dev/null || true +git commit -m "hil_report: fold hil_summary in; one module owns the document end to end + +The fold to per-board verdicts is the read half of the artifact the rest of this +module writes, and it carried the second copy of the cell classifier. The CLI +keeps its arguments; only its path changes, which the two harness docs that +invoke it by name follow." +``` + +--- + +## Validation + +- [ ] **Full gate** + +```bash +python3 -m unittest discover -s test/hil/test # 275 OK +pre-commit run --all-files +``` + +- [ ] **Prove the motion changed no behaviour.** Re-render the real fleet report captured + before the refactor and diff it against what the branch produces now: + +```bash +python3 - <<'EOF' +import json, sys +sys.path.insert(0, 'test/hil') +from helper import hil_report +doc = json.load(open('hil_report.json')) # the pair the rig produced pre-refactor +assert open('hil_report.md').read() == hil_report.render_report(doc) + '\n', 'render drifted' +print('render is byte-identical to the pre-refactor artifact') +EOF +``` + +- [ ] **Rig re-check.** `hil_report.py` must reach the rig and the CLI must run there: + +```bash +bash test/hil/hil_ci.sh -b stm32f407disco -b nanoch32v203 +ssh hathach@ci.lan 'cd /tmp/tinyusb-hil && python3 test/hil/helper/hil_report.py \ + test/hil/tinyusb.json -b stm32f407disco -b nanoch32v203' +``` + +Expect a two-board table, `md == render_report(json)`, and a `summarize` verdict naming both +boards — `nanoch32v203` proving the variant fold still works through the moved code. + +## Out of scope + +Each its own follow-up, unchanged from the spec: + +- Splitting `accumulate_report`'s `mret` folding from its merge. +- The flat `HIL_POOL_TIMEOUT` that does not scale with board count. +- Carrying `caveat` through the operator/workflow return contract (`hil-validate.js:34`). diff --git a/docs/superpowers/specs/2026-08-21-hil-report-module-design.md b/docs/superpowers/specs/2026-08-21-hil-report-module-design.md new file mode 100644 index 000000000..41e7000b7 --- /dev/null +++ b/docs/superpowers/specs/2026-08-21-hil-report-module-design.md @@ -0,0 +1,144 @@ +# hil_report.py: one owner for the HIL report document + +**Date:** 2026-08-21 +**Branch:** `hil-report` (continues the report-unification work already on it) + +## Motivation + +`hil_report.json` and `hil_report.md` are now one document rendered two ways, but the code that +produces, renders, merges and reads that document is spread across three modules: + +| Module | Report-related content | +|---|---| +| `hil_test.py` | `REPORT_CELL`, `BOUNDARY_CELL`, `REPORT_MD`, `REPORT_JSON`, `render_matrix`, `render_report`, `write_report`, `mark_report_abandoned`, `accumulate_report` | +| `helper/hil_health.py` | `write_timeout_report` — composes its own markdown | +| `helper/hil_summary.py` | `cell_state`, `variants_of`, `summarize`, CLI | + +Two concrete defects follow from that spread. + +**One classifier, two copies.** `hil_test.py:1966` (`cell_kind`, keyed off `REPORT_CELL`) and +`hil_summary.py:34` (`cell_state`, with its own re-typed `FAIL_ICON, SKIP_ICON = '❌', '⚪'`) +implement the same rule. The latter's docstring says it is *"the EXACT classifier hil_test.py's own +tally uses"* — the duplication was noticed and documented as an obligation to keep in sync, rather +than removed. Change `REPORT_CELL` and the human's table and the agent's verdict silently disagree: +the markdown says ❌ where the JSON says `pass`. That is the same class of defect this branch +exists to eliminate, one layer up. + +**A writer that cannot render.** `hil_test.py` imports `hil_health`, so `hil_health` cannot import +`hil_test` back. That is the only reason `write_timeout_report` composes its own markdown instead of +calling `render_report`, and the only reason the pool-guard fallback is held to a weaker promise +(same boards and caveat in both artifacts, not byte-identical) while the other four writers are +exact. The constraint is structural, not essential: a leaf module both can import dissolves it. + +## Goal / non-goals + +**Goal:** `test/hil/helper/hil_report.py` becomes the single owner of the report document. + +**This is NOT purely code motion, and the distinction matters for review.** Measured against +`master`, `hil_test.py` contains only `render_matrix` and `accumulate_report`. Everything else in +the new module — `render_report`, `write_report`, `mark_report_abandoned`, `mark_report_no_boards`, +`_load`, `_write_stuck_over_prior_md`, `cell_state`, and the `scope`/`caveat` plumbing — is NEW +code, roughly 150 lines of it, and two rounds of review found most of their defects there. Read +those functions as new, not as relocated. `hil_test.py`'s CLI, arguments and table format do stay +unchanged. + +**Deliberate user-visible changes:** +1. `hil_summary.py` is deleted; its CLI moves to `hil_report.py`. The documented command becomes + `python3 test/hil/helper/hil_report.py -b BOARD [-b BOARD…]`. +2. `write_timeout_report` re-renders from the merged sidecar instead of stapling its banner above + the previous attempt's markdown text. Output improves — one table containing the stuck boards, + rather than a fresh banner above a duplicate table — but it is a change (see Testing). + +**Non-goals (explicit follow-ups, not this change):** +- Splitting `accumulate_report`'s `mret` folding from its merge (see "Deliberate wart"). +- The flat `HIL_POOL_TIMEOUT` that does not scale with board count (`hil_test.py:225`). + +## Resulting layout (`test/hil/`) + +| File | ~Lines | Role | +|---|---|---| +| `hil_test.py` | 2390 (−250) | tests + orchestration + CLI | +| `helper/hil_report.py` (new) | ~400 | the report document: vocabulary, render, write, merge, fold, CLI | +| `helper/hil_health.py` | ~345 (−53) | killing wedged processes only | +| `helper/hil_summary.py` | deleted | superseded by `hil_report.py` | + +Import graph: `hil_health` is a leaf; `hil_report` → `hil_health` (for `_p`, the +BrokenPipeError-safe print used on containment paths); `hil_test` → both. No cycles. + +## hil_report.py + +Stdlib only (`json`, `argparse`, `pathlib`) beyond that one `_p` import. Sections, in order: + +**Vocabulary.** `REPORT_MD`, `REPORT_JSON`, `REPORT_CELL`, `BOUNDARY_CELL`, `LOCKED_CELL`. +`REPORT_CELL` becomes the single source of the status icons; `hil_summary.py`'s `FAIL_ICON`/ +`SKIP_ICON` literals are deleted. + +**Classifier.** One `cell_state(v) -> 'pass' | 'fail' | 'skip'`, replacing both `cell_kind` and the +old `cell_state`. Keeps the surviving docstring's warning that the `pass` arm is load-bearing: a +passing test may return an unprefixed metric string (`'480.0 MBps'`), while failures are guaranteed +icon-marked, so classifying unknown shapes as `fail` would publish a green table as a red verdict. + +**Render.** `render_matrix(rows_all)`, `render_report(doc)`. Unchanged; `render_matrix`'s inline +`cell_kind` is replaced by a call to the module-level `cell_state`. + +**Write.** `write_report`, `accumulate_report`, `mark_report_abandoned`, `write_timeout_report`. +Moved verbatim except `write_timeout_report`, which loses its `md_name` parameter (the module owns +`REPORT_MD`) and renders instead of concatenating. + +**Fold.** `variants_of`, `summarize`, and the `main()` CLI from `hil_summary.py`. + +## Deliberate wart + +`accumulate_report` moves wholesale, keeping its knowledge of `mret`'s worker-result tuple shape. +The cleaner boundary would split "fold `mret` → rows" (`hil_test`'s domain) from "merge rows → doc" +(`hil_report`'s), but that rewrites subtle, well-tested logic — stale `board-locked` clearing, +`BOUNDARY_CELL` dropping, `duration=None` preservation — for a tidier seam. It is a data-shape +coupling, not an import cycle. Moving it verbatim keeps the motion reviewable as motion. + +## The sharp edge + +`hil_ci.sh:222-228` stages helper modules by an **explicit scp list**. A new `helper/hil_report.py` +that is not added there reaches the rig missing, and the run dies with `ImportError` *after* +`REMOTE_DIR` has already been wiped — so the previous run's report and re-run spec are gone too. + +This is already guarded: `test_hil_bounded.py`'s `RemoteStaging.test_import_closure_is_staged_to_the_rig` +walks the AST import closure from `hil_test.py`, `usbtest.py` and `mtp_test.py` and requires an exact +scp entry for each file. Adding the module to the list is all this change needs; no new guard is +warranted, and an earlier draft of this document wrongly claimed none existed. + +## Consumers to update + +| File | Change | +|---|---| +| `test/hil/hil_ci.sh:226` | `hil_summary.py` → `hil_report.py` in the scp list | +| `.claude/agents/hil-operator.md:71` | the documented command | +| `.claude/workflows/hil-validate.js:58` | the command the operator is told to run | +| `.claude/workflows/hil-validate.js:14,17,54,67`, `test-hil-validate.mjs:7` | stale `hil_summary.py` mentions in comments | + +No logic in the `.claude` files changes — the operator's return contract +(`{results, banner, wedged}`) is untouched. + +## Testing + +New `test/hil/test/test_hil_report.py`. The report-specific classes move there from +`test_hil_bounded.py` (`CaveatSurvivesAccumulate`, `SummaryFoldsReportToBoards`, +`ScopeSurvivesInTheJson`, `RenderReportIsPureFunctionOfTheDocument`, +`EveryExitPathLeavesBothArtifacts`, `AbandonNoticeLandsInBothArtifacts`, +`MarkdownIsAlwaysARenderingOfTheJson`) and from `test_hil_health.py` (`WriteTimeoutReport`). + +Three test changes are substantive rather than mechanical: + +1. `WriteTimeoutReport.test_keeps_a_previous_attempts_table` asserts the prior **markdown text** + survives. It becomes an assertion that the prior attempt's **rows** survive — the same guarantee + against the new representation. +2. `MarkdownIsAlwaysARenderingOfTheJson` gains a fifth case for the pool-guard fallback, which now + satisfies the byte-identical invariant like the other four. +3. `test_the_pool_guard_fallback_agrees_even_if_it_does_not_render` — the weaker promise — is + deleted, because the promise it encoded no longer applies. + +Gate: `python3 -m unittest discover -s test/hil/test` at 275 — the current 266, minus the one +deleted test, plus the fifth invariant case, the scp-list guard, two dual-mode import tests, +five classifier tests and one pinning that the old entry point is gone — then +`pre-commit run --all-files`. Because this lands on a +branch already validated on hardware, it closes with a rig re-check: the invariant check against a +real report pair and a scoped `--accumulate` run, not the full fleet. diff --git a/test/hil/helper/hil_health.py b/test/hil/helper/hil_health.py index 92f0accc8..aa811eeb4 100644 --- a/test/hil/helper/hil_health.py +++ b/test/hil/helper/hil_health.py @@ -1,6 +1,6 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: MIT -"""Shutting a wedged HIL run down: kill what the workers spawned, then report. +"""Shutting a wedged HIL run down: kill what the workers spawned. A device whose usbfs node is held by a D-state process cannot be freed -- SIGKILL is not delivered in uninterruptible sleep -- so the goal is never to fix the rig from here. It is @@ -341,40 +341,3 @@ def kill_pool_children(pool, *extra) -> int: # SIGKILL is asynchronous and a D-state task ignores it: only a confirmed survivor # justifies the caller's power-cycle wording return len(_kill_and_confirm(killed_pids)) if killed_pids else 0 - - -def write_timeout_report(report_dir: Path, boards, secs: int, md_name: str, - banner: str = '', prefix: str = '') -> None: - """Leave a report behind when the worker pool has to be abandoned. - - map_async is all-or-nothing, so a timeout loses every per-board result and the report - dir would stay empty with no reason for the failure. Any prior attempt's markdown is - kept below the banner.""" - # `prefix` carries the preflight rig-health verdict: the timeout aborts before - # accumulate_report, so without it the report loses the one line saying WHY the pool - # never finished. The '\n' stops Markdown lazy continuation pulling the banner into - # the blockquote. - try: - # Built INSIDE the try: a roster entry without a 'name' key raises KeyError while - # assembling the board list, and outside the try that escaped and stranded the - # runner -- which is exactly what the broad handler below exists to prevent. - head = (prefix + '\n' if prefix else '') + (banner or ( - f'**HIL run abandoned: worker pool timed out after {secs}s.**\n\n' - f'No per-board results could be collected for this attempt, so the ' - f'table below (if any) is from an earlier one. Boards dispatched:\n\n' - + '\n'.join(f'- {b.get("name", "?")}' for b in boards) + '\n')) - report_dir.mkdir(parents=True, exist_ok=True) - md_path = report_dir / md_name - # Its own handler so it cannot take the write down with it: a report torn by an - # attempt killed mid-write raises UnicodeDecodeError (a ValueError, and prior - # reports always contain status emoji), which under a shared try skipped the write - # entirely. Losing the old table is a nicety; losing the banner is the failure. - try: - prior = md_path.read_text(encoding='utf-8') if md_path.is_file() else '' - except (OSError, ValueError): - prior = '' - md_path.write_text(head + (f'\n{prior}' if prior else ''), encoding='utf-8') - except Exception as e: # noqa: BLE001 - # Deliberately broad: this is the first statement of the pool-abandon path, so ANY - # escape skips kill_pool_children and os._exit and strands the runner. - _p(f'warning: cannot write {md_name} to {report_dir}: {e}', flush=True) diff --git a/test/hil/helper/hil_report.py b/test/hil/helper/hil_report.py new file mode 100644 index 000000000..b73c030a9 --- /dev/null +++ b/test/hil/helper/hil_report.py @@ -0,0 +1,544 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +"""The HIL report document: one owner for hil_report.json and hil_report.md. + +The markdown IS a rendering of the sidecar -- every writer goes through render_report(), so +a table can never contain something the JSON does not. This module owns the whole life of +that document: the cell vocabulary, the one classifier both artifacts share, rendering, the +writers, and the fold to one machine-readable verdict per board. + +Dual-mode by design: imported as `helper.hil_report` by hil_test.py, and run as a script by +the operator (see .claude/agents/hil-operator.md). A script run puts test/hil/helper on +sys.path rather than test/hil, hence the guarded hil_health import below. +""" +import argparse +import json +import sys +from pathlib import Path + + + +def _p(*args, **kwargs) -> None: + """Print that cannot raise. Defined here rather than imported from hil_health: this + module is ALSO run as a script (hil-operator.md invokes it by path), and under + PYTHONSAFEPATH=1 -- which the suite's own MTP fixtures set -- sys.path[0] is not the + script dir, so any sibling import dies before argparse runs. Five lines beat that.""" + try: + print(*args, **kwargs) + except (OSError, ValueError): + # ValueError too: printing to a CLOSED stream raises "I/O operation on closed + # file", and escaping here skips the containment path's os._exit. + pass + +REPORT_MD = 'hil_report.md' +REPORT_JSON = 'hil_report.json' +# The status vocabulary, shared by the code that WRITES a cell (hil_test's test runners) and +# the code that reads one back (cell_state). One dict, so the human's table and the agent's +# verdict cannot drift apart. +REPORT_CELL = {'pass': '✅', 'fail': '❌', 'skip': '⚪'} +BOUNDARY_CELL = 'same-PID boundary' +LOCKED_CELL = 'board-locked' +# A pseudo-test column, not a real one: write_timeout_report marks the boards that were +# still dispatched when the pool guard fired. accumulate_report clears it on a retry. +POOL_TIMEOUT_CELL = 'pool-timeout' + + +def _load(report_dir: Path) -> tuple: + """(doc, readable) for the sidecar, coerced to the canonical shape. + + hil_ci.sh uploads a sidecar as the --accumulate merge base, so a non-conforming one is + reachable from OUTSIDE the harness -- and every writer here runs on a path where a + TypeError costs the whole report. Coerce once, at the boundary, instead of guarding + each use: `banner: null` used to kill a fully successful run with a traceback and no + artifact at all, and `cells: null` sent write_timeout_report down its fallback so a + board that ate the whole pool guard was published as a pass. + + `readable` is False only when a sidecar EXISTS but could not be parsed, or is absent -- + both mean its rows are unrecoverable, which callers use to avoid destroying a markdown + that may still hold them.""" + jpath = report_dir / REPORT_JSON + if not jpath.is_file(): + return {'rows': [], 'banner': '', 'scope': '', 'caveat': ''}, False + try: + raw = json.loads(jpath.read_text()) + if not isinstance(raw, dict): + raise ValueError('sidecar is not an object') + except (OSError, ValueError, TypeError): + return {'rows': [], 'banner': '', 'scope': '', 'caveat': ''}, False + rows = [] + # isinstance, not `or []`: a sidecar with `rows: 1` iterates an int and raises outside + # the parse handler above. + for r in (raw.get('rows') if isinstance(raw.get('rows'), list) else []): + if not isinstance(r, dict) or 'board' not in r: + continue + cells = r.get('cells') + dur = r.get('duration') + # VALUES as well as keys: render_matrix does REPORT_CELL.get(v, v), which raises + # TypeError on an unhashable value, and cell_state does v.startswith. A non-str + # cell is corrupt, and dropping it renders blank -- "not run" -- which is the + # honest reading. Coercing it to str would make it classify as a PASS. + rows.append({'board': str(r['board']), + 'cells': {str(k): v for k, v in cells.items() if isinstance(v, str)} + if isinstance(cells, dict) else {}, + 'duration': dur if isinstance(dur, str) else None}) + text = lambda k: raw[k] if isinstance(raw.get(k), str) else '' + return {'rows': rows, 'banner': text('banner'), 'scope': text('scope'), + 'caveat': text('caveat')}, True + + +def cell_state(v) -> str: + """'pass' | 'fail' | 'skip' for one report cell. + + THE classifier -- the markdown tally and the per-board verdict both call this, so they + cannot disagree. 'fail' or a fail-icon prefix is a failure, 'skip' or a skip-icon prefix + is a skip, and EVERYTHING ELSE is a pass. That last arm is load-bearing: a passing test + may return a plain metric string ('480.0 MBps') that lands in the cell unprefixed, while + failures are guaranteed marked -- TestFail's docstring pins that its metric is + icon-prefixed precisely so render and tally treat it as a failure. Classifying unknown + shapes as fail here would publish a green table as a red verdict. + + isinstance-guarded: cells are usually str but a caller may hand over None or a number, + and .startswith on those raises inside a report writer that must not raise.""" + if v == 'fail' or (isinstance(v, str) and v.startswith(REPORT_CELL['fail'])): + return 'fail' + if v == 'skip' or (isinstance(v, str) and v.startswith(REPORT_CELL['skip'])): + return 'skip' + return 'pass' + + +def render_matrix(rows_all: list) -> str: + """Render rows (list of (row_label, {example: status}, duration)) as an aligned + markdown matrix: columns = tests (bare names) centered, boards left-aligned, + per-row duration as the trailing column.""" + seen = set() + for _, cells, _ in rows_all: + seen.update(cells) + if not seen: + return 'No tests were run.' + + # metric-bearing columns pinned first, the rest alphabetical: stable regardless of the + # shuffled execution order + pinned = ['usbtest', 'cdc_msc_throughput', 'msc_file_explorer', 'msc_file_explorer_freertos'] + + def col_key(t): + name = t.rsplit('/', 1)[-1] + return (pinned.index(name) if name in pinned else len(pinned), name, t) + + columns = sorted(seen, key=col_key) + headers = [c.rsplit('/', 1)[-1] for c in columns] + ['duration'] # bare example names + + def cell(cells, col): + v = cells.get(col) + if v is None: + return '' + return REPORT_CELL.get(v, v) # status symbol, or a metric string (e.g. speed) verbatim + + rows_vals = [(lbl, [cell(cells, c) for c in columns] + [dur or '']) + for lbl, cells, dur in rows_all] + board_hdr = 'Board' + board_w = max([len(board_hdr)] + [len(lbl) for lbl, _ in rows_vals]) + col_w = [max([len(h)] + [len(vals[i]) for _, vals in rows_vals]) + for i, h in enumerate(headers)] + + def line(label, values): + padded = [label.ljust(board_w)] + [v.center(w) for v, w in zip(values, col_w)] + return '| ' + ' | '.join(padded) + ' |' + + header = line(board_hdr, headers) + sep = '| ' + '-' * board_w + ' | ' + ' | '.join(':' + '-' * (w - 2) + ':' for w in col_w) + ' |' + body = [line(lbl, vals) for lbl, vals in rows_vals] + + # tally run cells (not-run cells are absent from the dicts). A cell is a bare status or + # a metric string carrying its own icon ("❌ 29/30"), so classify by the leading icon -- + # through cell_state, the same call the per-board verdict makes. + kinds = [cell_state(v) for _, cells, _ in rows_all for v in cells.values()] + failed = kinds.count('fail') + skipped = kinds.count('skip') + passed = kinds.count('pass') + summary = (f'**{REPORT_CELL["pass"]} {passed} passed · {REPORT_CELL["fail"]} {failed} failed · ' + f'{REPORT_CELL["skip"]} {skipped} skipped · blank not run**') + + return summary + '\n\n' + '\n'.join([header, sep] + body) + + +def render_report(doc: dict) -> str: + """The markdown IS a rendering of the sidecar. Every writer goes through here, so a + table can never contain something the JSON does not.""" + # .get throughout, not subscripts: mark_report_abandoned renders a sidecar it did NOT + # write (hil_ci.sh reuses a persistent REMOTE_DIR, so it may be an older version's or + # a torn one) on the way to os._exit, and a KeyError there is not in its handler -- + # it would unwind into multiprocessing's unbounded join and hang the runner it is + # trying to free. Same reason summarize() below reads cells as `r.get('cells') or {}`. + md = render_matrix([(r.get('board', '?'), r.get('cells') or {}, r.get('duration')) + for r in doc.get('rows') or [] if isinstance(r, dict)]) + if doc.get('scope'): + # a scoped run's small table is otherwise indistinguishable from a full one, and + # it replaces the previous full table in the sticky PR comment + md = f'_Scoped run: {doc["scope"]}. Boards/tests not listed were not run._\n\n' + md + # banner, then caveat: a rig-health caveat outranks the table AND the scope note, and an + # abandon notice outranks even that -- the top of the report is where hil/SKILL.md tells + # the agent to look + if doc.get('banner'): + md = doc['banner'] + '\n' + md + if doc.get('caveat'): + md = doc['caveat'] + '\n' + md + return md + + +def write_report(report_dir: Path, doc: dict) -> None: + """Write both artifacts from one document. + + RAISES on failure, deliberately: every caller is on a path whose own handler exists to + report exactly this (write_timeout_report's _p warning, hil_test's fallback-of-the- + fallback). Swallowing OSError here made both of those dead code, so an unwritable or + root-owned report dir produced no artifact AND no message. + + Renders BEFORE writing anything: committing the JSON first and then raising in + render_report left a sidecar saying "abandoned" beside a markdown still reading as a + clean green table -- the one invariant this module exists to hold.""" + md = render_report(doc) + '\n' + report_dir.mkdir(parents=True, exist_ok=True) + (report_dir / REPORT_JSON).write_text(json.dumps(doc, indent=2) + '\n') + (report_dir / REPORT_MD).write_text(md, encoding='utf-8') + + +def _abandon_notice(why: str) -> str: + # Wording is a CONTRACT: .claude/skills/hil/SKILL.md pins this banner as the case where + # "the table below IS this run's ... Report the results AND the abandonment". Calling + # the table partial would send the reading agent to re-run boards that already passed. + return (f'**HIL run abandoned: {why}** The table below was collected before the ' + f'abandon; treat board results as unverified.\n') + + +def _already_abandoned(doc: dict) -> bool: + """Whether THIS attempt already recorded how it ended. + + `caveat` only. It used to check `banner` too, because hil_test.py folded its abandon + notices in there -- but banner is carried across an --accumulate retry by design, so a + stale notice from an earlier attempt silenced a genuinely new abandon and the run's own + failure went unrecorded. banner now carries rig HEALTH (which describes the conditions + the cells were collected under, and so must persist); caveat carries the run's OUTCOME + (which must not).""" + return '**HIL run ab' in doc.get('caveat', '') + + +def _stamp_markdown(report_dir: Path, notice: str) -> None: + """Last line of defence: prepend the notice to the markdown itself. + + pr_comment.yml cats only hil_report.md, so a path that gives up here publishes a clean + green table under an abandoned, non-zero job. Master did this unconditionally.""" + mpath = report_dir / REPORT_MD + if not mpath.is_file(): + return + # errors='replace' and catch ValueError: a torn report or a LANG=C locale raises + # UnicodeDecodeError -- NOT an OSError -- straight past os._exit. + body = mpath.read_text(encoding='utf-8', errors='replace') + if '**HIL run ab' not in body[:2000]: + mpath.write_text(notice + '\n' + body, encoding='utf-8') + + +def mark_report_abandoned(report_dir: Path, why: str) -> None: + """Stamp an existing report as abandoned, in BOTH artifacts. + + Best-effort and silent: this runs while the interpreter is being torn down, and an + exception here hangs the process in multiprocessing's unbounded join().""" + notice = _abandon_notice(why) + try: + doc, readable = _load(report_dir) + if readable: + if _already_abandoned(doc): + return # whoever got there first wins, WRITE included + doc['caveat'] = notice + write_report(report_dir, doc) + return + except (OSError, ValueError, TypeError, AttributeError): + pass # fall through -- a failure here must not cost the stamp entirely + # Unreadable sidecar, or the document write failed. Either way the markdown is what + # the PR comment reads, so stamp it directly rather than giving up. + try: + _stamp_markdown(report_dir, notice) + except (OSError, ValueError, TypeError, AttributeError): + pass + + +def mark_report_no_boards(report_dir: Path, msg: str, fresh: bool = True) -> None: + """Record that the board filters intersected to nothing. + + `fresh` mirrors hil_test's own flag, because this runs BEFORE the fresh wipe: without + it a fresh run whose filter emptied re-published the PREVIOUS run's green rows under + this run's red job -- the stale-table failure it exists to prevent. An --accumulate run + keeps them, since nothing this attempt did invalidates them.""" + try: + doc, _ = _load(report_dir) + if not fresh and _already_abandoned(doc): + # SKILL.md gives the two notices OPPOSITE rules, and an abandon outranks a + # filter that matched nothing -- do not overwrite the record of a failed run. + # Only while ACCUMULATING, though: this runs before the fresh wipe, so guarding + # a fresh run would leave the previous attempt's rows AND its abandon notice + # published as this run's. + return + # A fresh run carries NOTHING from the prior sidecar -- rows, banner and scope + # alike, matching accumulate_report, which builds from an empty prior when fresh. + # Resetting only rows republished a stale rig-health note and a stale scope line + # under this run's notice, from a leftover or uploaded sidecar. + prior = {'rows': [], 'banner': '', 'scope': ''} if fresh else doc + write_report(report_dir, {'rows': prior['rows'], 'banner': prior['banner'], + 'scope': prior['scope'], + 'caveat': f'**HIL run selected no boards.** {msg}\n'}) + except (OSError, ValueError, TypeError, AttributeError): + pass # loud on stdout already; the exit code is what the job reads + + +def accumulate_report(mret: list, report_dir: Path, fresh: bool, scope: str = '', + banner: str = '', caveat: str = '') -> str: + """Merge this run's results into json in report_dir, then (re)write + the markdown matrix to md. `fresh` (a first run, no --accumulate) + starts a new report; otherwise a re-run accumulates so boards/tests that + already passed are preserved while re-run cells are updated. `scope` names the + board filter, if any, so a scoped table is not mistaken for a full one. + Returns the md. + + `mret` is hil_test.py's worker-result shape (name, err, fts, rows, ...), so this one + function knows something about its caller that the rest of the module does not. Folding + mret into rows could live in hil_test and only the merge here, but that would rewrite + the subtle parts -- stale board-locked clearing, BOUNDARY_CELL dropping, duration=None + preservation -- for a tidier seam. Data-shape coupling, not an import cycle.""" + # ONE canonical load: a sidecar reaching here may have been uploaded by hil_ci.sh as + # the merge base, so it is untrusted input. `banner` carries forward -- it describes + # the conditions the earlier cells were collected under, and the .failed spec re-runs + # only FAILURES so those passes are never re-earned. `caveat` does NOT: it records how + # a RUN ENDED, and this attempt has not ended yet. Carrying it made a clean retry + # publish "HIL run abandoned" over a run where nothing was abandoned. + prior = {'rows': [], 'banner': ''} + if not fresh: + prior, _ = _load(report_dir) + acc = {r['board']: [dict(r['cells']), r['duration']] for r in prior['rows']} + prior_banner = prior['banner'] + + # current cells override prior for boards/tests that ran; a filtered run reports + # duration None, keeping the previous full-run value + for name, _, _, rows, *_ in mret: + if rows and not any(LOCKED_CELL in cells for _, cells, _ in rows): + # board ran for real: clear a stale lock-failure cell (its row is keyed by + # board name; test rows may be variant names) + stale = acc.get(name) + if stale is not None: + stale[0].pop(LOCKED_CELL, None) + # and the pool-timeout mark: write_timeout_report stamps it on a board that + # never reported, and update() below MERGES, so without this a board that + # passed clean on the retry kept a red cell for ever. + stale[0].pop(POOL_TIMEOUT_CELL, None) + if not stale[0]: + # variant-keyed boards never repopulate the board-name row, so drop it + # or it renders as a blank ghost row + del acc[name] + for row_label, cells, dur in rows: + row = acc.setdefault(row_label, [{}, None]) + # a row that ran is no longer pool-timed-out, whatever it is keyed by + row[0].pop(POOL_TIMEOUT_CELL, None) + # the boundary cell is only ever written on failure, so a re-run of this + # variant that cleared the boundary must drop the previous attempt's ❌ + if BOUNDARY_CELL not in cells: + row[0].pop(BOUNDARY_CELL, None) + row[0].update(cells) + if dur is not None: + row[1] = dur + + report_dir.mkdir(parents=True, exist_ok=True) + # by LINE, deduped: attempts repeat the same caveat far more often than they add a new + # one, and three copies of the D-state note reads as three incidents + seen, merged = set(), [] + for line in (prior_banner + banner).splitlines(): + if line.strip() and line not in seen: + seen.add(line) + merged.append(line) + banner = '\n'.join(merged) + '\n' if merged else '' + doc = {'rows': [{'board': k, 'cells': c, 'duration': d} for k, (c, d) in acc.items()], + 'banner': banner, 'scope': scope, 'caveat': caveat} + # through write_report, not hand-rolled: writing the JSON and only then rendering is + # the ordering write_report exists to forbid -- a render failure left the sidecar ahead + # of the markdown, which is the one invariant this module holds. + write_report(report_dir, doc) + return render_report(doc) + + +def _write_stuck_over_prior_md(report_dir: Path, doc: dict) -> None: + """Sidecar unrecoverable: rebuild it from the stuck rows alone, but leave the + markdown's existing table beneath the caveat rather than throwing real results away. + + The one place the md-is-a-rendering-of-the-json invariant is deliberately suspended, + because there is no readable json left for it to be a rendering of.""" + try: + prior = (report_dir / REPORT_MD).read_text(encoding='utf-8') + except (OSError, ValueError): + prior = '' + # Say so explicitly: those rows exist only as rendered text, so no later --accumulate + # can merge them back. Claiming the sidecar represents them would be false. + note = ('_The table below is a previous attempt\'s rendered output. The sidecar could ' + 'not be read, so those rows are NOT in it and will not survive another run._\n') + head = (doc['banner'] + '\n' if doc['banner'] else '') + doc['caveat'] + '\n' + note + body = prior if prior.strip() else render_matrix( + [(r['board'], r['cells'], r['duration']) for r in doc['rows']]) + report_dir.mkdir(parents=True, exist_ok=True) + (report_dir / REPORT_JSON).write_text(json.dumps(doc, indent=2) + '\n') + (report_dir / REPORT_MD).write_text(head + '\n' + body, encoding='utf-8') + + +def write_timeout_report(report_dir: Path, boards, secs: int, + banner: str = '', prefix: str = '') -> None: + """Leave a report behind when the worker pool has to be abandoned. + + map_async is all-or-nothing, so a timeout loses every per-board result and the report + dir would stay empty with no reason for the failure. Any prior attempt's rows are kept + and each stuck board is marked with a POOL_TIMEOUT_CELL beside them. + + `prefix` is the preflight rig-health verdict and goes to the BANNER, where rig health + lives and where an --accumulate retry carries it forward; the abandon notice goes to + the caveat, which does not carry. Folding both into the caveat is what made a clean + retry report an abandonment that had not happened.""" + try: + # names INSIDE the try: a roster entry that is not a dict raises here, and outside + # it that escaped and stranded the runner. + names = [b.get('name', '?') if isinstance(b, dict) else '?' for b in boards] + caveat = banner or ( + f'**HIL run abandoned: worker pool timed out after {secs}s.**\n\n' + f'No per-board results could be collected for this attempt. Rows other than ' + f'the {POOL_TIMEOUT_CELL} cells below are from an earlier attempt. Boards ' + f'dispatched:\n\n' + '\n'.join(f'- {n}' for n in names) + '\n') + doc, readable = _load(report_dir) + rows = doc['rows'] + by_board = {r['board']: r for r in rows} + for name in names: + row = by_board.get(name) + if row is None: + rows.append({'board': name, 'cells': {POOL_TIMEOUT_CELL: 'fail'}, + 'duration': None}) + else: + # _load guarantees `cells` is a dict, so a null-cells row from an uploaded + # sidecar can no longer send this down the fallback and publish a board + # that ate the whole pool guard as a pass. + row['cells'][POOL_TIMEOUT_CELL] = 'fail' + out = {'rows': rows, 'scope': doc['scope'], 'caveat': caveat, + 'banner': ((doc['banner'] + prefix) if prefix not in doc['banner'] + else doc['banner'])} + if not readable and (report_dir / REPORT_MD).is_file(): + # `readable` covers ABSENT as well as torn: an absent sidecar beside an intact + # markdown used to re-render from the stuck row alone and destroy real results. + _write_stuck_over_prior_md(report_dir, out) + return + write_report(report_dir, out) + except Exception as e: # noqa: BLE001 + # Deliberately broad: this is the first statement of the pool-abandon path, so ANY + # escape skips kill_pool_children and os._exit and strands the runner. + _p(f'warning: cannot write {REPORT_MD} to {report_dir}: {e}', flush=True) + try: + # Same wording as above and the same guarded name extraction -- the fallback + # used to re-derive b.get("name") outside any try and raise identically, so a + # malformed roster left NO artifact at all. + names = [b.get('name', '?') if isinstance(b, dict) else '?' for b in boards] + head = (prefix + '\n' if prefix else '') + (banner or ( + f'**HIL run abandoned: worker pool timed out after {secs}s.**\n\n' + f'No per-board results could be collected for this attempt, so the table ' + f'below (if any) is from an earlier one. Boards dispatched:\n\n' + + '\n'.join(f'- {n}' for n in names) + '\n')) + try: + prior = (report_dir / REPORT_MD).read_text(encoding='utf-8') + except (OSError, ValueError): + prior = '' + report_dir.mkdir(parents=True, exist_ok=True) + (report_dir / REPORT_MD).write_text( + head + (f'\n{prior}' if prior else ''), encoding='utf-8') + except Exception as e2: # noqa: BLE001 + _p(f'warning: fallback {REPORT_MD} write failed too: {e2}', flush=True) + + +def variants_of(cfg: dict, board: str) -> list: + for b in cfg.get('boards', []): + if b['name'] == board: + return [v['name'] for v in (b.get('variant') or [])] or [board] + return [board] + + +def summarize(cfg: dict, boards: list, report: dict) -> dict: + # .get, not a subscript: this is the one reader an agent's verdict depends on, and a + # row without 'board' used to kill the CLI with a traceback and no results at all -- + # hil-validate.js then reports every board as "hil-operator returned no entry". + rows = {r['board']: r.get('cells') or {} + for r in (report.get('rows') or []) + if isinstance(r, dict) and 'board' in r} + owner = {v['name']: b['name'] for b in cfg.get('boards', []) + for v in (b.get('variant') or [])} + results = [] + for board in boards: + names = variants_of(cfg, board) + mine = {n: rows[n] for n in names if n in rows} + # a variant name that is neither declared nor prefixed cannot be attributed; the + # `-` fallback only helps ad-hoc builds, it is not the primary path. It must + # also never steal a row DECLARED by another board: a declared variant need not start + # with its own board's name, so it may happen to start with this board's name plus '-'. + mine.update({n: c for n, c in rows.items() + if n.startswith(f'{board}-') and n not in mine + and owner.get(n, board) == board}) + # the BOARD-name row too: hil_test writes lock contention and pool timeouts keyed + # by board name, but variants_of returns only DECLARED variant names -- and + # nanoch32v203 / ch32v307v_r1_1v0 declare none equal to their board name. Without + # this those rows are invisible, so a lock held by concurrent CI is published as a + # hardware FAIL and hil-validate.js never retries it. + if board in rows and board not in mine: + mine[board] = rows[board] + if not mine: + results.append({'board': board, 'ran': False, 'pass': False, 'locked': False, + 'detail': 'no report row for this board'}) + continue + # a wedge outranks lock contention: `locked` short-circuits `detail` below, so a + # stale board-locked cell from an earlier attempt used to mask the pool-timeout + # cell the retry added -- publishing a board that hung the rig as LOCKED, which + # hil-validate.js then RE-RUNS, paying another pool guard on it. + wedged = any(POOL_TIMEOUT_CELL in cells for cells in mine.values()) + locked = not wedged and any(LOCKED_CELL in cells for cells in mine.values()) + bad = [] + for vname, cells in sorted(mine.items()): + for test, val in sorted(cells.items()): + if test == LOCKED_CELL: + continue + if cell_state(val) == 'fail': + bad.append(f'{vname} {test}: {val}') + ok = not bad and not locked + if locked: + detail = 'held by another holder; not flashed' + elif bad: + detail = '; '.join(bad) + else: + detail = f'{len(mine)} variant(s), {sum(len(c) for c in mine.values())} cell(s) ok' + results.append({'board': board, 'ran': True, 'pass': ok, 'locked': locked, + 'detail': detail}) + # `caveat` too: an abandoned or no-boards run says so THERE, and this JSON is all + # an agent gets -- leaving it in the sidecar puts it back where only a human looks. + return {'results': results, 'banner': report.get('banner', ''), + 'caveat': report.get('caveat', '')} + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument('config_file') + ap.add_argument('-b', '--board', action='append', default=[], + help='boards to report on; default: every board in the config') + ap.add_argument('--report-dir', default='.', help=f'where {REPORT_JSON} lives (default: cwd)') + a = ap.parse_args() + + cfg = json.loads(Path(a.config_file).read_text()) + boards = a.board or [b['name'] for b in cfg.get('boards', [])] + jpath = Path(a.report_dir) / REPORT_JSON + if not jpath.is_file(): + print(f'error: {jpath} not found -- did hil_test.py run in this directory?', + file=sys.stderr) + return 1 + # through _load, like every writer: feeding raw JSON to summarize left the one reader an + # agent's verdict depends on crashing on the malformed sidecars the writers tolerate. + doc, _ = _load(Path(a.report_dir)) + json.dump(summarize(cfg, boards, doc), sys.stdout, indent=2) + print() + return 0 + +if __name__ == '__main__': + sys.exit(main()) diff --git a/test/hil/helper/hil_summary.py b/test/hil/helper/hil_summary.py deleted file mode 100644 index e566bead0..000000000 --- a/test/hil/helper/hil_summary.py +++ /dev/null @@ -1,115 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-License-Identifier: MIT -"""Fold hil_report.json into one machine-readable verdict per BOARD. - -A workflow driving hil_test.py through an operator agent has no filesystem access, so the -agent has to carry the results across. It must carry them, not retype them: the previous -design asked the agent to transcribe the markdown table, and every defect found in four -review rounds came from re-parsing that prose -- variant row names vs board names, -`board locked` vs `board-locked`, folding several variant rows into one verdict, rows that -matched no board. All of it is a join, and the join belongs here, where the roster is. - -Report rows are named per VARIANT (hil_test.py builds them from `vname`), and a variant name -is not required to start with the board name -- nanoch32v203 produces only `-fsdev`/`-usbfs`, -ch32v307v_r1_1v0 only `-usbhs`/`-usbfs`. The config is what maps them back. - -Emits, on stdout: - {"results": [{"board", "ran", "pass", "locked", "detail"}...], "banner": str} - -`locked` is a field, not a prefix to grep for. `ran` false means the board produced no row at -all, which is not the same as failing. - -Usage: hil_summary.py [-b BOARD]... [--report-dir DIR] -""" -import argparse -import json -import sys -from pathlib import Path - -FAIL_ICON, SKIP_ICON = '❌', '⚪' # a pass needs no icon: unmarked = pass -LOCKED_CELL = 'board-locked' - - -def cell_state(v: str) -> str: - """'pass' | 'fail' | 'skip' -- the EXACT classifier hil_test.py's own tally uses - (cell_kind in render_matrix): 'fail' or a ❌ prefix is a failure, 'skip' or a ⚪ - prefix is a skip, and EVERYTHING ELSE is a pass. That last arm is load-bearing: a - passing test may return a plain metric string ('480.0 MBps') that lands in the cell - unprefixed, while failures are guaranteed marked -- TestFail's docstring pins that its - metric is icon-prefixed precisely so render/tally treat it as a failure. Classifying - unknown shapes as fail here would publish a green table as a red verdict.""" - if v == 'fail' or v.startswith(FAIL_ICON): - return 'fail' - if v == 'skip' or v.startswith(SKIP_ICON): - return 'skip' - return 'pass' - - -def variants_of(cfg: dict, board: str) -> list: - for b in cfg.get('boards', []): - if b['name'] == board: - return [v['name'] for v in (b.get('variant') or [])] or [board] - return [board] - - -def summarize(cfg: dict, boards: list, report: dict) -> dict: - rows = {r['board']: r.get('cells') or {} for r in report.get('rows', [])} - owner = {v['name']: b['name'] for b in cfg.get('boards', []) - for v in (b.get('variant') or [])} - results = [] - for board in boards: - names = variants_of(cfg, board) - mine = {n: rows[n] for n in names if n in rows} - # a variant name that is neither declared nor prefixed cannot be attributed; the - # `-` fallback only helps ad-hoc builds, it is not the primary path. It must - # also never steal a row DECLARED by another board: a declared variant need not start - # with its own board's name, so it may happen to start with this board's name plus '-'. - mine.update({n: c for n, c in rows.items() - if n.startswith(f'{board}-') and n not in mine - and owner.get(n, board) == board}) - if not mine: - results.append({'board': board, 'ran': False, 'pass': False, 'locked': False, - 'detail': 'no report row for this board'}) - continue - locked = any(LOCKED_CELL in cells for cells in mine.values()) - bad = [] - for vname, cells in sorted(mine.items()): - for test, val in sorted(cells.items()): - if test == LOCKED_CELL: - continue - if cell_state(str(val)) == 'fail': - bad.append(f'{vname} {test}: {val}') - ok = not bad and not locked - if locked: - detail = 'held by another holder; not flashed' - elif bad: - detail = '; '.join(bad) - else: - detail = f'{len(mine)} variant(s), {sum(len(c) for c in mine.values())} cell(s) ok' - results.append({'board': board, 'ran': True, 'pass': ok, 'locked': locked, - 'detail': detail}) - return {'results': results, 'banner': report.get('banner', '')} - - -def main() -> int: - ap = argparse.ArgumentParser() - ap.add_argument('config_file') - ap.add_argument('-b', '--board', action='append', default=[], - help='boards to report on; default: every board in the config') - ap.add_argument('--report-dir', default='.', help='where hil_report.json lives (default: cwd)') - a = ap.parse_args() - - cfg = json.loads(Path(a.config_file).read_text()) - boards = a.board or [b['name'] for b in cfg.get('boards', [])] - jpath = Path(a.report_dir) / 'hil_report.json' - if not jpath.is_file(): - print(f'error: {jpath} not found -- did hil_test.py run in this directory?', - file=sys.stderr) - return 1 - json.dump(summarize(cfg, boards, json.loads(jpath.read_text())), sys.stdout, indent=2) - print() - return 0 - - -if __name__ == '__main__': - sys.exit(main()) diff --git a/test/hil/hil_ci.sh b/test/hil/hil_ci.sh index 514b0f174..daa787242 100644 --- a/test/hil/hil_ci.sh +++ b/test/hil/hil_ci.sh @@ -210,6 +210,69 @@ rm -rf -- "$1" mkdir -p -- "$1/test/hil/helper" "$1/examples" REMOTE +# The --accumulate merge base. The wipe above just cleared REMOTE_DIR, and +# accumulate_report merges onto the sidecar in the RUN's cwd (hil_test.py:2193 sets +# `fresh = not args.accumulate`, and only a non-fresh run reads it) -- so without this a +# remote retry starts from nothing and its one-row table REPLACES the full-fleet one it was +# meant to extend. The copy-back at the end of this script has always existed; this is the +# other half of it. +# +# Gated, not unconditional: a fresh run unlinks the sidecar anyway (hil_test.py:2244), so +# uploading there is wasted work that also obscures what the wipe means. +# +# .failed is deliberately NOT uploaded: hil_test.py only ever writes it, never +# reads it -- the retry spec reaches the rig as the -b/-bt arguments the caller expanded +# from it (`hil_ci.sh $(cat .failed)`). +# argparse decides, not a case arm: hil_test.py declares `-a, --accumulate`, so argparse +# also accepts `-av`, `-va`, `--accum` and `--acc` -- and hil-validate.js tells the +# operator to retry "adding -v", which makes `-av` the natural spelling. A hand-rolled +# match missed all four: no upload, and the else-branch warning never fired either, so the +# one-row table replaced the full-fleet one in silence. +ACCUMULATE=$(python3 - ${ARGS[@]+"${ARGS[@]}"} <<'PY' +import argparse, sys +p = argparse.ArgumentParser(add_help=False) +p.add_argument('-a', '--accumulate', action='store_true') +p.add_argument('-v', '--verbose', action='store_true') # so -av/-va bundle as they do there +print(1 if p.parse_known_args(sys.argv[1:])[0].accumulate else 0) +PY +) || ACCUMULATE=0 +if [ "$ACCUMULATE" = 1 ]; then + if [ -f "$ROOT_DIR/hil_report.json" ]; then + # Provenance: hil_report.json is not namespaced by CONFIG or REMOTE (build.yml and + # pr_comment.yml read that exact name), so a `REMOTE=hifiphile CONFIG=.../hfp.json` + # run leaves an hfp sidecar behind that a later ci.lan retry would merge, publishing + # boards that never ran here. Require at least one row to belong to THIS roster. + if python3 - "$ROOT_DIR/hil_report.json" "$CONFIG" <<'PY' +import json, sys +try: + rows = json.load(open(sys.argv[1])).get('rows') or [] + cfg = json.load(open(sys.argv[2])).get('boards') or [] +except Exception: + sys.exit(1) +known = set() +for b in cfg: + known.add(b.get('name')) + known.update(v.get('name') for v in (b.get('variant') or [])) +sys.exit(0 if not rows or any(r.get('board') in known for r in rows if isinstance(r, dict)) + else 1) +PY + then + echo "==> Uploading hil_report.json as the --accumulate merge base" + scp -q "$ROOT_DIR/hil_report.json" "$REMOTE:$REMOTE_DIR/" + else + echo "==> warning: $ROOT_DIR/hil_report.json holds no board from $(basename "$CONFIG")" \ + "-- it is from another rig or config, so it is NOT being uploaded; this run's" \ + "table will REPLACE rather than extend" >&2 + fi + else + # Loud, because this is the failure mode: the run still succeeds, and quietly + # publishes a small table where a full one used to be. + echo "==> warning: --accumulate was requested but $ROOT_DIR/hil_report.json does not" \ + "exist, so there is nothing to merge onto -- this run's table will REPLACE the" \ + "previous one rather than extend it" >&2 + fi +fi + # Copy HIL test script and config echo "==> Copying test scripts" scp -q "$ROOT_DIR/test/hil/hil_test.py" \ @@ -223,7 +286,7 @@ scp -q "$ROOT_DIR/test/hil/helper/__init__.py" \ "$ROOT_DIR/test/hil/helper/hil_util.py" \ "$ROOT_DIR/test/hil/helper/hil_health.py" \ "$ROOT_DIR/test/hil/helper/hil_lock.py" \ - "$ROOT_DIR/test/hil/helper/hil_summary.py" \ + "$ROOT_DIR/test/hil/helper/hil_report.py" \ "$REMOTE:$REMOTE_DIR/test/hil/helper/" # Copy only firmware binaries (elf/bin/hex) plus esptool metadata @@ -316,19 +379,41 @@ REMOTE # Copy the generated report back to the local checkout (best-effort; the run's # exit code is preserved regardless of whether a report was produced). -scp -q "$REMOTE:$REMOTE_DIR/hil_report.md" "$ROOT_DIR/hil_report.md" \ - && echo "==> Report copied to $ROOT_DIR/hil_report.md" \ - || echo "==> warning: no hil_report.md copied back" >&2 +# rm -f FIRST, exactly as the sidecar loop below does: the markdown and the JSON are two +# halves of ONE document now, so leaving a stale table behind when the copy fails -- beside +# a sidecar that was correctly removed -- publishes last run's green results under this +# run's red job, and the operator's hil_report.py call exits 1 against the missing sidecar. +# Fetch BOTH halves to temps and commit them as a pair. Separate fetch/rename meant a +# markdown that arrived beside a sidecar that did not left the local pair failing the +# rendering invariant, and the next --accumulate retry merging the wrong base. Deleting +# first and then scp'ing was worse still: an ssh drop at the end of a 60-minute run +# destroyed the report outright. +md_ok=0; json_ok=0 +scp -q "$REMOTE:$REMOTE_DIR/hil_report.md" "$ROOT_DIR/hil_report.md.tmp" 2>/dev/null \ + && [ -f "$ROOT_DIR/hil_report.md.tmp" ] && md_ok=1 +scp -q "$REMOTE:$REMOTE_DIR/hil_report.json" "$ROOT_DIR/hil_report.json.tmp" 2>/dev/null \ + && [ -f "$ROOT_DIR/hil_report.json.tmp" ] && json_ok=1 +if [ "$md_ok" = 1 ] && [ "$json_ok" = 1 ]; then + mv -f "$ROOT_DIR/hil_report.md.tmp" "$ROOT_DIR/hil_report.md" + mv -f "$ROOT_DIR/hil_report.json.tmp" "$ROOT_DIR/hil_report.json" + echo "==> Report copied to $ROOT_DIR/hil_report.md (+ sidecar)" +else + rm -f "$ROOT_DIR/hil_report.md.tmp" "$ROOT_DIR/hil_report.json.tmp" + # All or nothing: a half-copied pair is worse than none. The stale local markdown goes + # because that is what gets pasted into a PR as this run's results; the stale sidecar + # goes with it so the two cannot disagree. + rm -f "$ROOT_DIR/hil_report.md" "$ROOT_DIR/hil_report.json" + echo "==> warning: report copy-back incomplete (md=$md_ok json=$json_ok); removed the" \ + "stale local pair -- an --accumulate retry has no merge base until a run succeeds" >&2 +fi -# The re-run spec and the JSON sidecar live in the run's cwd on the rig (REMOTE_DIR), and the -# next invocation rm -rf's it. Without copying them back, the `--accumulate` retry every doc on -# this branch prescribes has nothing to read and nothing to merge onto. Delete the local copies -# FIRST: a green run writes no .failed, so a silent no-op scp would leave last run's spec in -# the checkout looking current, and "retry from the spec" would re-flash boards that passed. -for extra in "$(basename "$CONFIG").failed" hil_report.json; do - rm -f "$ROOT_DIR/$extra" - scp -q "$REMOTE:$REMOTE_DIR/$extra" "$ROOT_DIR/$extra" 2>/dev/null \ - && echo "==> $extra copied to $ROOT_DIR/$extra" || true -done +# The re-run spec lives in the run's cwd on the rig and the next invocation rm -rf's it. +# Delete the local copy first: a green run writes no .failed, so a silent no-op scp would +# leave last run's spec looking current and "retry from the spec" would re-flash boards +# that passed. +spec="$(basename "$CONFIG").failed" +rm -f "$ROOT_DIR/$spec" +scp -q "$REMOTE:$REMOTE_DIR/$spec" "$ROOT_DIR/$spec" 2>/dev/null \ + && echo "==> $spec copied to $ROOT_DIR/$spec" || true exit $rc diff --git a/test/hil/hil_test.py b/test/hil/hil_test.py index fcd7c7e6f..83e5082d4 100755 --- a/test/hil/hil_test.py +++ b/test/hil/hil_test.py @@ -64,7 +64,7 @@ from multiprocessing import TimeoutError as MpTimeoutError sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) # PYTHONSAFEPATH drops it import hil_flash -from helper import hil_health, hil_lock, hil_util +from helper import hil_health, hil_lock, hil_report, hil_util from helper.hil_util import device_tests, dual_tests, host_test # Raw Lock/Semaphore objects in Pool initargs are inheritable only under fork @@ -106,9 +106,6 @@ STATUS_OK = "\033[32mOK\033[0m" STATUS_FAILED = "\033[31mFailed\033[0m" STATUS_SKIPPED = "\033[33mSkipped\033[0m" -# Plain (non-ANSI) cell symbols for hil_report.md; a missing binary counts as skipped. -REPORT_CELL = {'pass': '✅', 'fail': '❌', 'skip': '⚪'} - class TestFail(AssertionError): """Fail a test but still surface a metric string in its report cell (e.g. usbtest's '❌ 29/30' @@ -871,7 +868,7 @@ def test_device_cdc_msc_throughput(board): # payload, so an HS board reads as suspiciously slow. Say so rather than publish a green # cell whose scale is a guess. scale = '' if speed_known else ' FS?' - return f'{REPORT_CELL["pass"]} C {pair(cdc_r, cdc_w)} M {pair(msc_r, msc_w)}{scale}' + return f'{hil_report.REPORT_CELL["pass"]} C {pair(cdc_r, cdc_w)} M {pair(msc_r, msc_w)}{scale}' def test_device_dfu(board): @@ -1364,7 +1361,7 @@ def test_device_usbtest(board): f'no cafe:4010 device with serial {uid}' if seen is False else f'cannot tell whether cafe:4010 {uid} is present: the bounded sysfs reads did ' f'not answer{hil_util.sysfs_blind_note()}', - metric=f'{REPORT_CELL["fail"]} 0/30') + metric=f'{hil_report.REPORT_CELL["fail"]} 0/30') # settle: right after flashing the enumeration can bounce once (and on dual-port parts # the other port's stale node — same serial and PID — lingers), and testusb run into # that gap sees the device drop mid-case @@ -1454,7 +1451,7 @@ def test_device_usbtest(board): board_wedged = (f'{board["name"]}: usbtest reported a hang and was killed ' f'before it could report a verdict') raise TestFail(f'usbtest did not run: {detail}', - metric=f'{REPORT_CELL["fail"]} 0/30') + metric=f'{hil_report.REPORT_CELL["fail"]} 0/30') # A HUNG case that recovery could not clear leaves a D-state holder on this board's # usbfs node. Latch it: the remaining examples would each flash THROUGH that node, @@ -1485,9 +1482,9 @@ def test_device_usbtest(board): # the re-run spec. parsed=True: a retry re-pays the whole battery to re-observe a # wedge, and flashes through the poisoned node to do it. raise TestFail(f'usbtest {passed}/{total} but the device wedged ({board_wedged})', - metric=f'{REPORT_CELL["fail"]} {passed}/{total}', parsed=True) + metric=f'{hil_report.REPORT_CELL["fail"]} {passed}/{total}', parsed=True) if failed == 0 and notrun == 0 and total > 0: - return f'{REPORT_CELL["pass"]} {passed}/{total}' + return f'{hil_report.REPORT_CELL["pass"]} {passed}/{total}' bad = [c.get('num') for c in data.get('cases', []) if c.get('status') not in ('PASS', 'BUDGET')] why = f'usbtest {passed}/{total}' @@ -1503,7 +1500,7 @@ def test_device_usbtest(board): why += f'; {notrun} case(s) never ran ({reason}), so this says nothing about them' # parsed ONLY when every case ran: an aborted battery (budget expiry, kernel hang, bus # drop) leaves BUDGET entries, and those are exactly what a reflash retry can fix. - raise TestFail(why, metric=f'{REPORT_CELL["fail"]} {passed}/{total}', + raise TestFail(why, metric=f'{hil_report.REPORT_CELL["fail"]} {passed}/{total}', parsed=(notrun == 0)) @@ -1705,10 +1702,6 @@ def build_board(board: Board) -> tuple[str, int]: return name, failed -# pseudo-test column for a variant boundary the park-flash could not clear (see below) -BOUNDARY_CELL = 'same-PID boundary' - - def test_board(board: Board) -> tuple[str, int, list[str], list, float]: swept = False name = board['name'] @@ -1722,7 +1715,7 @@ def test_board(board: Board) -> tuple[str, int, list[str], list, float]: log_line(f'{name:25} {STATUS_FAILED}: {e}') # visible report row so the ❌ matches the exit code; failed-tests stays empty so a # re-run repeats the whole board (no bogus -bt filter) - return name, 1, [], [(name, {'board-locked': 'fail'}, None)], 0.0 + return name, 1, [], [(name, {hil_report.LOCKED_CELL: 'fail'}, None)], 0.0 # after the lock: flock wait behind a concurrent run is not board cost t_board = time.monotonic() try: @@ -1809,7 +1802,7 @@ def test_board(board: Board) -> tuple[str, int, list[str], list, float]: # charging again would double-count one incident in the exit code if not wedge_skip: err_count += 1 - cells[BOUNDARY_CELL] = 'fail' + cells[hil_report.BOUNDARY_CELL] = 'fail' # blaming run_list[0] would re-run an innocent test that then passes, # leaving the boundary unretested; re-run the whole board instead board_wide_fail = True @@ -1825,7 +1818,7 @@ def test_board(board: Board) -> tuple[str, int, list[str], list, float]: # Do NOT flash through a poisoned node: each attempt enumerates into # it, blocks uninterruptibly and leaves another stray behind. Report # the skip so the cell is not mistaken for a pass. - cells[test] = f'{REPORT_CELL["skip"]} board wedged' + cells[test] = f'{hil_report.REPORT_CELL["skip"]} board wedged' # ...and re-run the WHOLE board, like the boundary-failure path above: # these tests never executed, so naming them individually in the .failed # spec is not enough -- an --accumulate re-run that fixes only the wedged @@ -1893,8 +1886,6 @@ def test_board(board: Board) -> tuple[str, int, list[str], list, float]: _lock_fh.close() -REPORT_MD = 'hil_report.md' -REPORT_JSON = 'hil_report.json' # controller hints from previous runs: uid -> {'name', 'pci', 'duration'}. Only 'pci' is # consumed (dispatch order and first-flash budgeting, never battery serialization). PCI # addresses are boot-stable, so the cache survives reboots and goes stale on re-cabling. @@ -1912,66 +1903,6 @@ def schedule_boards(boards: list, pci_of_uid: dict) -> list: return [b for grp in itertools.zip_longest(*buckets.values()) for b in grp if b is not None] -def render_matrix(rows_all: list) -> str: - """Render rows (list of (row_label, {example: status}, duration)) as an aligned - markdown matrix: columns = tests (bare names) centered, boards left-aligned, - per-row duration as the trailing column.""" - seen = set() - for _, cells, _ in rows_all: - seen.update(cells) - if not seen: - return 'No tests were run.' - - # metric-bearing columns pinned first, the rest alphabetical: stable regardless of the - # shuffled execution order - pinned = ['usbtest', 'cdc_msc_throughput', 'msc_file_explorer', 'msc_file_explorer_freertos'] - - def col_key(t): - name = t.rsplit('/', 1)[-1] - return (pinned.index(name) if name in pinned else len(pinned), name, t) - - columns = sorted(seen, key=col_key) - headers = [c.rsplit('/', 1)[-1] for c in columns] + ['duration'] # bare example names - - def cell(cells, col): - v = cells.get(col) - if v is None: - return '' - return REPORT_CELL.get(v, v) # status symbol, or a metric string (e.g. speed) verbatim - - rows_vals = [(lbl, [cell(cells, c) for c in columns] + [dur or '']) - for lbl, cells, dur in rows_all] - board_hdr = 'Board' - board_w = max([len(board_hdr)] + [len(lbl) for lbl, _ in rows_vals]) - col_w = [max([len(h)] + [len(vals[i]) for _, vals in rows_vals]) - for i, h in enumerate(headers)] - - def line(label, values): - padded = [label.ljust(board_w)] + [v.center(w) for v, w in zip(values, col_w)] - return '| ' + ' | '.join(padded) + ' |' - - header = line(board_hdr, headers) - sep = '| ' + '-' * board_w + ' | ' + ' | '.join(':' + '-' * (w - 2) + ':' for w in col_w) + ' |' - body = [line(lbl, vals) for lbl, vals in rows_vals] - - # tally run cells (not-run cells are absent from the dicts). A cell is a bare status or - # a metric string carrying its own icon ("❌ 29/30"), so classify by the leading icon. - def cell_kind(v): - if v == 'fail' or (isinstance(v, str) and v.startswith(REPORT_CELL['fail'])): - return 'fail' - if v == 'skip' or (isinstance(v, str) and v.startswith(REPORT_CELL['skip'])): - return 'skip' - return 'pass' - kinds = [cell_kind(v) for _, cells, _ in rows_all for v in cells.values()] - failed = kinds.count('fail') - skipped = kinds.count('skip') - passed = kinds.count('pass') - summary = (f'**{REPORT_CELL["pass"]} {passed} passed · {REPORT_CELL["fail"]} {failed} failed · ' - f'{REPORT_CELL["skip"]} {skipped} skipped · blank not run**') - - return summary + '\n\n' + '\n'.join([header, sep] + body) - - def _write_failed_spec(failed_fname: Path, report_dir: Path, mret: list) -> None: """Re-run spec: only the failed boards (-b), each restricted to its own failed tests (-bt); a board with failures but no test list re-runs entirely. @@ -2083,87 +2014,13 @@ def _blind_note(mret: list) -> str: f'{", ".join(blind)}. See the usb-kernel-recover skill.\n') -def accumulate_report(mret: list, report_dir: Path, fresh: bool, scope: str = '', - banner: str = '') -> str: - """Merge this run's results into hil_report.json in report_dir, then (re)write - the markdown matrix to hil_report.md. `fresh` (a first run, no --accumulate) - starts a new report; otherwise a re-run accumulates so boards/tests that - already passed are preserved while re-run cells are updated. `scope` names the - board filter, if any, so a scoped table is not mistaken for a full one. - Returns the md.""" - acc = {} # ordered {row_label: [cells dict, duration str|None]} - prior_banner = '' - jpath = report_dir / REPORT_JSON - if not fresh and jpath.is_file(): - try: - saved = json.loads(jpath.read_text()) - # CI keys the report dir by run id, so the sidecar is from an earlier attempt - for entry in saved.get('rows', []): - acc[entry['board']] = [dict(entry['cells']), entry.get('duration')] - # ... and so is the caveat those cells were collected under. A rerun on a rig - # that has since recovered contributes no banner, and the .failed spec reruns - # only FAILURES -- so the earlier attempt's passes are never re-earned and - # would be published as clean results of a rig that was not. - prior_banner = saved.get('banner', '') - except (ValueError, KeyError, TypeError): - pass # corrupt/old sidecar: start fresh - - # current cells override prior for boards/tests that ran; a filtered run reports - # duration None, keeping the previous full-run value - for name, _, _, rows, *_ in mret: - if rows and not any('board-locked' in cells for _, cells, _ in rows): - # board ran for real: clear a stale lock-failure cell (its row is keyed by - # board name; test rows may be variant names) - stale = acc.get(name) - if stale is not None: - stale[0].pop('board-locked', None) - if not stale[0]: - # variant-keyed boards never repopulate the board-name row, so drop it - # or it renders as a blank ghost row - del acc[name] - for row_label, cells, dur in rows: - row = acc.setdefault(row_label, [{}, None]) - # the boundary cell is only ever written on failure, so a re-run of this - # variant that cleared the boundary must drop the previous attempt's ❌ - if BOUNDARY_CELL not in cells: - row[0].pop(BOUNDARY_CELL, None) - row[0].update(cells) - if dur is not None: - row[1] = dur - - report_dir.mkdir(parents=True, exist_ok=True) - # by LINE, deduped: attempts repeat the same caveat far more often than they add a new - # one, and three copies of the D-state note reads as three incidents - seen, merged = set(), [] - for line in (prior_banner + banner).splitlines(): - if line.strip() and line not in seen: - seen.add(line) - merged.append(line) - banner = '\n'.join(merged) + '\n' if merged else '' - jpath.write_text(json.dumps({'rows': [{'board': k, 'cells': c, 'duration': d} - for k, (c, d) in acc.items()], - 'banner': banner}, indent=2) + '\n') - - md = render_matrix([(k, c, d) for k, (c, d) in acc.items()]) - if scope: - # a scoped run's small table is otherwise indistinguishable from a full one, and - # it replaces the previous full table in the sticky PR comment - md = f'_Scoped run: {scope}. Boards/tests not listed were not run._\n\n' + md - # LAST, so it is outermost: a rig-health caveat outranks the table AND the scope note, - # and the top of the report is where hil/SKILL.md tells the agent to look for it. - if banner: - md = banner + '\n' + md - (report_dir / REPORT_MD).write_text(md + '\n', encoding='utf-8') - return md - - # containment paths print through hil_health._p: stdout may already be a dead pipe (a # dropped ssh session), and a BrokenPipeError there would skip os._exit _p = hil_health._p def _abandon_exit(pool, mgr, abandoned: bool, err_count: int, - report: Path | None = None) -> None: + report_dir: Path | None = None) -> None: """Free the runner when the pool could not be shut down. Returns only if not abandoned. Must run even while an exception is propagating: multiprocessing's atexit handler @@ -2199,24 +2056,11 @@ def _abandon_exit(pool, mgr, abandoned: bool, err_count: int, 'stay locked.', flush=True) # A report already written by accumulate_report says nothing about the abandon, and a # green table under a red job is how an agent ends up pasting it as this run's result. - # Prepend the caveat; best-effort, never at the cost of exiting. - if report is not None: - try: - if report.exists(): - # utf-8 explicitly (the cells are ✅/❌/⚪) and catch ValueError too: a torn - # report or a LANG=C locale raises UnicodeDecodeError -- NOT an OSError -- - # straight past os._exit, stranding the runner. - body = report.read_text(encoding='utf-8', errors='replace') - # Only when no banner is there yet, searched anywhere in the head rather - # than at char 0: write_timeout_report's banner must stay FIRST (its table - # is a PREVIOUS attempt's) and it puts the rig-health quote above itself. - if '**HIL run ab' not in body[:2000]: - report.write_text( - '**HIL run abandoned: the worker pool would not shut down.** The ' - 'table below was collected before the abandon; treat board ' - 'results as unverified.\n\n' + body, encoding='utf-8') - except (OSError, ValueError): - pass + # Set the caveat in the DOCUMENT -- prepending to the markdown alone left the sidecar, + # which is all hil_report.summarize() and therefore an agent ever sees, saying nothing. + # Best-effort, never at the cost of exiting. + if report_dir is not None: + hil_report.mark_report_abandoned(report_dir, 'the worker pool would not shut down.') try: sys.stdout.flush() except OSError: @@ -2305,13 +2149,11 @@ def main() -> None: print(msg, flush=True) # loud AND leaving evidence: exiting with no report at all lets the PR comment # keep the previous push's stale table under a red job - try: - rd = Path(os.environ.get('HIL_REPORT_DIR', '.')) - rd.mkdir(parents=True, exist_ok=True) - (rd / REPORT_MD).write_text(f'**HIL run selected no boards.** {msg}\n', - encoding='utf-8') - except OSError: - pass + rd = Path(os.environ.get('HIL_REPORT_DIR', '.')) + # fresh must be threaded through: this runs BEFORE the `if fresh:` wipe below, so + # defaulting it here wiped an --accumulate run's accumulated rows -- the exact + # regression the parameter exists to prevent. + hil_report.mark_report_no_boards(rd, msg, fresh=not args.accumulate) sys.exit(1) @@ -2387,14 +2229,14 @@ def main() -> None: # BEFORE Manager()/Pool(), not inside the try: hil_ci.sh reuses a persistent REMOTE_DIR # and scp's the report back unconditionally, so if a fork failure (OSError/EAGAIN right # after a convoy -- the case this whole block guards) skipped the wipe, the finally's - # _abandon_exit would prepend "HIL run abandoned" to the PREVIOUS run's table and + # _abandon_exit would stamp "HIL run abandoned" onto the PREVIOUS run's report and # publish last night's board results as this run's. Nothing is live yet here, so an # OSError from the wipe itself just exits with its traceback -- it cannot strand the # interpreter in multiprocessing's unbounded atexit join, which is what deferring it # was protecting against. if fresh: report_dir.mkdir(parents=True, exist_ok=True) - for f in (REPORT_JSON, REPORT_MD): + for f in (hil_report.REPORT_JSON, hil_report.REPORT_MD): (report_dir / f).unlink(missing_ok=True) failed_fname.unlink(missing_ok=True) try: @@ -2446,9 +2288,9 @@ def main() -> None: f'are this run\'s; {len(stuck)} never reported and are NOT in ' f'the table: {", ".join(stuck)}. Re-run covers those.\n') try: - accumulate_report(mret, report_dir, fresh, '', + hil_report.accumulate_report(mret, report_dir, fresh, '', health_banner + _blind_note(mret) - + _stray_note(mret) + banner) + + _stray_note(mret), caveat=banner) except Exception as rerr: # noqa: BLE001 - the raise below must still happen # FALL BACK, do not just warn: accumulate_report can raise on an # unwritable/root-owned report dir or a torn JSON, and _abandon_exit @@ -2458,9 +2300,9 @@ def main() -> None: print(f'warning: partial report failed: {type(rerr).__name__}: {rerr}; ' f'falling back to the board list', flush=True) try: - hil_health.write_timeout_report( + hil_report.write_timeout_report( report_dir, [b for b in config_boards - if b['name'] in stuck], POOL_TIMEOUT, REPORT_MD, + if b['name'] in stuck], POOL_TIMEOUT, prefix=health_banner) except Exception as re2: # noqa: BLE001 print(f'warning: fallback report failed too: ' @@ -2485,9 +2327,9 @@ def main() -> None: f'{len(mret)} board(s) below finished and are this run\'s; ' f'{len(stuck)} did not report: {", ".join(stuck)}.\n') try: - accumulate_report(mret, report_dir, fresh, '', + hil_report.accumulate_report(mret, report_dir, fresh, '', health_banner + _blind_note(mret) - + _stray_note(mret) + banner) + + _stray_note(mret), caveat=banner) except Exception as re2: # noqa: BLE001 - the raise below must still happen print(f'warning: partial report failed: {type(re2).__name__}: {re2}', flush=True) @@ -2578,12 +2420,12 @@ def main() -> None: # looks exactly like a full run that happened to be small scoped = sorted(set(args.board) | set(board_test)) scope = f'{len(scoped)} board(s) — {", ".join(scoped)}' if scoped else '' - report = accumulate_report(mret, report_dir, fresh, scope, + report = hil_report.accumulate_report(mret, report_dir, fresh, scope, health_banner + _blind_note(mret) + _stray_note(mret)) print() print(report) - print(f'\nReport written to {(report_dir / REPORT_MD).resolve()}') + print(f'\nReport written to {(report_dir / hil_report.REPORT_MD).resolve()}') duration = time.time() - duration print() @@ -2594,7 +2436,7 @@ def main() -> None: # In the finally, not after: any raise above (accumulate_report sits outside the # OSError handler) would skip the abandon path and unwind into multiprocessing's # unbounded atexit join, hanging the runner. - _abandon_exit(pool, mgr, pool_abandoned, err_count, report_dir / REPORT_MD) + _abandon_exit(pool, mgr, pool_abandoned, err_count, report_dir) # Same clamp: exit status is a byte either way, so 256 failures would report green. sys.exit(min(err_count, 125)) diff --git a/test/hil/test/test_ci_select.py b/test/hil/test/test_ci_select.py index c34bccd1f..f3e000cb4 100644 --- a/test/hil/test/test_ci_select.py +++ b/test/hil/test/test_ci_select.py @@ -984,6 +984,7 @@ class TestTheHarnessTestsAreNotTheHarness(unittest.TestCase): 'test/hil/test/test_ci_select.py', 'test/hil/test/test_hil_bounded.py', 'test/hil/test/test_hil_health.py', + 'test/hil/test/test_hil_report.py', 'test/hil/test/test_hil_util.py', ], 'test/hil/test/ gained or lost a file; it is carved out of rule 2, so confirm ' 'the rig still does not read anything in there before updating this list') diff --git a/test/hil/test/test_hil_bounded.py b/test/hil/test/test_hil_bounded.py index c6d454f0e..715d520db 100644 --- a/test/hil/test/test_hil_bounded.py +++ b/test/hil/test/test_hil_bounded.py @@ -39,6 +39,7 @@ serial_stub.SerialTimeoutException = type('SerialTimeoutException', (Exception,) sys.modules.setdefault('serial', serial_stub) import hil_flash import hil_test +from helper import hil_report def write_script(path: Path, body: str) -> None: @@ -936,11 +937,17 @@ class RunWhileContract(unittest.TestCase): def boom(): raise AssertionError('x') + # a duration no other process would plausibly pick: `pgrep -f` searches the WHOLE + # machine, so a bare `sleep 20` matched an unrelated background job -- another + # agent session's retry loop, in the case that exposed this -- and failed a test + # about our own child. Observed failing 3/3 in isolation while that loop ran. + sentinel = '20.0451' with self.assertRaises(AssertionError): - self.hil_util.run_alongside(['sleep', '20'], boom, 1) + self.hil_util.run_alongside(['sleep', sentinel], boom, 1) # nothing of ours is left running: the reap ran on the error path too import subprocess - out = subprocess.run(['pgrep', '-f', '^sleep 20'], capture_output=True, text=True) + out = subprocess.run(['pgrep', '-f', f'^sleep {sentinel}'], + capture_output=True, text=True) seen['strays'] = [p for p in out.stdout.split() if p] self.assertEqual(seen['strays'], [], 'work() raising leaked the child') @@ -1146,10 +1153,16 @@ class AbandonExitSurvivesAFailedFork(unittest.TestCase): def test_none_pool_and_manager_still_write_the_banner(self): # a subprocess, because _abandon_exit ends in os._exit: in-process it would take # the test runner with it, before any assertion could run + import json import subprocess with TemporaryDirectory() as td: - report = Path(td) / 'hil_report.md' - report.write_text('| board | test |\n|---|---|\n', encoding='utf-8') + rd = Path(td) + # it takes the report DIRECTORY now and re-renders both artifacts from the + # sidecar, so seed the sidecar -- the markdown is output, not input + (rd / 'hil_report.json').write_text(json.dumps( + {'rows': [{'board': 'boardA', 'cells': {'cdc_msc': 'pass'}, + 'duration': '1s'}], + 'banner': '', 'scope': '', 'caveat': ''})) src = ( 'import sys, types\n' f'sys.path.insert(0, {str(Path(TEST_DIR).parents[0])!r})\n' @@ -1160,12 +1173,14 @@ class AbandonExitSurvivesAFailedFork(unittest.TestCase): 'sys.modules.setdefault("serial", st)\n' 'import hil_test\n' f'hil_test._abandon_exit(None, None, True, 1, __import__("pathlib")' - f'.Path({str(report)!r}))\n') + f'.Path({str(rd)!r}))\n') r = subprocess.run([sys.executable, '-c', src], capture_output=True, text=True, timeout=120) self.assertEqual(r.returncode, 1, r.stderr) - self.assertTrue(report.read_text().startswith('**HIL run abandoned'), - 'the abandon banner never reached the report') + self.assertTrue((rd / 'hil_report.md').read_text().startswith( + '**HIL run abandoned'), 'the abandon banner never reached the report') + self.assertIn('abandoned', + json.loads((rd / 'hil_report.json').read_text())['caveat']) def test_kill_pool_children_tolerates_a_pool_that_never_existed(self): from helper import hil_health @@ -1647,124 +1662,6 @@ class StagingCoversEveryBoardForm(unittest.TestCase): "last run's re-run spec survived a green run") -class SummaryFoldsReportToBoards(unittest.TestCase): - """hil_summary.py replaces the agent retyping the markdown table. Report rows are named per - VARIANT and a variant need not start with the board name, so the config is what maps them - back -- the previous string-matching design produced a defect in each of four review rounds.""" - - def _sum(self, boards, rows, cfg_boards=None, banner=''): - import json - import subprocess - td = TemporaryDirectory() - self.addCleanup(td.cleanup) - d = Path(td.name) - (d / 'hil_report.json').write_text(json.dumps( - {'rows': [{'board': b, 'cells': c, 'duration': '1s'} for b, c in rows], - 'banner': banner})) - cfg = d / 'cfg.json' - cfg.write_text(json.dumps({'boards': cfg_boards or [{'name': b} for b in boards]})) - args = [a for b in boards for a in ('-b', b)] - r = subprocess.run(['python3', str(Path(TEST_DIR).parents[0] / 'helper' / 'hil_summary.py'), - str(cfg), *args, '--report-dir', str(d)], - capture_output=True, text=True, timeout=60) - self.assertEqual(r.returncode, 0, r.stderr) - return json.loads(r.stdout)['results'] - - def test_variant_rows_fold_onto_their_board(self): - """nanoch32v203 never produces a row named after the board.""" - got = self._sum(['nanoch32v203'], - [('nanoch32v203-fsdev', {'usbtest': 'pass'}), - ('nanoch32v203-usbfs', {'usbtest': 'pass'})], - cfg_boards=[{'name': 'nanoch32v203', - 'variant': [{'name': 'nanoch32v203-fsdev'}, - {'name': 'nanoch32v203-usbfs'}]}]) - self.assertEqual([r['board'] for r in got], ['nanoch32v203']) - self.assertTrue(got[0]['pass']) - self.assertTrue(got[0]['ran']) - - def test_one_failing_variant_fails_the_board(self): - got = self._sum(['nano'], - [('nano-a', {'usbtest': 'pass'}), ('nano-b', {'usbtest': '❌ 29/30'})], - cfg_boards=[{'name': 'nano', 'variant': [{'name': 'nano-a'}, - {'name': 'nano-b'}]}]) - self.assertFalse(got[0]['pass']) - self.assertIn('29/30', got[0]['detail']) - - def test_lock_contention_is_a_field_not_a_prefix(self): - got = self._sum(['alpha'], [('alpha', {'board-locked': 'fail'})]) - self.assertTrue(got[0]['locked']) - self.assertFalse(got[0]['pass']) - - def test_a_board_with_no_row_is_marked_not_run(self): - got = self._sum(['alpha', 'beta'], [('alpha', {'usbtest': 'pass'})]) - self.assertTrue(got[0]['ran']) - self.assertFalse(got[1]['ran']) - self.assertFalse(got[1]['pass']) - - def test_a_metric_cell_counts_by_its_icon(self): - got = self._sum(['a', 'b'], [('a', {'cdc_msc_throughput': '✅ C 1.2 M 3.4'}), - ('b', {'cdc_msc_throughput': '❌ C 0.0 M 0.0'})]) - self.assertTrue(got[0]['pass']) - self.assertFalse(got[1]['pass']) - - def test_skipped_cells_do_not_fail_a_board(self): - got = self._sum(['a'], [('a', {'usbtest': 'skip', 'cdc_msc': 'pass'})]) - self.assertTrue(got[0]['pass']) - - def test_a_plain_metric_cell_is_a_pass(self): - """Mirrors hil_test.py's own tally (cell_kind): failures are ALWAYS marked -- 'fail' - or a ❌ prefix, per TestFail's docstring -- while a passing test may return a plain - metric string that lands in the cell unprefixed. Treating unknown shapes as fail - would publish a green table as a red verdict.""" - got = self._sum(['a'], [('a', {'device_speed': '480.0 MBps'})]) - self.assertTrue(got[0]['pass']) - - def test_a_declared_variant_of_another_board_is_not_stolen(self): - """A declared variant need not start with its own board's name, so it may start with - a DIFFERENT board's name plus '-'. The prefix fallback must not attribute it twice.""" - got = self._sum(['alpha', 'beta'], - [('beta-x', {'usbtest': 'fail'})], - cfg_boards=[{'name': 'alpha', 'variant': [{'name': 'beta-x'}]}, - {'name': 'beta'}]) - self.assertTrue(got[0]['ran']) - self.assertFalse(got[0]['pass']) - self.assertFalse(got[1]['ran'], "beta must not inherit alpha's row") - - -class CaveatSurvivesAccumulate(unittest.TestCase): - """CI reruns with --accumulate: the sidecar keeps every earlier attempt's cells, but the - banner was recomputed per attempt. A first attempt on a degraded rig and a clean rerun - therefore published the degraded attempt's PASSES with no caveat on them -- and the - generated .failed spec reruns only failures, so those cells are never re-earned.""" - - def _rows(self, board, cell): - return [(board, 0, 0, [(board, {cell: 'OK'}, '1s')], 0)] - - def test_an_earlier_attempts_caveat_is_still_on_the_report(self): - td = TemporaryDirectory() - self.addCleanup(td.cleanup) - rd = Path(td.name) - banner = '> **Rig note.** 2 process(es) in D state at start.\n' - - hil_test.accumulate_report(self._rows('boardA', 'cdc_msc'), rd, True, '', banner) - self.assertIn('Rig note', (rd / hil_test.REPORT_MD).read_text()) - - # the rerun: clean rig, so this attempt contributes no banner of its own - md = hil_test.accumulate_report(self._rows('boardB', 'cdc_msc'), rd, False, '', '') - self.assertIn('boardA', md) # the earlier cells are kept ... - self.assertIn('Rig note', md, - 'the caveat the earlier cells were collected under was dropped') - - def test_the_same_caveat_twice_is_not_stacked(self): - td = TemporaryDirectory() - self.addCleanup(td.cleanup) - rd = Path(td.name) - banner = '> **Rig note.** 2 process(es) in D state at start.\n' - hil_test.accumulate_report(self._rows('boardA', 'cdc_msc'), rd, True, '', banner) - md = hil_test.accumulate_report(self._rows('boardB', 'cdc_msc'), rd, False, '', banner) - self.assertEqual(md.count('Rig note'), 1) - - class BlindWorkerReachesTheReport(unittest.TestCase): """A worker that exhausts its bounded-read budget answers SYSFS_UNKNOWN for every attribute, so its "device not found" means "could not tell". That reached the log and @@ -1796,7 +1693,7 @@ class BlindWorkerReachesTheReport(unittest.TestCase): wide = ('boardA', 1, ['device/cdc_msc'], [('boardA', {'cdc_msc': '❌'}, '2s')], 2.0, True) narrow = ('stuck', 1, [], None, 0) # what the timeout path builds hil_test._write_failed_spec(rd / 'x.failed', rd, [wide, narrow]) - md = hil_test.accumulate_report([wide], rd, True, '', hil_test._blind_note([wide])) + md = hil_report.accumulate_report([wide], rd, True, '', hil_test._blind_note([wide])) self.assertIn('boardA', md) self.assertIn('not all verdicts are evidence', md.lower()) diff --git a/test/hil/test/test_hil_health.py b/test/hil/test/test_hil_health.py index 5695cad6d..c7864f568 100644 --- a/test/hil/test/test_hil_health.py +++ b/test/hil/test/test_hil_health.py @@ -467,49 +467,6 @@ class KillPoolChildren(PatchCase): self.assertEqual(hil_health.kill_pool_children(NoPool()), 0) -class WriteTimeoutReport(unittest.TestCase): - def test_prefix_carries_the_preflight_diagnosis(self): - """The timeout aborts before accumulate_report, so without the prefix the artifact - and the PR comment lose the one line saying WHY the pool never finished.""" - with TemporaryDirectory() as td: - d = Path(td) - hil_health.write_timeout_report(d, [{'name': 'b1'}], 4200, 'r.md', - prefix='> **wedged usb_hub_wq worker.**\n') - out = (d / 'r.md').read_text() - self.assertTrue(out.startswith('> **wedged usb_hub_wq worker.**')) - self.assertIn('timed out after 4200s', out) - self.assertIn('- b1', out) - - def test_writes_a_report_where_there_would_be_none(self): - with TemporaryDirectory() as td: - hil_health.write_timeout_report(Path(td), [{'name': 'ra6m5_ek'}], 4200, - 'hil_report.md') - md = (Path(td) / 'hil_report.md').read_text() - self.assertIn('4200s', md) - self.assertIn('ra6m5_ek', md) - - def test_keeps_a_previous_attempts_table(self): - with TemporaryDirectory() as td: - path = Path(td) / 'hil_report.md' - path.write_text('| board | cdc_msc |\n') - hil_health.write_timeout_report(Path(td), [{'name': 'b1'}], 4200, 'hil_report.md') - md = path.read_text() - self.assertIn('abandoned', md) - self.assertIn('| board | cdc_msc |', md) - self.assertLess(md.index('abandoned'), md.index('| board |')) - - def test_custom_banner_is_used(self): - with TemporaryDirectory() as td: - hil_health.write_timeout_report(Path(td), [], 0, 'hil_report.md', - banner='**refused to start.**\n') - self.assertIn('refused to start', (Path(td) / 'hil_report.md').read_text()) - - def test_unwritable_dir_does_not_raise(self): - """The caller may be about to os._exit; losing the report must not also lose the - exit path.""" - hil_health.write_timeout_report(Path('/proc/nonexistent/nope'), [], 0, 'x.md') - - class WorkerSweepsItsOwnChildren(unittest.TestCase): """maxtasksperchild=1 makes a worker exit the moment its task returns, so by the time main()'s finally sweeps, the strays have been reparented to init and are off the pool's diff --git a/test/hil/test/test_hil_report.py b/test/hil/test/test_hil_report.py new file mode 100644 index 000000000..9ab39bde3 --- /dev/null +++ b/test/hil/test/test_hil_report.py @@ -0,0 +1,1162 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +# Unit tests for the report document: the vocabulary, the one cell classifier, rendering, +# the four writers, and the fold to per-board verdicts. Split out of test_hil_bounded.py +# and test_hil_health.py when the report code moved into helper/hil_report.py. +# Run directly: +# python3 test/hil/test/test_hil_report.py +import json +import os +import subprocess +import sys +import unittest +from pathlib import Path +from tempfile import TemporaryDirectory + +TEST_DIR = os.path.dirname(os.path.abspath(__file__)) +HIL_DIR = os.path.dirname(TEST_DIR) +sys.path.insert(0, HIL_DIR) + +from helper import hil_report + + +class OneClassifierForBothArtifacts(unittest.TestCase): + """The markdown tally and the agent's verdict used to classify cells with two separate + copies of one rule -- hil_test's cell_kind against REPORT_CELL, and hil_summary's + cell_state against its own re-typed '❌'/'⚪' literals. Change the icons and the table + and the verdict silently disagree.""" + + def test_bare_states(self): + self.assertEqual(hil_report.cell_state('fail'), 'fail') + self.assertEqual(hil_report.cell_state('skip'), 'skip') + self.assertEqual(hil_report.cell_state('pass'), 'pass') + + def test_icon_prefixed_metrics_carry_their_verdict(self): + self.assertEqual(hil_report.cell_state(f'{hil_report.REPORT_CELL["fail"]} 29/30'), 'fail') + self.assertEqual(hil_report.cell_state(f'{hil_report.REPORT_CELL["skip"]} board wedged'), + 'skip') + + def test_an_unprefixed_metric_is_a_pass(self): + """Load-bearing: a passing test may return a plain metric string. Classifying + unknown shapes as fail would publish a green table as a red verdict.""" + self.assertEqual(hil_report.cell_state('480.0 MBps'), 'pass') + self.assertEqual(hil_report.cell_state('1103 KB/s'), 'pass') + + def test_a_non_string_cell_does_not_raise(self): + """render_matrix's copy guarded with isinstance; hil_summary's did not, because its + caller str()'d first. The merged one keeps the guard -- it is the safer superset.""" + self.assertEqual(hil_report.cell_state(None), 'pass') + + def test_the_icons_come_from_REPORT_CELL(self): + """No second copy of the emoji anywhere in the module.""" + src = (Path(HIL_DIR) / 'helper' / 'hil_report.py').read_text(encoding='utf-8') + # CODE only: prose may quote an icon to explain a rule. The old assertion counted + # the single-quoted spelling `'❌'`, which a second copy written as "❌" would have + # sailed past. + code = '\n'.join(line.split('#', 1)[0] for line in src.splitlines()) + for icon in ('❌', '⚪', '✅'): + self.assertEqual(code.count(icon), 1, + f'{icon} is spelled in code more than once; REPORT_CELL is' + f' meant to be the one source') + + +class ModuleWorksImportedAndAsAScript(unittest.TestCase): + """It is imported as helper.hil_report by hil_test, and run as a script by the operator + (.claude/agents/hil-operator.md). A script run puts helper/ on sys.path, NOT test/hil, + so a plain `from helper import hil_health` breaks the CLI and only the CLI.""" + + def test_importable_as_a_package_module(self): + r = subprocess.run( + [sys.executable, '-c', + f'import sys; sys.path.insert(0, {HIL_DIR!r}); ' + f'from helper import hil_report; print(hil_report.REPORT_JSON)'], + capture_output=True, text=True, timeout=60) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn('hil_report.json', r.stdout) + + def test_runnable_as_a_script(self): + r = subprocess.run( + [sys.executable, str(Path(HIL_DIR) / 'helper' / 'hil_report.py'), '--help'], + capture_output=True, text=True, timeout=60) + self.assertEqual(r.returncode, 0, r.stderr) + + +class RenderReportIsPureFunctionOfTheDocument(unittest.TestCase): + """Four writers used to compose the markdown independently, so a table could carry + something the sidecar did not. One renderer, and the ordering it guarantees, is what + stops that -- pinned here rather than left to the order of three concatenations.""" + + def _doc(self, **kw): + d = {'rows': [{'board': 'boardA', 'cells': {'cdc_msc': 'pass'}, 'duration': '1s'}], + 'banner': '', 'scope': '', 'caveat': ''} + d.update(kw) + return d + + def test_table_comes_from_rows(self): + md = hil_report.render_report(self._doc()) + self.assertIn('boardA', md) + self.assertIn('cdc_msc', md) + + def test_scope_note_appears_above_the_table(self): + md = hil_report.render_report(self._doc(scope='-b boardA')) + self.assertLess(md.index('Scoped run'), md.index('boardA')) + + def test_banner_outranks_the_scope_note(self): + md = hil_report.render_report(self._doc(scope='-b boardA', + banner='> **Rig dirty.** x\n')) + self.assertLess(md.index('Rig dirty'), md.index('Scoped run')) + + def test_caveat_is_outermost(self): + md = hil_report.render_report(self._doc(banner='> **Rig dirty.** x\n', + caveat='**HIL run abandoned.**\n')) + self.assertLess(md.index('abandoned'), md.index('Rig dirty')) + + def test_a_document_with_no_rows_still_renders(self): + md = hil_report.render_report(self._doc(rows=[])) + self.assertIn('No tests were run.', md) + + def test_a_malformed_row_does_not_raise(self): + """mark_report_abandoned renders a sidecar it did not write -- hil_ci.sh reuses a + persistent REMOTE_DIR, so it can be an older version's or a torn one -- and it runs + on the way to os._exit, where a KeyError hangs the runner in multiprocessing's + unbounded join() instead of freeing it.""" + md = hil_report.render_report(self._doc( + rows=[{'board': 'boardA', 'cells': {'cdc_msc': 'pass'}}, {'board': 'half'}, + {}])) + self.assertIn('boardA', md) # the intact row still renders ... + self.assertIn('half', md) # ... and a cell-less one becomes a blank row + + +class ScopeSurvivesInTheJson(unittest.TestCase): + """A scoped run's small table is indistinguishable from a full run that lost boards. + The markdown says so; the JSON did not, so any JSON consumer could not tell.""" + + def _rows(self, board, cell): + return [(board, 0, 0, [(board, {cell: 'OK'}, '1s')], 0)] + + def test_scope_is_recorded_in_the_sidecar(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report(self._rows('boardA', 'cdc_msc'), rd, True, + '-b boardA', '') + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertEqual(doc['scope'], '-b boardA') + + def test_an_unscoped_run_records_an_empty_scope(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report(self._rows('boardA', 'cdc_msc'), rd, True, '', '') + self.assertEqual(json.loads((rd / hil_report.REPORT_JSON).read_text())['scope'], '') + + +class EveryExitPathLeavesBothArtifacts(unittest.TestCase): + """summarize() builds an agent's verdicts from the JSON. A path that writes only + markdown reports the whole fleet as 'no report row' while a human sees the real story.""" + + def test_the_no_boards_exit_writes_json_too(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.write_report(rd, {'rows': [], 'banner': '', 'scope': '', + 'caveat': '**HIL run selected no boards.** why\n'}) + self.assertIn('selected no boards', (rd / hil_report.REPORT_MD).read_text()) + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertEqual(doc['rows'], []) + self.assertIn('selected no boards', doc['caveat']) + + def test_write_report_raises_so_its_callers_can_report_it(self): + """write_report is NOT best-effort. Swallowing the OSError made + write_timeout_report's _p warning and hil_test's fallback-of-the-fallback dead + code -- an unwritable report dir produced no artifact and no message.""" + with self.assertRaises(OSError): + hil_report.write_report(Path('/proc/nonexistent/nope'), + {'rows': [], 'banner': '', 'scope': '', 'caveat': 'x\n'}) + + def test_the_guarded_callers_still_do_not_raise(self): + """They are the ones on the way to os._exit, where a raise hangs the interpreter + in multiprocessing's unbounded join().""" + bad = Path('/proc/nonexistent/nope') + hil_report.mark_report_abandoned(bad, 'the worker pool would not shut down.') + hil_report.mark_report_no_boards(bad, 'filters intersected to nothing') + import io + from contextlib import redirect_stdout + with redirect_stdout(io.StringIO()): + hil_report.write_timeout_report(bad, [{'name': 'b1'}], 3600) + + +class AbandonNoticeLandsInBothArtifacts(unittest.TestCase): + """_abandon_exit did a text prepend on a file it had not written, so the caveat never + reached the JSON and an agent reading the sidecar saw a clean partial report under a + red job.""" + + def test_abandon_sets_the_caveat_not_just_the_markdown(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('boardA', 0, 0, [('boardA', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertIn('abandoned', doc['caveat']) + self.assertEqual(len(doc['rows']), 1, 'the finished board must survive') + md = (rd / hil_report.REPORT_MD).read_text() + self.assertLess(md.index('abandoned'), md.index('boardA')) + + def test_marking_a_missing_report_is_a_no_op(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + hil_report.mark_report_abandoned(Path(td.name), 'x') # must not raise + + def test_a_sidecar_with_a_malformed_row_still_gets_stamped(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_JSON).write_text(json.dumps( + {'rows': [{'board': 'boardA'}], 'banner': '', 'scope': '', 'caveat': ''})) + hil_report.mark_report_abandoned(rd, 'x') + self.assertIn('abandoned', + json.loads((rd / hil_report.REPORT_JSON).read_text())['caveat']) + self.assertIn('abandoned', (rd / hil_report.REPORT_MD).read_text()) + + def test_a_torn_sidecar_is_a_no_op(self): + """This runs while the interpreter is being torn down: a raise here hangs the + process in multiprocessing's unbounded join().""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_JSON).write_text('{ truncated mid-') + hil_report.mark_report_abandoned(rd, 'x') # must not raise + + def test_an_existing_abandon_caveat_is_not_overwritten(self): + """The pool-timeout path names the stuck boards and the rig-health verdict; this + one only knows the pool would not shut down. Whoever got there first wins -- + the guard _abandon_exit used to spell as "'**HIL run ab' not in body[:2000]".""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600, + prefix='> **wedged usb_hub_wq worker.**\n') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertIn('timed out after 3600s', doc['caveat']) + self.assertIn('wedged usb_hub_wq worker', doc['banner']) # rig health, not outcome + self.assertNotIn('would not shut down', doc['caveat']) + + +class CaveatSurvivesAccumulate(unittest.TestCase): + """CI reruns with --accumulate: the sidecar keeps every earlier attempt's cells, but the + banner was recomputed per attempt. A first attempt on a degraded rig and a clean rerun + therefore published the degraded attempt's PASSES with no caveat on them -- and the + generated .failed spec reruns only failures, so those cells are never re-earned.""" + + def _rows(self, board, cell): + return [(board, 0, 0, [(board, {cell: 'OK'}, '1s')], 0)] + + def test_an_earlier_attempts_caveat_is_still_on_the_report(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + banner = '> **Rig note.** 2 process(es) in D state at start.\n' + + hil_report.accumulate_report(self._rows('boardA', 'cdc_msc'), rd, True, '', banner) + self.assertIn('Rig note', (rd / hil_report.REPORT_MD).read_text()) + + # the rerun: clean rig, so this attempt contributes no banner of its own + md = hil_report.accumulate_report(self._rows('boardB', 'cdc_msc'), rd, False, '', '') + self.assertIn('boardA', md) # the earlier cells are kept ... + self.assertIn('Rig note', md, + 'the caveat the earlier cells were collected under was dropped') + + def test_the_same_caveat_twice_is_not_stacked(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + banner = '> **Rig note.** 2 process(es) in D state at start.\n' + hil_report.accumulate_report(self._rows('boardA', 'cdc_msc'), rd, True, '', banner) + md = hil_report.accumulate_report(self._rows('boardB', 'cdc_msc'), rd, False, '', banner) + self.assertEqual(md.count('Rig note'), 1) + + +class MarkdownIsAlwaysARenderingOfTheJson(unittest.TestCase): + """The property this whole change buys: whatever wrote the report, re-rendering the + sidecar reproduces the markdown byte for byte. Four writers, one renderer -- asserted + directly rather than inferred from the writers.""" + + def _check(self, rd): + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertEqual((rd / hil_report.REPORT_MD).read_text(), + hil_report.render_report(doc) + '\n') + + def test_normal_path(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('boardA', 0, 0, [('boardA', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, + '-b boardA', '> **Rig note.** x\n') + self._check(rd) + + def test_after_an_accumulate_rerun(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('boardA', 0, 0, [('boardA', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') + hil_report.accumulate_report( + [('boardB', 0, 0, [('boardB', {'cdc_msc': 'OK'}, '1s')], 0)], rd, False, '', '') + self._check(rd) + + def test_after_abandonment(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('boardA', 0, 0, [('boardA', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + self._check(rd) + + def test_no_boards_exit(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.write_report(rd, {'rows': [], 'banner': '', 'scope': '', + 'caveat': '**HIL run selected no boards.** why\n'}) + self._check(rd) + + def test_the_pool_guard_fallback(self): + """The last writer to join the invariant: it composed its own markdown only because + hil_health could not import the renderer.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('done', 0, 0, [('done', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600, + prefix='> **wedged usb_hub_wq worker.**\n') + self._check(rd) + +class WriteTimeoutReport(unittest.TestCase): + def test_prefix_carries_the_preflight_diagnosis(self): + """The timeout aborts before accumulate_report, so without the prefix the artifact + and the PR comment lose the one line saying WHY the pool never finished.""" + with TemporaryDirectory() as td: + d = Path(td) + hil_report.write_timeout_report(d, [{'name': 'b1'}], 4200, + prefix='> **wedged usb_hub_wq worker.**\n') + out = (d / hil_report.REPORT_MD).read_text() + # the abandon notice leads (run outcome), the rig-health prefix follows in the + # banner -- prefix used to be folded INTO the caveat, which is what made a clean + # --accumulate retry inherit an abandonment that had not happened + self.assertTrue(out.startswith('**HIL run abandoned: worker pool timed out'), out[:80]) + self.assertIn('> **wedged usb_hub_wq worker.**', out) + self.assertIn('timed out after 4200s', out) + self.assertIn('- b1', out) + + def test_writes_a_report_where_there_would_be_none(self): + with TemporaryDirectory() as td: + hil_report.write_timeout_report(Path(td), [{'name': 'ra6m5_ek'}], 4200) + md = (Path(td) / hil_report.REPORT_MD).read_text() + self.assertIn('4200s', md) + self.assertIn('ra6m5_ek', md) + + def test_the_prior_attempts_rows_survive(self): + """Was: the prior MARKDOWN TEXT survives below the banner. It now re-renders from + the merged sidecar, so the guarantee is stated against rows -- one table with the + stuck boards in it, rather than a banner stapled above a duplicate table.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('done', 0, 0, [('done', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600) + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertEqual([r['board'] for r in doc['rows']], ['done', 'stuck']) + md = (rd / hil_report.REPORT_MD).read_text() + self.assertIn('done', md) + self.assertIn('stuck', md) + self.assertIn('abandoned', md) + self.assertLess(md.index('abandoned'), md.index('done')) + self.assertEqual(md.count('| Board'), 1, 'the prior table was duplicated, not merged') + + def test_custom_banner_is_used(self): + with TemporaryDirectory() as td: + hil_report.write_timeout_report(Path(td), [], 0, + banner='**refused to start.**\n') + self.assertIn('refused to start', (Path(td) / hil_report.REPORT_MD).read_text()) + + def test_timeout_report_writes_the_sidecar(self): + """This path used to write markdown only, so summarize() -- which is all an + agent gets -- reported the whole fleet as 'no report row' on exactly the runs + that failed.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.write_timeout_report(rd, [{'name': 'boardA'}], 3600) + self.assertTrue((rd / hil_report.REPORT_JSON).is_file()) + self.assertIn('boardA', (rd / hil_report.REPORT_JSON).read_text()) + + def test_the_sidecar_keeps_a_previous_attempts_rows(self): + """An earlier attempt's finished boards are real results and this attempt has none + of its own, so the rows merge rather than replace.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_JSON).write_text(json.dumps( + {'rows': [{'board': 'done', 'cells': {'cdc_msc': 'pass'}, 'duration': '1s'}], + 'banner': '', 'scope': '', 'caveat': ''})) + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600) + rows = json.loads((rd / hil_report.REPORT_JSON).read_text())['rows'] + self.assertEqual([r['board'] for r in rows], ['done', 'stuck']) + + def test_a_torn_sidecar_does_not_lose_the_stuck_boards(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_JSON).write_text('{ truncated mid-') + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600) + rows = json.loads((rd / hil_report.REPORT_JSON).read_text())['rows'] + self.assertEqual([r['board'] for r in rows], ['stuck']) + + def test_a_roster_entry_without_a_name_does_not_escape(self): + """The broad handler exists to stop a KeyError here stranding the runner, but a + report that silently loses its only board is worse than one saying '?'.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.write_timeout_report(rd, [{}], 3600) + rows = json.loads((rd / hil_report.REPORT_JSON).read_text())['rows'] + self.assertEqual([r['board'] for r in rows], ['?']) + + def test_unwritable_dir_does_not_raise(self): + """The caller may be about to os._exit; losing the report must not also lose the + exit path.""" + hil_report.write_timeout_report(Path('/proc/nonexistent/nope'), [], 0) + + +class SummaryFoldsReportToBoards(unittest.TestCase): + """summarize() replaces the agent retyping the markdown table. Report rows are named per + VARIANT and a variant need not start with the board name, so the config is what maps them + back -- the previous string-matching design produced a defect in each of four review rounds.""" + + def _sum(self, boards, rows, cfg_boards=None, banner=''): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + d = Path(td.name) + (d / 'hil_report.json').write_text(json.dumps( + {'rows': [{'board': b, 'cells': c, 'duration': '1s'} for b, c in rows], + 'banner': banner})) + cfg = d / 'cfg.json' + cfg.write_text(json.dumps({'boards': cfg_boards or [{'name': b} for b in boards]})) + args = [a for b in boards for a in ('-b', b)] + r = subprocess.run(['python3', str(Path(TEST_DIR).parents[0] / 'helper' / 'hil_report.py'), + str(cfg), *args, '--report-dir', str(d)], + capture_output=True, text=True, timeout=60) + self.assertEqual(r.returncode, 0, r.stderr) + return json.loads(r.stdout)['results'] + + def test_variant_rows_fold_onto_their_board(self): + """nanoch32v203 never produces a row named after the board.""" + got = self._sum(['nanoch32v203'], + [('nanoch32v203-fsdev', {'usbtest': 'pass'}), + ('nanoch32v203-usbfs', {'usbtest': 'pass'})], + cfg_boards=[{'name': 'nanoch32v203', + 'variant': [{'name': 'nanoch32v203-fsdev'}, + {'name': 'nanoch32v203-usbfs'}]}]) + self.assertEqual([r['board'] for r in got], ['nanoch32v203']) + self.assertTrue(got[0]['pass']) + self.assertTrue(got[0]['ran']) + + def test_one_failing_variant_fails_the_board(self): + got = self._sum(['nano'], + [('nano-a', {'usbtest': 'pass'}), ('nano-b', {'usbtest': '❌ 29/30'})], + cfg_boards=[{'name': 'nano', 'variant': [{'name': 'nano-a'}, + {'name': 'nano-b'}]}]) + self.assertFalse(got[0]['pass']) + self.assertIn('29/30', got[0]['detail']) + + def test_lock_contention_is_a_field_not_a_prefix(self): + got = self._sum(['alpha'], [('alpha', {'board-locked': 'fail'})]) + self.assertTrue(got[0]['locked']) + self.assertFalse(got[0]['pass']) + + def test_a_board_with_no_row_is_marked_not_run(self): + got = self._sum(['alpha', 'beta'], [('alpha', {'usbtest': 'pass'})]) + self.assertTrue(got[0]['ran']) + self.assertFalse(got[1]['ran']) + self.assertFalse(got[1]['pass']) + + def test_a_metric_cell_counts_by_its_icon(self): + got = self._sum(['a', 'b'], [('a', {'cdc_msc_throughput': '✅ C 1.2 M 3.4'}), + ('b', {'cdc_msc_throughput': '❌ C 0.0 M 0.0'})]) + self.assertTrue(got[0]['pass']) + self.assertFalse(got[1]['pass']) + + def test_skipped_cells_do_not_fail_a_board(self): + got = self._sum(['a'], [('a', {'usbtest': 'skip', 'cdc_msc': 'pass'})]) + self.assertTrue(got[0]['pass']) + + def test_a_plain_metric_cell_is_a_pass(self): + """Mirrors hil_test.py's own tally (cell_kind): failures are ALWAYS marked -- 'fail' + or a ❌ prefix, per TestFail's docstring -- while a passing test may return a plain + metric string that lands in the cell unprefixed. Treating unknown shapes as fail + would publish a green table as a red verdict.""" + got = self._sum(['a'], [('a', {'device_speed': '480.0 MBps'})]) + self.assertTrue(got[0]['pass']) + + def test_a_declared_variant_of_another_board_is_not_stolen(self): + """A declared variant need not start with its own board's name, so it may start with + a DIFFERENT board's name plus '-'. The prefix fallback must not attribute it twice.""" + got = self._sum(['alpha', 'beta'], + [('beta-x', {'usbtest': 'fail'})], + cfg_boards=[{'name': 'alpha', 'variant': [{'name': 'beta-x'}]}, + {'name': 'beta'}]) + self.assertTrue(got[0]['ran']) + self.assertFalse(got[0]['pass']) + self.assertFalse(got[1]['ran'], "beta must not inherit alpha's row") + + + def test_the_caveat_reaches_the_agents_verdict(self): + """The abandon/no-boards notice lives in the document now, and this JSON is all an + agent gets -- dropping it here puts the caveat back where only a human sees it.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + d = Path(td.name) + (d / 'hil_report.json').write_text(json.dumps( + {'rows': [{'board': 'boardA', 'cells': {'cdc_msc': 'pass'}, 'duration': '1s'}], + 'banner': '', 'scope': '', + 'caveat': '**HIL run abandoned: the worker pool would not shut down.**\n'})) + cfg = d / 'cfg.json' + cfg.write_text(json.dumps({'boards': [{'name': 'boardA'}]})) + r = subprocess.run( + ['python3', str(Path(TEST_DIR).parents[0] / 'helper' / 'hil_report.py'), + str(cfg), '-b', 'boardA', '--report-dir', str(d)], + capture_output=True, text=True, timeout=60) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn('abandoned', json.loads(r.stdout)['caveat']) + + def test_an_older_sidecar_without_a_caveat_still_summarises(self): + got = self._sum(['boardA'], [('boardA', {'cdc_msc': 'pass'})]) + self.assertTrue(got[0]['pass']) + + def test_the_old_entry_point_is_gone(self): + """hil_summary.py's CLI moved here. A leftover file would keep working while + drifting from the module that now owns the fold.""" + self.assertFalse((Path(HIL_DIR) / 'helper' / 'hil_summary.py').exists()) + + +class AbandonStampIsNotDestructive(unittest.TestCase): + """mark_report_abandoned runs on the way to os._exit, on a report it did not write. + Every case here was a live regression found by review.""" + + def _doc(self, **kw): + d = {'rows': [{'board': 'OLD', 'cells': {'t': 'pass'}, 'duration': '9s'}], + 'banner': '', 'scope': '', 'caveat': ''} + d.update(kw) + return d + + def test_declining_to_stamp_does_not_republish_the_markdown(self): + """The guard skipped the caveat assignment but write_report ran anyway, so a + no-op call still overwrote THIS run's table with a re-render of an older sidecar.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_JSON).write_text(json.dumps(self._doc( + caveat='**HIL run abandoned: worker pool timed out after 3600s.**\n'))) + (rd / hil_report.REPORT_MD).write_text('THIS RUN table with boardX\n') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + self.assertEqual((rd / hil_report.REPORT_MD).read_text(), + 'THIS RUN table with boardX\n') + + def test_a_banner_borne_abandon_notice_also_wins(self): + """The pool-timeout path puts its notice in `banner` (hil_test.py:2300), not + `caveat`. SKILL.md gives the two notices OPPOSITE rules, so stamping the vaguer + one on top tells the agent to publish rows it is meant to discard.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('a', 0, 0, [('a', {'t': 'OK'}, '1s')], 0)], rd, True, '', '', + caveat='**HIL run abandoned: worker pool timed out after 3600s.** 2 never' + ' reported.\n') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + md = (rd / hil_report.REPORT_MD).read_text() + self.assertTrue(md.startswith('**HIL run abandoned: worker pool timed out'), md[:80]) + self.assertNotIn('would not shut down', md) + + def test_a_missing_sidecar_still_stamps_the_markdown(self): + """Master read the MARKDOWN and prepended unconditionally, so it always stamped. + pr_comment.yml cats only hil_report.md -- giving up here publishes a clean green + table under an abandoned, non-zero job.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_MD).write_text('**✅ 27 passed · ❌ 0 failed**\n') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + md = (rd / hil_report.REPORT_MD).read_text(encoding='utf-8') + self.assertIn('abandoned', md) + self.assertIn('27 passed', md) + + def test_a_torn_sidecar_still_stamps_the_markdown(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_JSON).write_text('{ truncated mid-') + (rd / hil_report.REPORT_MD).write_text('**✅ 27 passed**\n') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + self.assertIn('abandoned', (rd / hil_report.REPORT_MD).read_text(encoding='utf-8')) + + def test_the_wording_matches_the_skill_contract(self): + """SKILL.md pins this banner as 'the table below IS this run's ... Report the + results AND the abandonment'. Calling it 'partial' sends the agent to re-run + boards that already passed.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_JSON).write_text(json.dumps(self._doc())) + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + caveat = json.loads((rd / hil_report.REPORT_JSON).read_text())['caveat'] + self.assertNotIn('partial', caveat) + self.assertIn('unverified', caveat) + + +class WriteReportFailsLoudly(unittest.TestCase): + def test_a_render_failure_does_not_leave_a_committed_json(self): + """It wrote the JSON, then rendered. A render raise left the sidecar saying + 'abandoned' beside a markdown that still read as a clean green table -- breaking + the one invariant this module exists to hold.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_MD).write_text('STALE GREEN TABLE\n') + (rd / hil_report.REPORT_JSON).write_text(json.dumps( + {'rows': ['boardA'], 'banner': '', 'scope': '', 'caveat': ''})) + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + md = (rd / hil_report.REPORT_MD).read_text() + # either both moved or neither did -- never a sidecar the markdown contradicts + self.assertEqual('abandoned' in doc.get('caveat', ''), 'abandoned' in md, + 'the sidecar was committed without its markdown') + + def test_a_non_dict_row_does_not_cost_the_abandon_stamp(self): + """A row that is a bare string raised out of render_report, so the stamp was lost + entirely -- the failure mode this whole function exists to prevent.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_JSON).write_text(json.dumps( + {'rows': ['boardA', {'board': 'good', 'cells': {'t': 'pass'}}], + 'banner': '', 'scope': '', 'caveat': ''})) + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + md = (rd / hil_report.REPORT_MD).read_text() + self.assertIn('abandoned', md) + self.assertIn('good', md) + + def test_an_unwritable_dir_reaches_the_callers_warning(self): + """write_report swallowing OSError made write_timeout_report's broad handler -- + and hil_test's fallback-of-the-fallback -- dead code: no artifact, no message.""" + import io + from contextlib import redirect_stdout + buf = io.StringIO() + with redirect_stdout(buf): + hil_report.write_timeout_report(Path('/proc/nonexistent/nope'), + [{'name': 'b1'}], 3600, prefix='x\n') + self.assertIn('warning', buf.getvalue().lower(), 'the failure was silent') + + +class PoolTimeoutCellIsHonest(unittest.TestCase): + def test_a_stuck_board_with_a_prior_row_still_gets_the_cell(self): + """`not in done` skipped the cell for any board carrying an earlier attempt's row, + so a board that just ate the 60-minute guard summarized as pass:true.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('stm32f4', 0, 0, [('stm32f4', {'cdc_msc': 'OK'}, '1s')], 0)], rd, True, '', '') + hil_report.write_timeout_report(rd, [{'name': 'stm32f4'}], 3600) + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + verdict = hil_report.summarize({'boards': [{'name': 'stm32f4'}]}, ['stm32f4'], doc) + self.assertFalse(verdict['results'][0]['pass'], + 'a board that hung the pool was published as a pass') + + def test_a_clean_retry_clears_the_cell(self): + """accumulate_report clears stale board-locked and BOUNDARY_CELL cells but not + this one, so a board that passed clean on the retry stayed red forever.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600) + hil_report.accumulate_report( + [('stuck', 0, 0, [('stuck', {'cdc_msc': 'OK'}, '2s')], 0)], rd, False, '', '') + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertNotIn('pool-timeout', doc['rows'][0]['cells']) + verdict = hil_report.summarize({'boards': [{'name': 'stuck'}]}, ['stuck'], doc) + self.assertTrue(verdict['results'][0]['pass']) + + def test_a_torn_sidecar_does_not_destroy_an_intact_markdown(self): + """Re-rendering from an unusable sidecar threw away real results the human copy + still had. Master concatenated below its banner and kept them.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_MD).write_text('| Board | t |\n| a | OK |\n| b | OK |\n') + (rd / hil_report.REPORT_JSON).write_text('{ truncated') + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600) + md = (rd / hil_report.REPORT_MD).read_text() + self.assertIn('| a | OK |', md, "an earlier attempt's real results were destroyed") + self.assertIn('stuck', md) + + +class SummarizeSeesEveryRow(unittest.TestCase): + def test_board_name_rows_reach_a_variant_boards_verdict(self): + """hil_test writes lock-contention and pool-timeout rows keyed by BOARD name, but + variants_of returns only declared variant names -- so for nanoch32v203 and + ch32v307v_r1_1v0 those rows were invisible and a held lock published as a + hardware FAIL that hil-validate.js never retried.""" + cfg = {'boards': [{'name': 'nano', + 'variant': [{'name': 'nano-fsdev'}, {'name': 'nano-usbfs'}]}]} + doc = {'rows': [{'board': 'nano', 'cells': {'board-locked': 'fail'}, + 'duration': None}], 'banner': '', 'scope': '', 'caveat': ''} + r = hil_report.summarize(cfg, ['nano'], doc)['results'][0] + self.assertTrue(r['ran']) + self.assertTrue(r['locked'], 'a held lock was published as a hardware failure') + + def test_a_malformed_row_does_not_kill_the_cli(self): + """summarize is the one reader with no defense, and it is the only one an agent's + verdict depends on.""" + out = hil_report.summarize({'boards': [{'name': 'a'}]}, ['a'], + {'rows': [{'cells': {}}, {'board': 'a', + 'cells': {'t': 'pass'}}]}) + self.assertTrue(out['results'][0]['pass']) + + +class NoBoardsExitKeepsWhatRan(unittest.TestCase): + def test_it_does_not_wipe_an_accumulated_sidecar(self): + """Master wrote only markdown here, so the sidecar survived. Writing rows:[] + unconditionally makes an --accumulate rerun whose filters empty erase every + board that had already passed.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('a', 0, 0, [('a', {'t': 'OK'}, '1s')], 0)], rd, True, '', '') + hil_report.mark_report_no_boards(rd, 'No boards left after the flasher filter', + fresh=False) + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertEqual([r['board'] for r in doc['rows']], ['a']) + self.assertIn('selected no boards', doc['caveat']) + self.assertIn('selected no boards', (rd / hil_report.REPORT_MD).read_text()) + + +class TheMergeBehavioursAreActuallyPinned(unittest.TestCase): + """accumulate_report's docstring cites these three as the reason not to split it, yet + deleting any of them left the whole suite green. Mutation-verified.""" + + def test_a_cleared_boundary_drops_the_previous_attempts_mark(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('b', 0, 0, [('b-v', {hil_report.BOUNDARY_CELL: 'fail'}, '1s')], 0)], + rd, True, '', '') + hil_report.accumulate_report( + [('b', 0, 0, [('b-v', {'cdc_msc': 'OK'}, '2s')], 0)], rd, False, '', '') + cells = json.loads((rd / hil_report.REPORT_JSON).read_text())['rows'][0]['cells'] + self.assertNotIn(hil_report.BOUNDARY_CELL, cells) + + def test_a_board_that_really_ran_drops_its_stale_lock_cell(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('b', 0, 0, [('b', {hil_report.LOCKED_CELL: 'fail'}, None)], 0)], rd, True, '', '') + hil_report.accumulate_report( + [('b', 0, 0, [('b', {'cdc_msc': 'OK'}, '2s')], 0)], rd, False, '', '') + rows = json.loads((rd / hil_report.REPORT_JSON).read_text())['rows'] + self.assertEqual([r['board'] for r in rows], ['b']) + self.assertNotIn(hil_report.LOCKED_CELL, rows[0]['cells']) + + def test_a_filtered_rerun_keeps_the_previous_duration(self): + """A -t-filtered re-run reports duration None; blanking the column loses the only + record of how long the full run took.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('b', 0, 0, [('b', {'cdc_msc': 'OK'}, '119s')], 0)], rd, True, '', '') + hil_report.accumulate_report( + [('b', 0, 0, [('b', {'cdc_msc': 'OK'}, None)], 0)], rd, False, '', '') + self.assertEqual(json.loads( + (rd / hil_report.REPORT_JSON).read_text())['rows'][0]['duration'], '119s') + + +class TheFooterCountsAreNotSwapped(unittest.TestCase): + """SKILL.md tells the operator to paste the footer counts verbatim, and swapping the + failed/skipped tallies left the suite green.""" + + def test_each_kind_is_counted_under_its_own_label(self): + md = hil_report.render_matrix([ + ('b', {'p1': 'pass', 'p2': 'pass', 'f1': 'fail', + 's1': f'{hil_report.REPORT_CELL["skip"]} board wedged'}, '1s')]) + self.assertIn(f'{hil_report.REPORT_CELL["pass"]} 2 passed', md) + self.assertIn(f'{hil_report.REPORT_CELL["fail"]} 1 failed', md) + self.assertIn(f'{hil_report.REPORT_CELL["skip"]} 1 skipped', md) + + +class HilCiUploadsTheAccumulateMergeBase(unittest.TestCase): + """hil_ci.sh rm -rf's REMOTE_DIR at the start of every run, and accumulate_report + merges onto the sidecar in the run's cwd -- so without an upload a remote + `--accumulate` retry silently starts from nothing and its one-row table REPLACES the + full-fleet one. The copy-back at the end has always existed; the upload did not.""" + + def _gate(self, *args): + """Run the real gate block out of hil_ci.sh and return its ACCUMULATE verdict. + + Executed, not grepped: the previous pair of tests searched the source text and + stayed green when `if [ "$ACCUMULATE" = 1 ]` was mutated to `if true`, because the + comment block above it mentions --accumulate five times.""" + sh = (Path(HIL_DIR) / 'hil_ci.sh').read_text(encoding='utf-8') + a = sh.index('ACCUMULATE=$(python3 -') + b = sh.index(') || ACCUMULATE=0', a) + len(') || ACCUMULATE=0') + script = 'ARGS=("$@")\n' + sh[a:b] + '\necho "$ACCUMULATE"' + r = subprocess.run(['bash', '-c', script, '_', *args], + capture_output=True, text=True, timeout=60) + self.assertEqual(r.returncode, 0, r.stderr) + return r.stdout.strip() + + def test_every_spelling_argparse_accepts_is_detected(self): + """hil_test.py declares `-a, --accumulate`, so argparse also takes -av, -va, + --accum and --acc; hil-validate.js tells the operator to retry 'adding -v'.""" + for spelling in ('--accumulate', '-a', '-av', '-va', '--accum', '--acc'): + self.assertEqual(self._gate(spelling), '1', f'{spelling} was not detected') + + def test_a_run_without_it_is_not_treated_as_accumulate(self): + for spelling in ('-b', '-v', '--retry'): + self.assertEqual(self._gate(spelling), '0', f'{spelling} falsely detected') + + def test_the_sidecar_is_uploaded_and_gated(self): + sh = (Path(HIL_DIR) / 'hil_ci.sh').read_text(encoding='utf-8') + up = [ln for ln in sh.splitlines() + if 'scp' in ln and 'hil_report.json' in ln and '$REMOTE:' in ln] + self.assertTrue(up, 'nothing uploads hil_report.json; --accumulate has no merge base') + self.assertIn('if [ "$ACCUMULATE" = 1 ]', sh, 'the upload is not gated') + + def test_a_missing_merge_base_is_loud(self): + """The damage: --accumulate with nothing to merge onto succeeds and quietly + publishes a small table where a full one used to be.""" + warn = [ln for ln in (Path(HIL_DIR) / 'hil_ci.sh').read_text().splitlines() + if 'warning' in ln.lower() and 'accumulate' in ln.lower()] + self.assertTrue(warn, 'no warning when --accumulate has no local sidecar') + + +class RunOutcomeAndRigHealthAreSeparate(unittest.TestCase): + """`banner` describes the CONDITIONS cells were collected under, so it carries across a + retry. `caveat` describes how a RUN ENDED, so it must not: a clean retry that reports + an earlier attempt's abandonment tells the agent a green run failed.""" + + def test_a_clean_retry_drops_the_previous_abandon_notice(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report([('a', 0, 0, [('a', {'t': 'OK'}, '1s')], 0)], + rd, True, '', '') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + hil_report.accumulate_report([('a', 0, 0, [('a', {'t': 'OK'}, '2s')], 0)], + rd, False, '', '') + self.assertEqual(json.loads((rd / hil_report.REPORT_JSON).read_text())['caveat'], '') + + def test_rig_health_still_carries_across_the_retry(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.write_timeout_report(rd, [{'name': 's'}], 3600, + prefix='> **Rig note.** wedged\n') + hil_report.accumulate_report([('a', 0, 0, [('a', {'t': 'OK'}, '1s')], 0)], + rd, False, '', '') + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertIn('Rig note', doc['banner']) + + def test_a_second_attempts_abandon_is_recorded(self): + """_already_abandoned matched a notice carried forward from an EARLIER attempt, so + a genuinely new abandon wrote nothing and the run's own failure vanished.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report( + [('a', 0, 0, [('a', {'t': 'OK'}, '1s')], 0)], rd, True, '', + '> **Rig note.** x\n', + caveat='**HIL run abandoned: worker pool timed out after 3600s.**\n') + hil_report.accumulate_report([('a', 0, 0, [('a', {'t': 'OK'}, '2s')], 0)], + rd, False, '', '') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + self.assertIn('would not shut down', + json.loads((rd / hil_report.REPORT_JSON).read_text())['caveat']) + + +class AMalformedSidecarNeverCostsTheReport(unittest.TestCase): + """hil_ci.sh now uploads a sidecar as the merge base, so a non-conforming one is + reachable from outside the harness.""" + + def _write(self, rd, doc): + (rd / hil_report.REPORT_JSON).write_text(json.dumps(doc)) + + def test_a_null_banner_does_not_kill_a_successful_run(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + self._write(rd, {'rows': [{'board': 'a', 'cells': {'t': 'pass'}, 'duration': '1s'}], + 'banner': None, 'caveat': None, 'scope': ''}) + hil_report.accumulate_report([('b', 0, 0, [('b', {'t': 'OK'}, '1s')], 0)], + rd, False, '', '') + self.assertTrue((rd / hil_report.REPORT_MD).is_file()) + + def test_a_null_cells_row_still_gets_its_pool_timeout_cell(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + self._write(rd, {'rows': [{'board': 'boardA', 'cells': None, 'duration': '61s'}], + 'banner': '', 'caveat': '', 'scope': ''}) + hil_report.write_timeout_report(rd, [{'name': 'boardA'}], 3600) + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + v = hil_report.summarize({'boards': [{'name': 'boardA'}]}, ['boardA'], doc) + self.assertFalse(v['results'][0]['pass'], + 'a board that ate the whole pool guard was published as a pass') + + def test_an_awkward_sidecar_still_gets_the_abandon_stamp(self): + """Any raise inside the dict branch was swallowed and the markdown fallback was + unreachable, so the stamp was lost from BOTH artifacts.""" + for bad in ({'rows': [{'board': 'a', 'cells': {'t': 'p'}, 'duration': 120}], + 'banner': '', 'caveat': '', 'scope': ''}, + {'rows': [{'board': 'a', 'cells': {'t': ['x']}, 'duration': '1s'}], + 'banner': None, 'caveat': '', 'scope': ''}): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + self._write(rd, bad) + (rd / hil_report.REPORT_MD).write_text('**✅ 27 passed**\n') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + self.assertIn('abandoned', (rd / hil_report.REPORT_MD).read_text(encoding='utf-8'), + f'no stamp for {bad}') + + def test_a_malformed_roster_entry_still_leaves_an_artifact(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + import io + from contextlib import redirect_stdout + with redirect_stdout(io.StringIO()): + hil_report.write_timeout_report(rd, ['plainstring'], 3600) + self.assertTrue((rd / hil_report.REPORT_MD).is_file(), 'no artifact at all') + + +class PoolTimeoutOutranksAStaleLock(unittest.TestCase): + def test_a_wedge_is_not_published_as_lock_contention(self): + """locked was computed across every cell and short-circuited detail, so a board + that wedged the rig on the retry was reported as LOCKED -- and hil-validate.js + re-runs those, paying another pool guard on a board that just hung it.""" + doc = {'rows': [{'board': 'boardX', + 'cells': {'board-locked': 'fail', 'pool-timeout': 'fail'}, + 'duration': None}], 'banner': '', 'caveat': '', 'scope': ''} + r = hil_report.summarize({'boards': [{'name': 'boardX'}]}, ['boardX'], doc)['results'][0] + self.assertFalse(r['locked'], 'a wedge was published as lock contention') + self.assertFalse(r['pass']) + + +class NoBoardsExitRespectsFreshness(unittest.TestCase): + def test_a_fresh_run_does_not_republish_the_previous_rows(self): + """It is called BEFORE the fresh wipe, so it re-published last run's green table + under this run's red job -- the stale-table failure it exists to prevent.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report([('a', 0, 0, [('a', {'t': 'OK'}, '1s')], 0)], + rd, True, '', '') + hil_report.mark_report_no_boards(rd, 'filters emptied', fresh=True) + self.assertEqual(json.loads((rd / hil_report.REPORT_JSON).read_text())['rows'], []) + + def test_an_accumulate_run_keeps_them(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report([('a', 0, 0, [('a', {'t': 'OK'}, '1s')], 0)], + rd, True, '', '') + hil_report.mark_report_no_boards(rd, 'filters emptied', fresh=False) + self.assertEqual([r['board'] for r in json.loads( + (rd / hil_report.REPORT_JSON).read_text())['rows']], ['a']) + + def test_it_does_not_overwrite_an_abandon_notice(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report([('a', 0, 0, [('a', {'t': 'OK'}, '1s')], 0)], + rd, True, '', '') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + hil_report.mark_report_no_boards(rd, 'filters emptied', fresh=False) + self.assertIn('abandoned', + json.loads((rd / hil_report.REPORT_JSON).read_text())['caveat']) + + +class TheNoBoardsCallSiteIsWired(unittest.TestCase): + """The fresh/accumulate branches of mark_report_no_boards were tested by calling it + DIRECTLY, so both passed while hil_test.py's one real call site never passed the flag + at all -- an --accumulate run whose filter emptied still wiped the accumulated rows. + This drives hil_test.py itself; the no-boards exit needs only a config and a filter + that matches nothing, so it costs no hardware.""" + + def _run(self, *extra): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / 'cfg.json').write_text(json.dumps( + {'boards': [{'name': 'alpha', 'uid': '1', 'flasher': {'name': 'jlink', 'uid': '2'}}]})) + (rd / hil_report.REPORT_JSON).write_text(json.dumps( + {'rows': [{'board': 'earlier', 'cells': {'t': 'pass'}, 'duration': '1s'}], + 'banner': '', 'scope': '', 'caveat': ''})) + r = subprocess.run( + [sys.executable, str(Path(HIL_DIR) / 'hil_test.py'), + '--flasher', 'nonexistent', *extra, str(rd / 'cfg.json')], + capture_output=True, text=True, timeout=120, + env={**os.environ, 'HIL_REPORT_DIR': str(rd)}) + self.assertEqual(r.returncode, 1, r.stdout + r.stderr) + return json.loads((rd / hil_report.REPORT_JSON).read_text()) + + def test_an_accumulate_run_keeps_the_accumulated_rows(self): + doc = self._run('--accumulate') + self.assertEqual([r['board'] for r in doc['rows']], ['earlier'], + "the call site did not pass fresh=not args.accumulate") + self.assertIn('selected no boards', doc['caveat']) + + def test_a_fresh_run_does_not_republish_them(self): + doc = self._run() + self.assertEqual(doc['rows'], []) + self.assertIn('selected no boards', doc['caveat']) + + +class EveryWriterRendersBeforeItCommits(unittest.TestCase): + def test_accumulate_report_does_not_commit_json_then_fail_to_render(self): + """accumulate_report hand-rolled the write instead of calling write_report, so a + render failure left the sidecar ahead of the markdown -- the exact ordering + write_report's docstring forbids.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_JSON).write_text(json.dumps( + {'rows': [{'board': 'boardA', 'cells': {'t': 'pass'}, 'duration': 119.0}], + 'banner': '', 'caveat': '', 'scope': ''})) + hil_report.accumulate_report([('boardB', 0, 0, [('boardB', {'t': 'OK'}, '1s')], 0)], + rd, False, '', '') + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertEqual((rd / hil_report.REPORT_MD).read_text(), + hil_report.render_report(doc) + '\n') + + +class MissingSidecarDoesNotDestroyTheMarkdown(unittest.TestCase): + def test_an_absent_sidecar_keeps_the_prior_table(self): + """`recovered` was only cleared when the sidecar was TORN, not when it was absent + -- reachable from hil_ci.sh's asymmetric copy-back and build.yml's skip marker.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / hil_report.REPORT_MD).write_text('| Board | t |\n| a | OK |\n| b | OK |\n') + hil_report.write_timeout_report(rd, [{'name': 'stuck'}], 3600) + self.assertIn('| a | OK |', (rd / hil_report.REPORT_MD).read_text()) + + +class LoadIsTheOnlyTrustBoundary(unittest.TestCase): + """hil_ci.sh uploads a sidecar as the merge base, so these shapes arrive from OUTSIDE + the harness. Every one of these raised past a handler before.""" + + def _seed(self, rd, raw): + (rd / hil_report.REPORT_JSON).write_text(raw if isinstance(raw, str) + else json.dumps(raw)) + + def test_a_non_list_rows_does_not_raise(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + self._seed(rd, {'rows': 1, 'banner': '', 'caveat': '', 'scope': ''}) + hil_report.accumulate_report([('a', 0, 0, [('a', {'t': 'OK'}, '1s')], 0)], + rd, False, '', '') + self.assertTrue((rd / hil_report.REPORT_MD).is_file()) + + def test_an_unhashable_cell_value_does_not_raise(self): + """render_matrix does REPORT_CELL.get(v, v); an unhashable value raised TypeError + on the NORMAL accumulate path.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + self._seed(rd, {'rows': [{'board': 'a', 'cells': {'t': ['x'], 'u': 'pass'}, + 'duration': '1s'}], + 'banner': '', 'caveat': '', 'scope': ''}) + hil_report.accumulate_report([('b', 0, 0, [('b', {'t': 'OK'}, '1s')], 0)], + rd, False, '', '') + cells = {r['board']: r['cells'] + for r in json.loads((rd / hil_report.REPORT_JSON).read_text())['rows']} + self.assertNotIn('t', cells['a'], 'a corrupt cell must drop, not become a pass') + self.assertIn('u', cells['a']) + + def test_summarize_survives_a_malformed_sidecar_via_load(self): + """The CLI is the one reader an agent's verdict depends on.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + (rd / 'cfg.json').write_text(json.dumps({'boards': [{'name': 'a'}]})) + self._seed(rd, {'rows': [{'board': 1, 'cells': 'notadict'}, + {'board': 'a', 'cells': {'t': 'pass'}}], + 'banner': '', 'caveat': '', 'scope': ''}) + r = subprocess.run( + [sys.executable, str(Path(HIL_DIR) / 'helper' / 'hil_report.py'), + str(rd / 'cfg.json'), '-b', 'a', '--report-dir', str(rd)], + capture_output=True, text=True, timeout=60) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertTrue(json.loads(r.stdout)['results'][0]['pass']) + + +class NoBoardsGuardOnlyAppliesWhenAccumulating(unittest.TestCase): + def test_a_fresh_run_carries_nothing_from_the_prior_sidecar(self): + """rows were reset on fresh but banner and scope were not, so a leftover or + uploaded sidecar republished a stale rig-health note and a stale scope line under + this run's notice.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report([('old', 0, 0, [('old', {'t': 'OK'}, '1s')], 0)], + rd, True, '3 board(s) — a, b, c', + '> **Rig note.** stale D-state holder\n') + hil_report.mark_report_no_boards(rd, 'filters emptied', fresh=True) + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertEqual(doc['rows'], []) + self.assertEqual(doc['banner'], '', 'a stale rig-health banner was republished') + self.assertEqual(doc['scope'], '', 'a stale scope note was republished') + self.assertNotIn('Rig note', (rd / hil_report.REPORT_MD).read_text()) + + def test_an_accumulate_run_keeps_banner_and_scope(self): + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report([('old', 0, 0, [('old', {'t': 'OK'}, '1s')], 0)], + rd, True, '3 board(s) — a, b, c', + '> **Rig note.** real\n') + hil_report.mark_report_no_boards(rd, 'filters emptied', fresh=False) + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertIn('Rig note', doc['banner']) + self.assertEqual([r['board'] for r in doc['rows']], ['old']) + + def test_a_fresh_run_is_not_blocked_by_a_prior_abandon(self): + """The guard runs BEFORE the fresh wipe, so guarding a fresh run left the previous + attempt's rows AND its abandon notice published as this run's.""" + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + rd = Path(td.name) + hil_report.accumulate_report([('old', 0, 0, [('old', {'t': 'OK'}, '1s')], 0)], + rd, True, '', '') + hil_report.mark_report_abandoned(rd, 'the worker pool would not shut down.') + hil_report.mark_report_no_boards(rd, 'filters emptied', fresh=True) + doc = json.loads((rd / hil_report.REPORT_JSON).read_text()) + self.assertEqual(doc['rows'], []) + self.assertIn('selected no boards', doc['caveat']) + + +if __name__ == '__main__': + unittest.main() -- cgit v1.3.1