summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorhathach <[email protected]>2026-08-20 16:43:49 +0700
committerhathach <[email protected]>2026-08-20 16:43:49 +0700
commit6905639b07c69fec68e9ebc77f7d27ac2775ee41 (patch)
tree2d46b0b6ae45b27df4874583c5830997df5829bd
parent7800876bf151a239046521232dc4603b156061be (diff)
hil: run every board in one hil_test.py and hand results across as JSON
hil-validate ran one hil-operator per board. That parallelizes at the wrong layer: hil_test.py already schedules boards across host controllers and budgets concurrent flashes and usbtest batteries per controller (FLASH_PARALLEL/USBTEST_PARALLEL), and those permits live in one process - N parallel runs multiply the budget onto the same uPD720201 cards for no wall-clock gain over one run that already parallelizes. The workflow now spawns ONE operator with every board as repeated -b. The operator no longer retypes the report table. Four consecutive max-effort review rounds found ~15 defects in this file and every one was in reconstructing board identity from transcribed prose: report rows are named per VARIANT (nanoch32v203 only ever produces -fsdev/-usbfs rows), a variant need not start with its board's name, lock contention is a `board-locked` cell rather than a phrase, and each fix introduced the next round's bug - including a fake-green test that asserted an invariant with the one input shape that could not break it. The new helper test/hil/helper/hil_summary.py does the join where the roster lives and emits one machine verdict per board ({board, ran, pass, locked, detail}); the operator returns that JSON verbatim plus `wedged`, the only field it authors, and the workflow reads fields, never parses a string. Its cell classifier mirrors hil_test.py's own tally exactly: failures are always marked ('fail' or a ❌ prefix, TestFail's contract), everything unmarked is a pass - a passing test may return a plain metric cell like '13443 KB/s', and the mirrored rule is what keeps a green table from becoming a red verdict. hil_ci.sh kept only the LAST -b, so multi-board remote runs staged one board's binaries and every other board died on the rig after its lock and flash slot were spent. It now parses every -b spelling argparse accepts (with the -bt arms ordered first, longest-match, so the <config>.failed retry form is never read as a board named "t..."), pre-flights roster membership and build dirs for ALL boards before anything is wiped or staged, warns per declared variant with no build dir (which hil_test.py would silently green-skip), forwards HIL_* knobs as export lines in one %q word the remote evals ('; '-joined so it round-trips under dash - an authorized HIL_NO_BOARD_LOCK force must not silently no-op), keeps HIL_REPORT_DIR local because the copy-backs look in REMOTE_DIR, and copies hil_report.json and the .failed re-run spec back beside the markdown, deleting stale local copies first so a green run cannot leave last run's spec looking current. Retries preserve the fleet: the documented path is the <config>.failed spec, which already begins with --accumulate; a fresh scoped re-run would unlink the report and collapse the whole-fleet table to the retried boards alone. The risky logic is executable, not argued about: .claude/workflows/test-hil-validate.mjs pins the lookup/verdict helpers and runs in pre-commit (hil-validate-logic); nine staging tests drive hil_ci.sh through an ssh stub that models the real thing (argv joined into one string the remote re-splits, heredoc on stdin - the naive echo-stub passed while the feature was broken); and deliberate mutations of the verdict logic are all caught. Validated on the rig: a 2-board run (usbtest 30/30 on both; the pre-fix classifier, replayed against that run's real report, fails the fully-green stm32f723disco on its two passing '13443 KB/s' cells), the .failed --accumulate retry (merged report kept every earlier row), and a 10-run soak over random subsets of a 22-board pool - 43 board-slots, every failure signature matched pre-existing CI state or known flake, zero tooling failures, no locks left behind.
-rw-r--r--.claude/agents/hil-operator.md49
-rw-r--r--.claude/skills/hil/SKILL.md18
-rw-r--r--.claude/workflows/hil-validate.js134
-rw-r--r--.claude/workflows/test-hil-validate.mjs59
-rw-r--r--.gitignore1
-rw-r--r--.pre-commit-config.yaml9
-rw-r--r--test/hil/helper/hil_summary.py115
-rw-r--r--test/hil/hil_ci.sh250
-rw-r--r--test/hil/test/test_hil_bounded.py317
9 files changed, 854 insertions, 98 deletions
diff --git a/.claude/agents/hil-operator.md b/.claude/agents/hil-operator.md
index ebc9251cc..a37501211 100644
--- a/.claude/agents/hil-operator.md
+++ b/.claude/agents/hil-operator.md
@@ -15,14 +15,22 @@ You operate physical USB test hardware. These repo skills are your source of tru
The GitHub Actions runner keeps running during your work. Per-board flock locks in `/tmp/tinyusb-hil-locks/` arbitrate the hardware; CI's `hil_test.py` fails fast on locked boards (re-runnable later).
-- `python3 test/hil/hil_test.py ...` runs: do NOT pre-hold those boards — `hil_test.py` self-locks each board for its flash+test and would fail fast with `board locked` against your own hold.
+- `python3 test/hil/hil_test.py ...` runs: do NOT pre-hold those boards — `hil_test.py` self-locks each board for its flash+test and would fail fast with `board locked` against your own hold. Several boards go into ONE run as repeated `-b`, never into several runs.
- ANY other hardware action (JLinkExe/openocd/GDB, manual flash, usbtest.py, serial poking): hold first, release when done — release is mandatory cleanup (a crashed holder auto-releases via kernel flock, but do not rely on it):
```bash
python3 test/hil/helper/hil_lock.py hold <board...> --reason "<task>"
# ... hardware work ...
python3 test/hil/helper/hil_lock.py release <board...>
```
-- Rig-wide operations (uhubctl power cycling, controller resets — they renumber buses): `python3 test/hil/helper/hil_lock.py hold --all --reason "<why>"` first.
+- Rig-wide operations — uhubctl power cycling, `usb_recover.sh root-cycle`, pci-rebind,
+ controller resets — need `python3 test/hil/helper/hil_lock.py hold --all --config <this host's config> --reason "<why>"`
+ first, even a single root-port bounce. `--all` is coarse for a bounce, but it is the only
+ correct reservation available: the affected siblings are sysfs busports, nothing maps a
+ busport to a board name (the pool check's topology report counts devices per subtree, it does
+ not name them), and `hil_lock.py hold` validates nothing against the roster — so passing it
+ `13-1.6` creates a lock file for a board that does not exist and reserves nothing while
+ reporting success. If `--all` cannot be taken, wait: a partial hold is worse than none,
+ because it reads as protection.
- If a lock is already held by someone else: report holder/reason (`hil_lock.py status`) — never force, never kill the holder. If the holder's reason is `hil_test.py`, that is a concurrent CI job mid-test on the board: waiting a few minutes and retrying once is appropriate when your task allows; otherwise return the holder info so the orchestrator can ask the user.
- You cannot ask the user anything. Bypassing a lock (`HIL_NO_BOARD_LOCK=1`, or proceeding with manual hardware work despite a held lock) is allowed ONLY when your prompt explicitly states the user authorized forcing.
@@ -33,11 +41,40 @@ The GitHub Actions runner keeps running during your work. Per-board flock locks
Bash tool's 10 min foreground cap: run it in the background and wait
for the completion notification. A foreground timeout kills the run before hil_test.py
can write its report. NEVER cancel early.
-- One hardware action at a time. You are never run concurrently with another hil-operator.
-- On test failure: retry once with `-v -r 1` appended (one verbose attempt for diagnosis; a usbtest battery that produced per-case verdicts is NOT auto-retried, so its result already stands). If a board/fixture stops enumerating, or a tool of YOURS hangs in D state, consult usb-kernel-recover and capture `dmesg | tail -50` into `detail`; set `wedged` true. A `> **Rig note.**` banner reporting someone else's D-state process is not that — see the hil skill's banner list.
+- One hardware action at a time. You are never run concurrently with another hil-operator, and a
+ multi-board `hil_test.py` run is ONE action: hand it every board as repeated `-b` and let it
+ schedule them — it round-robins boards across host controllers and budgets simultaneous flashes
+ and usbtest batteries per controller. Those budgets live in one process, so a second
+ `hil_test.py` alongside the first does not share them and the rig sees double the configured
+ width. (Do not read that as the cause of a dead card: hil_lock.py:128-131 records that every
+ observed uPD720201 death traced to a marginal DUT port bouncing under concurrent batteries,
+ and that lowering the widths does not fix a bad port — fix the port or pull the board.)
+- On test failure, retry ONCE, with `-v` for diagnosis. Retry from the spec the run just wrote —
+ `<config>.failed`, which already begins with `--accumulate` and restricts each board to its
+ failed tests via `-bt`. If you compose the retry by hand you MUST pass `--accumulate` yourself:
+ a fresh run unlinks the report, so a hand-scoped `-b <board>` retry replaces the whole-fleet
+ table with a one-row table. A usbtest battery that produced per-case verdicts is NOT auto-retried,
+ so its result already stands. If a board/fixture stops enumerating, or a tool of YOURS hangs in D
+ state, consult usb-kernel-recover and capture `dmesg | tail -50` into `detail`; set `wedged` true.
+ A `> **Rig note.**` banner reporting someone else's D-state process is not that — see the hil
+ skill's banner list.
## Output contract
-Your final message is parsed by a program. Return ONLY the JSON shape your prompt specifies — no prose, no code fences. Typical board-run shape:
+Your final message is parsed by a program. Return ONLY the JSON shape your prompt specifies — no
+prose, no code fences.
-{"board": "raspberry_pi_pico", "pass": true, "detail": "<per-test summary or first failure>", "wedged": false}
+For a board run, do NOT transcribe the report table. Run the tests, then hand back the machine
+output verbatim:
+
+```bash
+python3 test/hil/helper/hil_summary.py <config> -b BOARD [-b BOARD...] # from the report dir
+```
+
+`{"results": <its results array, verbatim>, "banner": <its banner, verbatim>, "wedged": ["board", ...]}`
+
+`results` and `banner` are copied, never retyped, reworded or re-ordered: report rows are named
+per variant, a variant name need not start with the board name, and lock contention is a cell
+rather than a phrase, so re-deriving any of it by hand is how this contract broke before.
+`wedged` is yours — the boards your run left unresponsive, usually none — and the only field you
+author.
diff --git a/.claude/skills/hil/SKILL.md b/.claude/skills/hil/SKILL.md
index 093f345b2..f0c449d33 100644
--- a/.claude/skills/hil/SKILL.md
+++ b/.claude/skills/hil/SKILL.md
@@ -32,7 +32,7 @@ python3 test/hil/helper/hil_lock.py release BOARD [BOARD...]
```
- Never pre-hold boards you are about to run `hil_test.py` on — it self-locks and would treat your own hold as a conflict.
-- Rig-wide operations (uhubctl power cycling, controller resets — bus renumbering) affect every board: `hil_lock.py hold --all --reason "..."` first.
+- Rig-wide operations (uhubctl power cycling, `usb_recover.sh root-cycle`, pci-rebind, controller resets — bus renumbering) affect every board: `hil_lock.py hold --all --config <this host's config> --reason "..."` first — `--all` defaults to `tinyusb.json`, so on `tusb` it would reserve 27 boards that do not exist there and none of the three that do. Even a single root-port bounce needs `--all`: nothing maps a sysfs busport to a board name, and `hil_lock.py hold` accepts any string, so a "just the siblings" hold reserves nothing while reporting success.
- `hil_lock.py status` lists holders. Locks auto-release when the holder process dies (kernel flock); `/tmp` clears on reboot.
- Forcing past a lock: `HIL_NO_BOARD_LOCK=1 python3 test/hil/hil_test.py ...` bypasses the guard without killing the holder. Only with the user's explicit go-ahead — they accept the risk of colliding with whatever holds the board.
@@ -81,7 +81,7 @@ Examples must be built for the target board(s) — see CLAUDE.md "Build" → "Al
## Arguments
-- **Board:** `-b BOARD_NAME` for one board; omit to run all boards in the config.
+- **Board:** `-b BOARD_NAME`, repeatable for a subset (`-b a -b b`); omit to run all boards in the config. Give a whole set to ONE run rather than one run per board: it schedules the boards across host controllers and budgets concurrent flashes and usbtest batteries per controller (`hil_lock.py` `FLASH_PARALLEL`/`USBTEST_PARALLEL`). Those permits are in-process semaphores — a second `hil_test.py` running alongside does not share them, it multiplies the load on the same xHCI cards.
- **Pass-through:** `-v`, `-r N`, etc. forwarded unchanged.
If `local.json` is missing on a dev PC, ask the user to supply one (only fall back to `tinyusb.json` if told to).
@@ -108,10 +108,13 @@ python3 test/hil/hil_test.py -b stm32f723disco -B examples "$CONFIG"
# All boards:
bash test/hil/hil_ci.sh
-# A single board, with pass-through flags:
-bash test/hil/hil_ci.sh -b raspberry_pi_pico2 -t host/cdc_msc_hid -r 1
+# A subset — repeat -b, ONE invocation for the whole set:
+bash test/hil/hil_ci.sh -b raspberry_pi_pico2 -b stm32f723disco -t host/cdc_msc_hid -r 1
```
+One invocation per board is wrong here, not merely slow: each run `rm -rf`s `REMOTE_DIR`
+and rewrites the report, so only the last board's rows survive.
+
Env overrides: `REMOTE`, `REMOTE_DIR`, `CONFIG`. Fails fast if the build dir/repo layout is missing.
## Timing
@@ -155,5 +158,8 @@ PREFIX, since each carries trailing detail and one is a blockquote:
"device not found" from the named boards means "could not tell". Do NOT report their red
cells as broken boards.
-On failure, retry with `-v`; if that's not enough, add temporary debug prints to
-`hil_test.py`.
+On failure, retry once with `-v` — from the `<config>.failed` spec the run just wrote, which
+already begins with `--accumulate` and restricts each board to its failed tests. A hand-scoped
+`-b <board>` retry MUST pass `--accumulate` too: a fresh run unlinks the report, replacing the
+whole-fleet table with a one-row table. If that is still not enough, add temporary debug prints
+to `hil_test.py`.
diff --git a/.claude/workflows/hil-validate.js b/.claude/workflows/hil-validate.js
index 136f9075e..bc0bda8b9 100644
--- a/.claude/workflows/hil-validate.js
+++ b/.claude/workflows/hil-validate.js
@@ -1,60 +1,134 @@
export const meta = {
name: 'hil-validate',
- description: 'Serialized hardware-in-the-loop run: flash+test each board with hil-operator; per-board flock locks arbitrate with concurrent CI (the actions-runner keeps running)',
- whenToUse: 'After validate passes, to exercise built firmware on the physical rig. Requires examples/cmake-build-<board> for each board. If the result has non-empty `locked`, ask the user: force (re-invoke with force: true), continue waiting (re-invoke later), or accept the partial result. Pass force: true ONLY with explicit user authorization.',
- phases: [{ title: 'HIL', detail: 'strictly serial per-board hil-operator runs' }],
+ description: 'Hardware-in-the-loop run: one hil-operator flashes and tests every board in a single hil_test.py run; per-board flock locks arbitrate with concurrent CI (the actions-runner keeps running)',
+ whenToUse: 'After validate passes, to exercise built firmware on the physical rig. Requires the boards to be built (examples/cmake-build-<board>, plus a dir per declared variant). If the result has non-empty `locked`, ask the user: force (re-invoke with force: true), continue waiting (re-invoke later), or accept the partial result. Pass force: true ONLY with explicit user authorization.',
+ phases: [{ title: 'HIL', detail: 'one hil-operator, every board in one hil_test.py run' }],
}
// args: { boards: string[], force?: boolean }
if (typeof args === 'string') { try { args = JSON.parse(args) } catch { /* not JSON: shape check below reports it */ } }
if (!args || !Array.isArray(args.boards) || args.boards.length === 0) {
- throw new Error('args must be { boards: string[], force? } with examples/cmake-build-<board> already built')
+ throw new Error('args must be { boards: string[], force? } with the boards already built')
}
+// The operator returns hil_summary.py's JSON verbatim plus its own observations. It does NOT
+// retype the report table: rows are named per variant, a variant need not start with the board
+// name, and lock contention is a cell rather than a phrase — rebuilding board identity from
+// prose produced a defect in each of four review rounds. hil_summary.py does that join against
+// the roster, so `locked` and `ran` arrive as fields and nothing here parses a detail string.
+const BOARD = {
+ type: 'object', additionalProperties: false,
+ required: ['board', 'ran', 'pass', 'locked', 'detail'],
+ properties: {
+ board: { type: 'string' }, ran: { type: 'boolean' }, pass: { type: 'boolean' },
+ locked: { type: 'boolean' }, detail: { type: 'string' },
+ },
+}
const HIL = {
type: 'object', additionalProperties: false,
- required: ['board', 'pass', 'detail', 'wedged'],
+ required: ['results', 'wedged'],
properties: {
- board: { type: 'string' }, pass: { type: 'boolean' },
- detail: { type: 'string' }, wedged: { type: 'boolean' },
+ results: { type: 'array', items: BOARD },
+ // the operator's own observation — not derivable from the report
+ wedged: { type: 'array', items: { type: 'string' } },
+ banner: { type: 'string' },
},
}
-const runBoard = (b) => agent(
- `Run the HIL test for board ${b} per .claude/skills/hil/SKILL.md. Do NOT touch the actions-runner service and do NOT pre-hold the board lock — hil_test.py self-locks the board while testing. ` +
+// ONE operator for the whole set, and one hil_test.py inside it. hil_test.py already runs the
+// boards concurrently: it round-robins them across host controllers and holds per-controller
+// permits (hil_lock.py FLASH_PARALLEL/USBTEST_PARALLEL) that bound simultaneous flashes and
+// usbtest batteries. Those permits live in one process, so a second hil_test.py does not share
+// them - N parallel single-board runs multiply the budget by N onto the same xHCI cards, for no
+// wall-clock gain over one run that already parallelizes them.
+const runBoards = (boards, isRetry = false) => agent(
+ `Run the HIL tests for these boards per .claude/skills/hil/SKILL.md: ${boards.join(', ')}. ` +
+ `Pass them ALL to ONE hil_test.py invocation as repeated -b flags (${boards.map((b) => `-b ${b}`).join(' ')}) — it schedules them across host controllers and budgets concurrent flashes and usbtest batteries itself. Never start a second hil_test.py alongside it. ` +
+ (isRetry
+ ? 'This is a RE-RUN of boards an earlier run could not take: pass --accumulate as well, or hil_test.py unlinks the report and the whole-fleet table collapses to just these boards. '
+ : '') +
+ 'Do NOT touch the actions-runner service and do NOT pre-hold the board locks — hil_test.py self-locks each board for its flash+test. ' +
(args.force
? 'THE USER HAS EXPLICITLY AUTHORIZED FORCING: run hil_test.py with HIL_NO_BOARD_LOCK=1 in the environment (bypasses the board lock check; do NOT release or kill the existing holder). '
- : 'If the run fails because the board lock is held (a dev session or concurrent CI job), report pass=false and set detail to start EXACTLY with "board locked:" followed by the holder JSON verbatim — never force the lock. ') +
- 'Reserve the phrase "board locked" strictly for lock contention; describe a frozen or non-enumerating board as "unresponsive" instead. ' +
- `Firmware is in examples/cmake-build-${b}. Use the config for this host (hostname first), single-board flag -b ${b}. Run hil_test.py as a BACKGROUND Bash task and wait for it (a stuck fleet runs to its pool guard, 60 min by default — beyond any foreground timeout); never cancel it early. ` +
- 'On non-lock failures retry once with -v -r 1 (one verbose attempt for diagnosis; note a usbtest battery that produced per-case verdicts is NOT auto-retried, so its result already stands). wedged=true if the board/fixture is unresponsive after the run (capture dmesg | tail -50 into detail).',
- { label: `hil:${b}`, phase: 'HIL', agentType: 'hil-operator', schema: HIL },
+ : 'A board whose lock is held (a dev session or concurrent CI job) fails fast inside the run without blocking the others — never force the lock. ') +
+ 'If hil_test.py refuses the run with "board(s) not in <config>", re-run it WITHOUT the unknown names but keep the FULL board list on the hil_summary call below — it emits a ran:false entry for every board you name, so the unknown ones surface as "no report row" instead of costing the whole batch. ' +
+ 'Use the config for this host (hostname first). Run hil_test.py as a BACKGROUND Bash task and wait for it (a stuck fleet runs to its pool guard, 60 min by default — beyond any foreground timeout); never cancel it early. ' +
+ 'On non-lock failures retry ONCE from the re-run spec hil_test.py just wrote — `<config>.failed`, which already begins with --accumulate — adding -v. A usbtest battery that produced per-case verdicts is NOT auto-retried, so its result already stands. ' +
+ 'THEN, from the directory the run wrote its report to, produce the results with:\n' +
+ ` python3 test/hil/helper/hil_summary.py <the config you used> ${boards.map((b) => `-b ${b}`).join(' ')}\n` +
+ 'Return its `results` array and `banner` EXACTLY as printed — do not retype, reword, re-order or "correct" them, and never transcribe the markdown table instead. ' +
+ 'Add `wedged`: the board names whose board or fixture your run left unresponsive (usually none). That is your own observation and the one field you author; put `dmesg | tail -50` in your reply text for any board you list.',
+ {
+ label: boards.length === 1 ? `hil:${boards[0]}` : `hil:${boards.length} boards`,
+ phase: 'HIL', agentType: 'hil-operator', schema: HIL,
+ },
)
-const results = []
-for (const b of args.boards) {
- const r = await runBoard(b)
- results.push(r || { board: b, pass: false, detail: 'hil-operator agent died', wedged: false })
- log(`${b}: ${results[results.length - 1].pass ? 'PASS' : 'FAIL'}`)
-}
+// A lookup, not a reconciliation: hil_summary.py emits exactly one entry per requested board,
+// so a missing entry means the operator dropped it rather than that the names disagree.
+const byBoard = (out) => new Map((out?.results || [])
+ .filter((r) => r && typeof r.board === 'string')
+ .map((r) => [r.board, r]))
+
+// `wedged` is the one field the operator authors, so it may echo a report row name
+// ('nano-fsdev') where the prompt asked for a board name. Accept the variant spelling
+// rather than dropping a wedge over it -- only `wedged` sends anyone to usb-kernel-recover.
+const wedgedFor = (list, b) => (Array.isArray(list) ? list : [])
+ .some((w) => typeof w === 'string' && (w === b || w.startsWith(`${b}-`)))
+
+const first = await runBoards(args.boards)
+const firstRows = byBoard(first)
+const firstWedged = first?.wedged || []
+const results = args.boards.map((b) => {
+ const r = firstRows.get(b)
+ if (!r) {
+ return {
+ board: b, pass: false, locked: false, ran: false, wedged: wedgedFor(firstWedged, b),
+ detail: first ? 'hil-operator returned no entry for this board' : 'hil-operator agent died',
+ }
+ }
+ return { ...r, wedged: wedgedFor(firstWedged, b) }
+})
+for (const r of results) log(`${r.board}: ${r.pass ? 'PASS' : r.locked ? 'LOCKED' : 'FAIL'}`)
+if (first?.banner) log(`report banner: ${first.banner.trim().split('\n')[0]}`)
// A concurrent CI job may have held some boards (its hil_test.py flock).
// CI finishes a board in minutes — retry locked boards once, at the end.
if (!args.force) {
- for (let i = 0; i < results.length; i++) {
- if (results[i].pass || !results[i].detail.startsWith('board locked')) continue
- log(`${results[i].board}: was locked — retrying once`)
- const r = await runBoard(results[i].board)
- if (r) results[i] = r
- else results[i].detail += ' (retry operator died)'
- log(`${results[i].board}: retry ${results[i].pass ? 'PASS' : 'FAIL'}`)
+ const relock = results.filter((r) => r.locked && !r.pass).map((r) => r.board)
+ if (relock.length) {
+ log(`was locked, retrying once: ${relock.join(', ')}`)
+ const again = await runBoards(relock, true)
+ const rows = byBoard(again)
+ const againWedged = again?.wedged || []
+ for (let i = 0; i < results.length; i++) {
+ const b = results[i].board
+ if (!relock.includes(b)) continue
+ const r = rows.get(b)
+ // No entry keeps the board `locked`, so it still reaches the user's force/wait/accept
+ // decision instead of being published as a hardware failure.
+ if (r) {
+ // a passing retry does NOT clear a wedge the first run left behind: only `wedged`
+ // sends anyone to usb-kernel-recover
+ results[i] = { ...r, wedged: results[i].wedged || wedgedFor(againWedged, b) }
+ } else {
+ results[i].detail += again ? ' (retry returned no entry)' : ' (retry operator died)'
+ }
+ log(`${b}: retry ${results[i].pass ? 'PASS' : 'FAIL'}`)
+ }
}
}
-const wedged = results.filter(r => r.wedged).map(r => r.board)
+// pass/wedged/locked in one place so it can be exercised without running an agent
+const summarize = (rs, force) => ({
+ pass: rs.every((r) => r.pass),
+ wedged: rs.filter((r) => r.wedged).map((r) => r.board),
+ locked: force ? [] : rs.filter((r) => !r.pass && r.locked).map((r) => r.board),
+})
+
+const { pass, wedged, locked } = summarize(results, args.force)
if (wedged.length) log(`WEDGED boards needing usb-kernel-recover: ${wedged.join(', ')}`)
// Workers cannot prompt the user — surface still-locked boards for the main
// session to ask: force (re-invoke with force: true), wait, or accept.
-const locked = args.force ? [] : results.filter(r => !r.pass && r.detail.startsWith('board locked')).map(r => r.board)
if (locked.length) log(`still locked after retry: ${locked.join(', ')} — ask the user: force / keep waiting / accept`)
-return { pass: results.every(r => r.pass), results, wedged, locked }
+return { pass, results, wedged, locked }
diff --git a/.claude/workflows/test-hil-validate.mjs b/.claude/workflows/test-hil-validate.mjs
new file mode 100644
index 000000000..80f37b27d
--- /dev/null
+++ b/.claude/workflows/test-hil-validate.mjs
@@ -0,0 +1,59 @@
+// Executable checks for hil-validate.js's result handling.
+//
+// The join that used to live here -- matching variant row names to boards, parsing
+// `board locked` out of a prose detail, folding rows, keeping a wedged flag alive -- produced
+// a defect in each of four review rounds, including a test that asserted an invariant using
+// the one input shape that could not break it. That logic now lives in
+// test/hil/helper/hil_summary.py, where the roster is, and arrives here as fields. What is
+// left is a lookup and a verdict, and this pins both.
+//
+// Run: node .claude/workflows/test-hil-validate.mjs
+import { readFileSync } from 'node:fs'
+
+const src = readFileSync(new URL('./hil-validate.js', import.meta.url), 'utf8')
+const body = src.slice(src.indexOf('const byBoard ='), src.indexOf('const first = await runBoards'))
+ + src.slice(src.indexOf('const summarize ='), src.indexOf('const { pass, wedged, locked } ='))
+const { byBoard, summarize, wedgedFor } = new Function(`${body}; return { byBoard, summarize, wedgedFor }`)()
+
+let failed = 0
+const check = (name, got, want) => {
+ const g = JSON.stringify(got), w = JSON.stringify(want)
+ if (g === w) return console.log(` ok ${name}`)
+ failed++
+ console.log(` FAIL ${name}\n got ${g}\n want ${w}`)
+}
+const R = (board, pass, locked = false, wedged = false, detail = '') =>
+ ({ board, ran: true, pass, locked, detail, wedged })
+
+console.log('byBoard — indexing the operator payload')
+check('indexes by board', [...byBoard({ results: [R('a', true)] }).keys()], ['a'])
+check('null payload', [...byBoard(null).keys()], [])
+check('missing results', [...byBoard({}).keys()], [])
+check('rows not an array', [...byBoard({ results: null }).keys()], [])
+for (const bad of [[null], [undefined], [{ pass: true }], [{ board: 42 }]]) {
+ try { check(`malformed row ${JSON.stringify(bad)}`, [...byBoard({ results: bad }).keys()], []) }
+ catch (e) { failed++; console.log(` FAIL malformed row threw ${e}`) }
+}
+
+console.log('wedgedFor — operator-authored names, variant spellings tolerated')
+check('board name matches', wedgedFor(['nano'], 'nano'), true)
+check('variant spelling matches', wedgedFor(['nano-fsdev'], 'nano'), true)
+check('another board does not', wedgedFor(['other'], 'nano'), false)
+check('prefix without dash does not', wedgedFor(['nanoch32'], 'nano'), false)
+check('null list', wedgedFor(null, 'nano'), false)
+check('non-string entry does not throw', wedgedFor([42, 'nano'], 'nano'), true)
+
+console.log('summarize — the ship/no-ship verdict')
+check('all pass', summarize([R('a', true)], false).pass, true)
+check('one fail sinks it', summarize([R('a', true), R('b', false)], false).pass, false)
+check('a locked board is not a pass', summarize([R('a', false, true)], false).pass, false)
+check('locked is a field, not a prefix', summarize([R('a', false, true)], false).locked, ['a'])
+check('a real failure is not locked', summarize([R('a', false, false)], false).locked, [])
+check('a PASSING board is never locked', summarize([R('a', true, true)], false).locked, [])
+check('force zeroes locked', summarize([R('a', false, true)], true).locked, [])
+check('wedged surfaces', summarize([R('a', false, false, true)], false).wedged, ['a'])
+check('a wedged board that passed still surfaces',
+ summarize([R('a', true, false, true)], false).wedged, ['a'])
+
+console.log(failed ? `\n${failed} FAILED` : '\nall checks passed')
+process.exit(failed ? 1 : 0)
diff --git a/.gitignore b/.gitignore
index 8773322e4..f6c8702b6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,6 +2,7 @@ html
latex
hil_report.md
hil_report.json
+*.json.failed
*.a
*.d
*.o
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 3d9c8482b..17170b7d6 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -62,6 +62,15 @@ repos:
entry: python3 -m unittest discover -s test/hil/test
pass_filenames: false
language: system
+ # hil-validate.js decides which boards ship. Its result join has been wrong three times --
+ # dropping variant-named rows, letting a PASS erase a FAIL, breaking the `board locked`
+ # anchor it had just fixed -- each time because the logic was reasoned about instead of run.
+ - id: hil-validate-logic
+ name: hil-validate-logic
+ files: ^\.claude/workflows/
+ entry: node .claude/workflows/test-hil-validate.mjs
+ pass_filenames: false
+ language: system
- id: hil-select-test
name: hil-select-test
files: ^(hw/bsp/|src/|examples/)
diff --git a/test/hil/helper/hil_summary.py b/test/hil/helper/hil_summary.py
new file mode 100644
index 000000000..e566bead0
--- /dev/null
+++ b/test/hil/helper/hil_summary.py
@@ -0,0 +1,115 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: MIT
+"""Fold hil_report.json into one machine-readable verdict per BOARD.
+
+A workflow driving hil_test.py through an operator agent has no filesystem access, so the
+agent has to carry the results across. It must carry them, not retype them: the previous
+design asked the agent to transcribe the markdown table, and every defect found in four
+review rounds came from re-parsing that prose -- variant row names vs board names,
+`board locked` vs `board-locked`, folding several variant rows into one verdict, rows that
+matched no board. All of it is a join, and the join belongs here, where the roster is.
+
+Report rows are named per VARIANT (hil_test.py builds them from `vname`), and a variant name
+is not required to start with the board name -- nanoch32v203 produces only `-fsdev`/`-usbfs`,
+ch32v307v_r1_1v0 only `-usbhs`/`-usbfs`. The config is what maps them back.
+
+Emits, on stdout:
+ {"results": [{"board", "ran", "pass", "locked", "detail"}...], "banner": str}
+
+`locked` is a field, not a prefix to grep for. `ran` false means the board produced no row at
+all, which is not the same as failing.
+
+Usage: hil_summary.py <config.json> [-b BOARD]... [--report-dir DIR]
+"""
+import argparse
+import json
+import sys
+from pathlib import Path
+
+FAIL_ICON, SKIP_ICON = '❌', '⚪' # a pass needs no icon: unmarked = pass
+LOCKED_CELL = 'board-locked'
+
+
+def cell_state(v: str) -> str:
+ """'pass' | 'fail' | 'skip' -- the EXACT classifier hil_test.py's own tally uses
+ (cell_kind in render_matrix): 'fail' or a ❌ prefix is a failure, 'skip' or a ⚪
+ prefix is a skip, and EVERYTHING ELSE is a pass. That last arm is load-bearing: a
+ passing test may return a plain metric string ('480.0 MBps') that lands in the cell
+ unprefixed, while failures are guaranteed marked -- TestFail's docstring pins that its
+ metric is icon-prefixed precisely so render/tally treat it as a failure. Classifying
+ unknown shapes as fail here would publish a green table as a red verdict."""
+ if v == 'fail' or v.startswith(FAIL_ICON):
+ return 'fail'
+ if v == 'skip' or v.startswith(SKIP_ICON):
+ return 'skip'
+ return 'pass'
+
+
+def variants_of(cfg: dict, board: str) -> list:
+ for b in cfg.get('boards', []):
+ if b['name'] == board:
+ return [v['name'] for v in (b.get('variant') or [])] or [board]
+ return [board]
+
+
+def summarize(cfg: dict, boards: list, report: dict) -> dict:
+ rows = {r['board']: r.get('cells') or {} for r in report.get('rows', [])}
+ owner = {v['name']: b['name'] for b in cfg.get('boards', [])
+ for v in (b.get('variant') or [])}
+ results = []
+ for board in boards:
+ names = variants_of(cfg, board)
+ mine = {n: rows[n] for n in names if n in rows}
+ # a variant name that is neither declared nor prefixed cannot be attributed; the
+ # `<board>-` fallback only helps ad-hoc builds, it is not the primary path. It must
+ # also never steal a row DECLARED by another board: a declared variant need not start
+ # with its own board's name, so it may happen to start with this board's name plus '-'.
+ mine.update({n: c for n, c in rows.items()
+ if n.startswith(f'{board}-') and n not in mine
+ and owner.get(n, board) == board})
+ if not mine:
+ results.append({'board': board, 'ran': False, 'pass': False, 'locked': False,
+ 'detail': 'no report row for this board'})
+ continue
+ locked = any(LOCKED_CELL in cells for cells in mine.values())
+ bad = []
+ for vname, cells in sorted(mine.items()):
+ for test, val in sorted(cells.items()):
+ if test == LOCKED_CELL:
+ continue
+ if cell_state(str(val)) == 'fail':
+ bad.append(f'{vname} {test}: {val}')
+ ok = not bad and not locked
+ if locked:
+ detail = 'held by another holder; not flashed'
+ elif bad:
+ detail = '; '.join(bad)
+ else:
+ detail = f'{len(mine)} variant(s), {sum(len(c) for c in mine.values())} cell(s) ok'
+ results.append({'board': board, 'ran': True, 'pass': ok, 'locked': locked,
+ 'detail': detail})
+ return {'results': results, 'banner': report.get('banner', '')}
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser()
+ ap.add_argument('config_file')
+ ap.add_argument('-b', '--board', action='append', default=[],
+ help='boards to report on; default: every board in the config')
+ ap.add_argument('--report-dir', default='.', help='where hil_report.json lives (default: cwd)')
+ a = ap.parse_args()
+
+ cfg = json.loads(Path(a.config_file).read_text())
+ boards = a.board or [b['name'] for b in cfg.get('boards', [])]
+ jpath = Path(a.report_dir) / 'hil_report.json'
+ if not jpath.is_file():
+ print(f'error: {jpath} not found -- did hil_test.py run in this directory?',
+ file=sys.stderr)
+ return 1
+ json.dump(summarize(cfg, boards, json.loads(jpath.read_text())), sys.stdout, indent=2)
+ print()
+ return 0
+
+
+if __name__ == '__main__':
+ sys.exit(main())
diff --git a/test/hil/hil_ci.sh b/test/hil/hil_ci.sh
index c7dfa95df..f474c3055 100644
--- a/test/hil/hil_ci.sh
+++ b/test/hil/hil_ci.sh
@@ -1,8 +1,9 @@
#!/usr/bin/env bash
# Run HIL test remotely on ci.lan
-# Usage: test/hil/hil_ci.sh [-b BOARD] [-t TEST] [extra hil_test.py args...]
+# Usage: test/hil/hil_ci.sh [-b BOARD]... [-t TEST] [extra hil_test.py args...]
# Example:
# test/hil/hil_ci.sh -b stm32f723disco
+# test/hil/hil_ci.sh -b stm32f723disco -b raspberry_pi_pico
# test/hil/hil_ci.sh -b stm32f723disco -t host/cdc_msc_hid -r 1
#
# Env overrides: REMOTE, REMOTE_DIR, CONFIG (path to HIL config json),
@@ -44,17 +45,46 @@ for a in "$@"; do
exit 1
done
-# Parse -b BOARD from arguments to know which build to copy
-BOARD=""
+# Parse -b BOARD from arguments to know which builds to copy. Repeatable: hil_test.py
+# takes the whole board set in ONE run (it schedules them across host controllers and
+# budgets the flashes itself), so every -b needs its binaries staged, not just the last.
+BOARDS=()
ARGS=()
while [[ $# -gt 0 ]]; do
case "$1" in
- -b)
- [[ $# -ge 2 ]] || { echo "error: -b requires a BOARD argument" >&2; exit 1; }
- BOARD="$2"
+ # hil_test.py declares `-b, --board` with action='append', so argparse also accepts
+ # --board=X and -bX. Recognising only the bare `-b X` forwarded the others to the rig
+ # while never staging them: the board ran with no firmware and reported a green row.
+ -b|--board)
+ [[ $# -ge 2 ]] || { echo "error: $1 requires a BOARD argument" >&2; exit 1; }
+ BOARDS+=("$2")
ARGS+=("$1" "$2")
shift 2
;;
+ --board=*)
+ BOARDS+=("${1#--board=}")
+ ARGS+=("$1")
+ shift
+ ;;
+ # -bt (--board-test) BEFORE the glued -b?* arm, mirroring argparse's longest-match: it is
+ # the form <config>.failed uses, and a bare -b?* would register a board named "t..." that
+ # the roster check below rejects -- killing every documented retry.
+ -bt|--board-test)
+ [[ $# -ge 2 ]] || { echo "error: $1 requires NAME:tests" >&2; exit 1; }
+ ARGS+=("$1" "$2")
+ shift 2
+ ;;
+ -bt?*|--board-test=*)
+ ARGS+=("$1")
+ shift
+ ;;
+ # glued short form: argparse resolves -bNAME to --board NAME, so staging must too --
+ # unparsed it fell through to the all-boards branch and silently staged everything built
+ -b?*)
+ BOARDS+=("${1#-b}")
+ ARGS+=("$1")
+ shift
+ ;;
*)
ARGS+=("$1")
shift
@@ -62,6 +92,107 @@ while [[ $# -gt 0 ]]; do
esac
done
+# Resolve a board to its build dirs: its own dir, the cmake-build-<board>-* glob (ad-hoc
+# local builds), and the variant dirs named in $CONFIG -- variant names are NOT required to
+# be prefixed with the board name, so the glob alone is not enough. Prints one dir per line.
+variant_names() {
+ python3 -c '
+import json, sys
+cfg = json.load(open(sys.argv[1]))
+for b in cfg.get("boards", []):
+ if b["name"] == sys.argv[2]:
+ for v in b.get("variant") or []:
+ print(v["name"])
+' "$CONFIG" "$1"
+}
+
+resolve_build_dirs() {
+ local board="$1" d v
+ declare -A seen=()
+ shopt -s nullglob
+ for d in "$ROOT_DIR"/examples/cmake-build-"$board" "$ROOT_DIR"/examples/cmake-build-"$board"-*; do
+ [[ -d $d && -z ${seen[$d]:-} ]] && { seen[$d]=1; printf '%s\n' "$d"; }
+ done
+ shopt -u nullglob
+ # to a file, not a process substitution: `set -e`/pipefail cannot see the exit status of
+ # the latter, so a malformed roster silently yielded zero variant dirs
+ local vf; vf=$(mktemp)
+ variant_names "$board" > "$vf" || { rm -f "$vf"; echo "Error: could not read variants for $board from $CONFIG" >&2; exit 1; }
+ while IFS= read -r v; do
+ d="$ROOT_DIR/examples/cmake-build-$v"
+ [[ -d $d && -z ${seen[$d]:-} ]] && { seen[$d]=1; printf '%s\n' "$d"; }
+ done < "$vf"
+ rm -f "$vf"
+}
+
+# Pre-flight: EVERY board must resolve to at least one build dir before anything is wiped or
+# copied. This check used to live in the copy loop, so an unbuilt board late in the list
+# aborted the run after the remote tree had been rm -rf'd and earlier boards fully rsynced --
+# zero coverage, a half-staged rig, and a stale local hil_report.md left in place. Report all
+# missing boards at once so one build round fixes them.
+MANIFEST=$(mktemp)
+trap 'rm -f "$MANIFEST"' EXIT
+# Roster membership first: hil_test.py rejects an unknown -b with sys.exit(1) for the WHOLE
+# run (hil_test.py:2297), and it does so AFTER this script has wiped REMOTE_DIR and staged
+# every board -- one typo then costs the entire batch. We already parse $CONFIG here, so
+# catch it before anything is touched. Note -b matches board names only, never variant names.
+if [ ${#BOARDS[@]} -gt 0 ]; then
+ROSTER=$(python3 -c '
+import json, sys
+print("\n".join(b["name"] for b in json.load(open(sys.argv[1])).get("boards", [])))
+' "$CONFIG") || { echo "error: could not read the board roster from $CONFIG" >&2; exit 1; }
+notinroster=()
+for b in ${BOARDS[@]+"${BOARDS[@]}"}; do
+ grep -qxF -- "$b" <<< "$ROSTER" || notinroster+=("$b")
+done
+if [ ${#notinroster[@]} -gt 0 ]; then
+ echo "error: not in $(basename "$CONFIG"): ${notinroster[*]}" >&2
+ echo " (-b takes board names, not variant names)" >&2
+ exit 1
+fi
+fi # BOARDS non-empty: nothing to validate for an all-boards run
+
+missing=()
+for b in ${BOARDS[@]+"${BOARDS[@]}"}; do
+ dirs=$(resolve_build_dirs "$b")
+ if [ -z "$dirs" ]; then
+ missing+=("$b")
+ else
+ while IFS= read -r d; do printf '%s\t%s\n' "$b" "$d" >> "$MANIFEST"; done <<< "$dirs"
+ # A declared variant with no build dir is NOT an error -- no cmake preset is
+ # variant-suffixed, so this is the normal state for e.g. the -DMA variants. It is worth
+ # saying out loud: hil_test.py logs `Skip (no binary)` and counts zero errors for it, so
+ # the run exits 0 and the operator reads a green table for cells that never ran.
+ while IFS= read -r v; do
+ [ -z "$v" ] && continue
+ # whole lines: a substring match lets cmake-build-<v>-DMA silence the warning for <v>
+ grep -qxF -- "$ROOT_DIR/examples/cmake-build-$v" <<< "$dirs" \
+ || echo "warning: $b variant '$v' has no build dir -- its cells will be skipped, not tested" >&2
+ done < <(variant_names "$b")
+ fi
+done
+if [ ${#missing[@]} -gt 0 ]; then
+ echo "Error: no build directory under $ROOT_DIR/examples/ for: ${missing[*]}" >&2
+ for b in "${missing[@]}"; do
+ echo " cd examples && cmake --preset $b && cmake --build --preset $b" >&2
+ done
+ exit 1
+fi
+
+# The all-boards form needs its emptiness check HERE too: below the setup ssh it fired after
+# the remote tree was already rm -rf'd, destroying the previous run's report and re-run spec
+# on the rig before deciding there was nothing to do.
+if [ ${#BOARDS[@]} -eq 0 ]; then
+ shopt -s nullglob
+ allbuilds=("$ROOT_DIR"/examples/cmake-build-*/)
+ shopt -u nullglob
+ if [ ${#allbuilds[@]} -eq 0 ]; then
+ echo "error: no examples/cmake-build-* directories under $ROOT_DIR -- nothing to test" >&2
+ echo " build first, e.g.: cd examples && cmake --preset <board> && cmake --build --preset <board>" >&2
+ exit 1
+ fi
+fi
+
# Setup remote directory. `bash -s` + heredoc so REMOTE_DIR arrives as a positional
# parameter, keeping the `rm -rf` target out of the command string the heredoc runs.
echo "==> Setting up remote $REMOTE:$REMOTE_DIR"
@@ -89,6 +220,7 @@ scp -q "$ROOT_DIR/test/hil/helper/__init__.py" \
"$ROOT_DIR/test/hil/helper/hil_util.py" \
"$ROOT_DIR/test/hil/helper/hil_health.py" \
"$ROOT_DIR/test/hil/helper/hil_lock.py" \
+ "$ROOT_DIR/test/hil/helper/hil_summary.py" \
"$ROOT_DIR/test/hil/helper/hil_select.py" \
"$REMOTE:$REMOTE_DIR/test/hil/helper/"
@@ -103,52 +235,23 @@ copy_board_binaries() {
"$src" "$REMOTE:$REMOTE_DIR/examples/"
}
-if [ -n "$BOARD" ]; then
- # Copy the board's build dir plus its variant dirs. Variant names come from
- # $CONFIG (they are not required to be prefixed with the board name); the
- # cmake-build-<BOARD>-* glob is kept as a fallback for ad-hoc local builds.
- # Collect only dirs that actually exist, deduplicated.
- declare -A SEEN_DIRS=()
- BUILD_DIRS=()
- add_build_dir() {
- [[ -d "$1" && -z "${SEEN_DIRS[$1]:-}" ]] || return 0
- SEEN_DIRS[$1]=1
- BUILD_DIRS+=("$1")
- }
- shopt -s nullglob
- for d in "$ROOT_DIR"/examples/cmake-build-"$BOARD" "$ROOT_DIR"/examples/cmake-build-"$BOARD"-*; do
- add_build_dir "$d"
- done
- shopt -u nullglob
- # to a file, not a process substitution: `set -e`/pipefail cannot see the exit
- # status of the latter, so a malformed roster silently yielded zero variant dirs
- VARIANTS_FILE=$(mktemp)
- python3 -c '
-import json, sys
-cfg = json.load(open(sys.argv[1]))
-for b in cfg.get("boards", []):
- if b["name"] == sys.argv[2]:
- for v in b.get("variant") or []:
- print(v["name"])
-' "$CONFIG" "$BOARD" > "$VARIANTS_FILE" || {
- echo "Error: could not read variants for $BOARD from $CONFIG"
- rm -f "$VARIANTS_FILE"
- exit 1
- }
- while IFS= read -r v; do
- add_build_dir "$ROOT_DIR/examples/cmake-build-$v"
- done < "$VARIANTS_FILE"
- rm -f "$VARIANTS_FILE"
- if [ ${#BUILD_DIRS[@]} -eq 0 ]; then
- echo "Error: no build directory found for $BOARD under $ROOT_DIR/examples/"
- echo "Build first with: cd examples && cmake --preset $BOARD && cmake --build --preset $BOARD"
- exit 1
- fi
- echo "==> Copying binaries for $BOARD (${#BUILD_DIRS[@]} build dir(s))"
- for d in "${BUILD_DIRS[@]}"; do
- copy_board_binaries "$d"
+if [ ${#BOARDS[@]} -gt 0 ]; then
+ # Replay the pre-flight manifest: the dirs were already resolved and proved non-empty
+ # for every board, so nothing here can abort mid-staging. Plain reads of the manifest --
+ # a process substitution would hide a reader failure from set -e (the comment in
+ # resolve_build_dirs is about exactly that trap).
+ for b in "${BOARDS[@]}"; do
+ dirs=()
+ while IFS=$'\t' read -r bb d; do
+ [ "$bb" = "$b" ] && [ -n "$d" ] && dirs+=("$d")
+ done < "$MANIFEST"
+ echo "==> Copying binaries for $b (${#dirs[@]} build dir(s))"
+ for d in ${dirs[@]+"${dirs[@]}"}; do
+ copy_board_binaries "$d"
+ done
done
else
+ # emptiness was already refused in pre-flight, before the remote wipe
echo "==> Copying all built binaries"
# Use `%/` parameter expansion to strip the trailing slash from the glob —
# rsync needs the bare dir name so the per-board cmake-build-<BOARD>/ subdir
@@ -170,14 +273,38 @@ for a in ${ARGS[@]+"${ARGS[@]}"}; do ARGS_Q+=("$(printf '%q' "$a")"); done
CONFIG_Q="$(printf '%q' "test/hil/$(basename "$CONFIG")")"
echo "==> Running HIL test on $REMOTE"
rc=0
-# --retry 1 FIRST, before the user's args: this targets the same shared rig CI uses, and
-# the pool guard is a flat constant that does not scale with max_retry -- argparse's
-# default of 3 lets a few flaky boards re-pay 510s each until the 3600s guard fires,
-# abandoning the pool and holding board flocks against concurrent CI. Placed first, not
-# appended, so argparse's last-wins means `hil_ci.sh -r 3` still gets 3.
-ssh "$REMOTE" bash -s -- "$REMOTE_DIR" --retry 1 ${ARGS_Q[@]+"${ARGS_Q[@]}"} "$CONFIG_Q" <<'REMOTE' || rc=$?
+# --retry 1 FIRST, before the user's args: this targets the same shared rig CI uses, and the
+# pool guard is a flat constant that does not scale with max_retry, so a few flaky boards can
+# re-pay ~510s each until the 3600s guard fires, abandoning the pool and holding board flocks
+# against concurrent CI. hil_test.py's own default is already 1; passing it explicitly keeps
+# that true if the default ever moves. Placed first, not appended, so argparse's last-wins
+# means `hil_ci.sh -r 3` still gets 3.
+# Forward the HIL_* knobs (HIL_NO_BOARD_LOCK for an authorized force, the parallel widths,
+# HIL_POOL_TIMEOUT). ssh passes no environment and joins its argv into one string the remote
+# shell re-splits, so a bare NAME=value element would arrive as a positional argument to
+# hil_test.py and argparse would exit 2. Build `export` lines instead and hand them over as a
+# single %q-quoted word for the remote to eval.
+# Joined with '; ', NOT newlines: %q renders a newline as bash-only $'...' quoting, which the
+# remote LOGIN shell must parse from the joined command string -- under dash the force arrives
+# as garbage and silently does nothing. Backslash escaping round-trips in both shells.
+# HIL_REPORT_DIR stays local: where the report lands on the rig is this script's contract
+# (REMOTE_DIR, where all three copy-backs below look), so forwarding it would relocate the
+# report and every copy-back would come home empty.
+HIL_EXPORTS=""
+while IFS= read -r v; do
+ [ -z "$v" ] && continue
+ HIL_EXPORTS+="export $(printf '%s=%q' "$v" "${!v}"); "
+done < <(compgen -v | grep -x 'HIL_[A-Z0-9_]*' | grep -vxE 'HIL_EXPORTS|HIL_REPORT_DIR' || true)
+[ -n "$HIL_EXPORTS" ] && echo "==> Forwarding: $HIL_EXPORTS"
+# One %q-quoted word, so ssh's argv join and the remote shell's re-split hand it back
+# byte-for-byte, and the remote evals it. Empty stays `''` -- a real, shiftable argument --
+# rather than vanishing from the joined string and shifting the run's own flags out of place.
+HIL_EXPORTS_Q=$(printf '%q' "$HIL_EXPORTS")
+
+ssh "$REMOTE" bash -s -- "$REMOTE_DIR" "$HIL_EXPORTS_Q" --retry 1 ${ARGS_Q[@]+"${ARGS_Q[@]}"} "$CONFIG_Q" <<'REMOTE' || rc=$?
cd -- "$1"
shift
+eval "$1"; shift # HIL_* exports, %q-quoted locally into one word
# Flasher CLIs live in the user bin dirs on ci.lan (esptool/idf in ~/.local/bin,
# STM32CubeProgrammer's STM32_Programmer_CLI in ~/bin); the non-interactive shell
# subprocess used for flashing doesn't source profile/rc, so add them explicitly.
@@ -191,4 +318,15 @@ scp -q "$REMOTE:$REMOTE_DIR/hil_report.md" "$ROOT_DIR/hil_report.md" \
&& echo "==> Report copied to $ROOT_DIR/hil_report.md" \
|| echo "==> warning: no hil_report.md copied back" >&2
+# The re-run spec and the JSON sidecar live in the run's cwd on the rig (REMOTE_DIR), and the
+# next invocation rm -rf's it. Without copying them back, the `--accumulate` retry every doc on
+# this branch prescribes has nothing to read and nothing to merge onto. Delete the local copies
+# FIRST: a green run writes no .failed, so a silent no-op scp would leave last run's spec in
+# the checkout looking current, and "retry from the spec" would re-flash boards that passed.
+for extra in "$(basename "$CONFIG").failed" hil_report.json; do
+ rm -f "$ROOT_DIR/$extra"
+ scp -q "$REMOTE:$REMOTE_DIR/$extra" "$ROOT_DIR/$extra" 2>/dev/null \
+ && echo "==> $extra copied to $ROOT_DIR/$extra" || true
+done
+
exit $rc
diff --git a/test/hil/test/test_hil_bounded.py b/test/hil/test/test_hil_bounded.py
index 908a142d5..c6d454f0e 100644
--- a/test/hil/test/test_hil_bounded.py
+++ b/test/hil/test/test_hil_bounded.py
@@ -1364,7 +1364,14 @@ class RemoteDirIsScreened(unittest.TestCase):
rc = 0 if keep_going else 77
for tool in ('ssh', 'scp', 'rsync'):
write_script(Path(td) / tool, f'echo "stub-{tool} $*" >&2; exit {rc}')
+ # hil_ci.sh now refuses an all-boards run with nothing built, so this arg-quoting
+ # test needs a checkout stub with one build dir to reach the run invocation
+ root = Path(td) / 'root'
+ (root / 'test' / 'hil').mkdir(parents=True)
+ (root / 'test' / 'hil' / 'hil_test.py').touch()
+ (root / 'examples' / 'cmake-build-alpha').mkdir(parents=True)
env = {**os.environ, 'REMOTE_DIR': remote_dir, 'REMOTE': 'stub',
+ 'ROOT_DIR': str(root),
'PATH': td + os.pathsep + os.environ['PATH']}
return subprocess.run(
['bash', str(Path(TEST_DIR).parents[0] / 'hil_ci.sh'), *args],
@@ -1414,6 +1421,316 @@ class RemoteDirIsScreened(unittest.TestCase):
self.assertIn(r'host/cdc\ msc', run_line)
+class EveryBoardIsStaged(unittest.TestCase):
+ """One hil_test.py run takes several `-b` flags, and hil-operator hands it the whole board
+ set that way. The `-b` parse loop kept a single BOARD, so only the LAST board's binaries
+ were rsynced and every other board died on the rig with a missing firmware path -- after
+ its flash slot and lock were already spent.
+
+ Three of these five fail against the pre-fix script (the discriminating unbuilt case
+ puts the board FIRST, because the old single-BOARD parse happened to handle a trailing
+ one correctly); the run-line and variant-dir tests are characterization -- the old script
+ already forwarded ARGS whole and read variants from the config for its one board."""
+
+ def _run(self, boards, cfg_boards=None, variants=None):
+ import json
+ import subprocess
+ td = TemporaryDirectory()
+ self.addCleanup(td.cleanup)
+ root = Path(td.name) / 'root'
+ (root / 'test' / 'hil').mkdir(parents=True)
+ (root / 'test' / 'hil' / 'hil_test.py').touch()
+ built = cfg_boards if cfg_boards is not None else boards
+ (root / 'examples').mkdir(parents=True, exist_ok=True)
+ for b in built:
+ (root / 'examples' / f'cmake-build-{b}').mkdir(parents=True)
+ roster = [{'name': b} for b in boards]
+ for entry in roster:
+ for v in (variants or {}).get(entry['name'], []):
+ entry.setdefault('variant', []).append({'name': v})
+ cfg = root / 'test' / 'hil' / 'cfg.json'
+ cfg.write_text(json.dumps({'boards': roster}))
+ stubs = Path(td.name) / 'bin'
+ stubs.mkdir()
+ # real ssh/scp/rsync would reach the rig; these just record the argv
+ for tool in ('ssh', 'scp', 'rsync'):
+ write_script(stubs / tool, f'echo "stub-{tool} $*" >&2; exit 0')
+ env = {**os.environ, 'REMOTE': 'stub', 'ROOT_DIR': str(root), 'CONFIG': str(cfg),
+ 'PATH': str(stubs) + os.pathsep + os.environ['PATH']}
+ args = [a for b in boards for a in ('-b', b)]
+ r = subprocess.run(['bash', str(Path(TEST_DIR).parents[0] / 'hil_ci.sh'), *args],
+ capture_output=True, text=True, timeout=60, env=env)
+ r.rsyncs = [l for l in r.stderr.splitlines() if l.startswith('stub-rsync')]
+ # the RUN ssh is the one carrying hil_test.py's args; the setup ssh is not
+ r.run_lines = [l for l in r.stderr.splitlines() if '--retry 1' in l]
+ return r
+
+ def test_binaries_for_every_requested_board_are_copied(self):
+ r = self._run(['alpha', 'beta', 'gamma'])
+ self.assertEqual(r.returncode, 0, r.stderr)
+ for b in ('alpha', 'beta', 'gamma'):
+ self.assertTrue(any(f'cmake-build-{b} ' in l for l in r.rsyncs),
+ f'{b} binaries never staged: {r.rsyncs}')
+
+ def test_every_board_reaches_hil_test(self):
+ r = self._run(['alpha', 'beta'])
+ self.assertEqual(len(r.run_lines), 1, r.stderr)
+ self.assertIn('-b alpha', r.run_lines[0])
+ self.assertIn('-b beta', r.run_lines[0])
+
+ def test_an_unbuilt_board_aborts_before_anything_is_staged(self):
+ """The discriminating case: the unbuilt board is FIRST. The pre-fix script kept only
+ the last -b, found it built, and ran happily while silently testing one board. It also
+ has to fail BEFORE staging -- the old in-loop check fired after the remote tree was
+ wiped and earlier boards were rsynced, costing a run and leaving a half-staged rig."""
+ r = self._run(['alpha', 'beta'], cfg_boards=['beta'])
+ self.assertNotEqual(r.returncode, 0, 'unbuilt first board was accepted')
+ self.assertIn('alpha', r.stdout + r.stderr)
+ self.assertEqual(r.rsyncs, [], f'staged despite an unbuilt board: {r.rsyncs}')
+ self.assertEqual(r.run_lines, [], 'reached the run despite an unbuilt board')
+
+ def test_a_board_whose_firmware_is_only_a_variant_dir_is_accepted(self):
+ """Variant names are not required to be prefixed with the board name, so a board can
+ own no `cmake-build-<board>` dir at all. A pre-flight that only globs the board name
+ rejects it and tells the user to build firmware that is already there."""
+ r = self._run(['alpha', 'beta'], cfg_boards=['alpha', 'odd-name-v'],
+ variants={'beta': ['odd-name-v']})
+ self.assertEqual(r.returncode, 0, r.stderr)
+ self.assertTrue(any('cmake-build-odd-name-v ' in l for l in r.rsyncs),
+ f"beta's variant dir never staged: {r.rsyncs}")
+
+ def test_all_unbuilt_boards_are_named_at_once(self):
+ """One build round should fix every complaint, so the guard reports the whole set."""
+ r = self._run(['alpha', 'beta', 'gamma'], cfg_boards=['beta'])
+ self.assertNotEqual(r.returncode, 0)
+ out = r.stdout + r.stderr
+ self.assertIn('alpha', out)
+ self.assertIn('gamma', out)
+
+
+class StagingCoversEveryBoardForm(unittest.TestCase):
+ """hil_test.py declares `-b, --board` with action='append', so argparse accepts --board X,
+ --board=X and -bX too. Staging only the bare form sent boards to the rig with no firmware,
+ where every test logs `Skip (no binary)` and counts zero errors -- a green row for a board
+ that was never flashed. Also covers the roster check, which has to fire BEFORE the remote
+ tree is wiped, since hil_test.py rejects an unknown -b for the whole run."""
+
+ def _run(self, argv, built, roster=None, variants=None, env_extra=None, stale=None):
+ import json
+ import subprocess
+ td = TemporaryDirectory()
+ self.addCleanup(td.cleanup)
+ root = Path(td.name) / 'root'
+ (root / 'test' / 'hil').mkdir(parents=True)
+ (root / 'test' / 'hil' / 'hil_test.py').touch()
+ (root / 'examples').mkdir(parents=True, exist_ok=True)
+ for b in built:
+ (root / 'examples' / f'cmake-build-{b}').mkdir(parents=True)
+ entries = [{'name': b} for b in (roster if roster is not None else built)]
+ for e in entries:
+ for v in (variants or {}).get(e['name'], []):
+ e.setdefault('variant', []).append({'name': v})
+ cfg = root / 'test' / 'hil' / 'cfg.json'
+ cfg.write_text(json.dumps({'boards': entries}))
+ stubs = Path(td.name) / 'bin'
+ stubs.mkdir()
+ # ssh joins its argv into ONE string that the REMOTE shell re-splits, and feeds the
+ # heredoc on stdin. A stub that echoes "$*" hides exactly that, which is how a
+ # completely broken env-forwarding change once passed its own test -- so this stub
+ # re-splits like the real thing and reports the script body separately.
+ write_script(stubs / 'ssh', 'shift; printf "REMOTE-ARGV: %s\\n" "$*" >&2; '
+ 'body=$(cat); printf "REMOTE-BODY: %s\\n" "$body" >&2; exit 0')
+ for tool in ('scp', 'rsync'):
+ write_script(stubs / tool, f'echo "stub-{tool} $*" >&2; exit 0')
+ for name, content in (stale or {}).items():
+ (root / name).write_text(content)
+ env = {**os.environ, 'REMOTE': 'stub', 'ROOT_DIR': str(root), 'CONFIG': str(cfg),
+ 'PATH': str(stubs) + os.pathsep + os.environ['PATH'], **(env_extra or {})}
+ r = subprocess.run(['bash', str(Path(TEST_DIR).parents[0] / 'hil_ci.sh'), *argv],
+ capture_output=True, text=True, timeout=60, env=env)
+ r.rsyncs = [l for l in r.stderr.splitlines() if l.startswith('stub-rsync')]
+ r.run_lines = [l for l in r.stderr.splitlines() if '--retry 1' in l]
+ r.body = '\n'.join(l for l in r.stderr.splitlines() if l.startswith('REMOTE-BODY'))
+ r.stale_left = {name: (root / name).exists() for name in (stale or {})}
+ return r
+
+ def test_long_board_forms_are_staged_and_only_that_board(self):
+ """Two boards are built so the pre-fix 'copy all built binaries' else-branch cannot
+ stage the right one by accident -- that is what made the first version of this test
+ pass against master while the feature was broken. -balpha is the glued short form
+ argparse resolves to --board alpha; unparsed it fell through to the all-boards branch
+ and silently staged everything built with no roster check."""
+ for argv in (['--board', 'alpha'], ['--board=alpha'], ['-balpha']):
+ with self.subTest(argv=argv):
+ r = self._run(argv, built=['alpha', 'beta'], roster=['alpha', 'beta'])
+ self.assertEqual(r.returncode, 0, r.stderr)
+ self.assertTrue(any('cmake-build-alpha ' in l for l in r.rsyncs),
+ f'{argv} never staged: {r.rsyncs}')
+ self.assertFalse(any('cmake-build-beta ' in l for l in r.rsyncs),
+ f'{argv} staged an unrequested board: {r.rsyncs}')
+
+ def test_board_test_flag_is_not_mistaken_for_a_board(self):
+ """-bt is hil_test.py's --board-test and is exactly what <config>.failed contains, so
+ a glued -b?* pattern turns the documented retry into 'not in the roster: t'."""
+ for argv in (['-b', 'alpha', '-bt', 'alpha:device/cdc_msc'],
+ ['-b', 'alpha', '-btalpha:device/cdc_msc']):
+ with self.subTest(argv=argv):
+ r = self._run(argv, built=['alpha'])
+ self.assertEqual(r.returncode, 0, r.stderr)
+ self.assertNotIn('not in', r.stderr)
+ self.assertTrue(any('alpha:device/cdc_msc' in l for l in r.run_lines),
+ f'-bt never reached the rig: {r.run_lines}')
+
+ def test_a_board_outside_the_roster_is_refused_before_staging(self):
+ r = self._run(['-b', 'alpha', '-b', 'ghost'], built=['alpha', 'ghost'], roster=['alpha'])
+ self.assertNotEqual(r.returncode, 0)
+ self.assertIn('ghost', r.stdout + r.stderr)
+ self.assertEqual(r.rsyncs, [], 'staged despite an unknown board')
+ self.assertEqual(r.run_lines, [], 'reached the run despite an unknown board')
+
+ def test_a_variant_with_no_build_dir_warns_instead_of_passing_silently(self):
+ r = self._run(['-b', 'alpha'], built=['alpha'], variants={'alpha': ['alpha-DMA']})
+ self.assertEqual(r.returncode, 0, r.stderr)
+ self.assertIn('alpha-DMA', r.stderr)
+ self.assertIn('skipped, not tested', r.stderr)
+
+ def test_no_build_dirs_at_all_aborts_the_all_boards_form(self):
+ """`hil_ci.sh` with no -b stages everything built. With nothing built it used to wipe
+ the rig, stage nothing, and return a green all-skip table."""
+ r = self._run([], built=[], roster=['alpha'])
+ self.assertNotEqual(r.returncode, 0)
+ self.assertIn('nothing to test', r.stdout + r.stderr)
+ self.assertEqual(r.run_lines, [], 'reached the run with nothing staged')
+ self.assertNotIn('Setting up remote', r.stdout + r.stderr,
+ 'the guard fired only after the remote tree was already wiped')
+
+ def test_hil_env_reaches_the_rig_as_environment_not_argv(self):
+ """An authorized force is HIL_NO_BOARD_LOCK=1. Passed through ssh's argv it arrives as a
+ positional argument and argparse exits 2, so it has to travel in the script body."""
+ r = self._run(['-b', 'alpha'], built=['alpha'], env_extra={'HIL_NO_BOARD_LOCK': '1'})
+ self.assertEqual(r.returncode, 0, r.stderr)
+ # one %q-quoted word of `export NAME=value; ` fragments, evaluated by the remote —
+ # NOT a bare NAME=value element, which hil_test.py's argparse takes as a positional.
+ # %q backslash-escapes the spaces, so match the pieces rather than the plain phrase.
+ run = '\n'.join(r.run_lines)
+ self.assertIn('HIL_NO_BOARD_LOCK=1', run)
+ self.assertIn('export', run)
+ self.assertFalse(any(' HIL_NO_BOARD_LOCK=1 ' in l for l in r.run_lines),
+ 'env reached argv unquoted, where hil_test.py sees a positional')
+
+ def test_a_value_with_spaces_survives_forwarding(self):
+ r = self._run(['-b', 'alpha'], built=['alpha'],
+ env_extra={'HIL_SCRATCH': '/tmp/my scratch'})
+ self.assertEqual(r.returncode, 0, r.stderr)
+ run = '\n'.join(r.run_lines)
+ self.assertIn('HIL_SCRATCH', run)
+ self.assertIn('scratch', run)
+
+ def test_hil_report_dir_is_never_forwarded(self):
+ """Where the report lands on the rig is this script's contract (REMOTE_DIR, where all
+ three copy-backs look); forwarding a local HIL_REPORT_DIR relocates it there and every
+ copy-back comes home empty -- two of the three silently."""
+ r = self._run(['-b', 'alpha'], built=['alpha'],
+ env_extra={'HIL_REPORT_DIR': '/tmp/elsewhere'})
+ self.assertEqual(r.returncode, 0, r.stderr)
+ self.assertFalse(any('HIL_REPORT_DIR' in l for l in r.run_lines),
+ f'HIL_REPORT_DIR reached the rig: {r.run_lines}')
+
+ def test_a_stale_local_failed_spec_does_not_survive_a_green_run(self):
+ """A green run writes no .failed on the rig, so the copy-back scp no-ops; the local
+ spec from a previous FAILED run must not survive it looking current -- a later
+ "retry from the spec" would re-flash boards that already passed."""
+ r = self._run(['-b', 'alpha'], built=['alpha'],
+ stale={'cfg.json.failed': '--accumulate -b alpha'})
+ self.assertEqual(r.returncode, 0, r.stderr)
+ self.assertFalse(r.stale_left['cfg.json.failed'],
+ "last run's re-run spec survived a green run")
+
+
+class SummaryFoldsReportToBoards(unittest.TestCase):
+ """hil_summary.py replaces the agent retyping the markdown table. Report rows are named per
+ VARIANT and a variant need not start with the board name, so the config is what maps them
+ back -- the previous string-matching design produced a defect in each of four review rounds."""
+
+ def _sum(self, boards, rows, cfg_boards=None, banner=''):
+ import json
+ import subprocess
+ td = TemporaryDirectory()
+ self.addCleanup(td.cleanup)
+ d = Path(td.name)
+ (d / 'hil_report.json').write_text(json.dumps(
+ {'rows': [{'board': b, 'cells': c, 'duration': '1s'} for b, c in rows],
+ 'banner': banner}))
+ cfg = d / 'cfg.json'
+ cfg.write_text(json.dumps({'boards': cfg_boards or [{'name': b} for b in boards]}))
+ args = [a for b in boards for a in ('-b', b)]
+ r = subprocess.run(['python3', str(Path(TEST_DIR).parents[0] / 'helper' / 'hil_summary.py'),
+ str(cfg), *args, '--report-dir', str(d)],
+ capture_output=True, text=True, timeout=60)
+ self.assertEqual(r.returncode, 0, r.stderr)
+ return json.loads(r.stdout)['results']
+
+ def test_variant_rows_fold_onto_their_board(self):
+ """nanoch32v203 never produces a row named after the board."""
+ got = self._sum(['nanoch32v203'],
+ [('nanoch32v203-fsdev', {'usbtest': 'pass'}),
+ ('nanoch32v203-usbfs', {'usbtest': 'pass'})],
+ cfg_boards=[{'name': 'nanoch32v203',
+ 'variant': [{'name': 'nanoch32v203-fsdev'},
+ {'name': 'nanoch32v203-usbfs'}]}])
+ self.assertEqual([r['board'] for r in got], ['nanoch32v203'])
+ self.assertTrue(got[0]['pass'])
+ self.assertTrue(got[0]['ran'])
+
+ def test_one_failing_variant_fails_the_board(self):
+ got = self._sum(['nano'],
+ [('nano-a', {'usbtest': 'pass'}), ('nano-b', {'usbtest': '❌ 29/30'})],
+ cfg_boards=[{'name': 'nano', 'variant': [{'name': 'nano-a'},
+ {'name': 'nano-b'}]}])
+ self.assertFalse(got[0]['pass'])
+ self.assertIn('29/30', got[0]['detail'])
+
+ def test_lock_contention_is_a_field_not_a_prefix(self):
+ got = self._sum(['alpha'], [('alpha', {'board-locked': 'fail'})])
+ self.assertTrue(got[0]['locked'])
+ self.assertFalse(got[0]['pass'])
+
+ def test_a_board_with_no_row_is_marked_not_run(self):
+ got = self._sum(['alpha', 'beta'], [('alpha', {'usbtest': 'pass'})])
+ self.assertTrue(got[0]['ran'])
+ self.assertFalse(got[1]['ran'])
+ self.assertFalse(got[1]['pass'])
+
+ def test_a_metric_cell_counts_by_its_icon(self):
+ got = self._sum(['a', 'b'], [('a', {'cdc_msc_throughput': '✅ C 1.2 M 3.4'}),
+ ('b', {'cdc_msc_throughput': '❌ C 0.0 M 0.0'})])
+ self.assertTrue(got[0]['pass'])
+ self.assertFalse(got[1]['pass'])
+
+ def test_skipped_cells_do_not_fail_a_board(self):
+ got = self._sum(['a'], [('a', {'usbtest': 'skip', 'cdc_msc': 'pass'})])
+ self.assertTrue(got[0]['pass'])
+
+ def test_a_plain_metric_cell_is_a_pass(self):
+ """Mirrors hil_test.py's own tally (cell_kind): failures are ALWAYS marked -- 'fail'
+ or a ❌ prefix, per TestFail's docstring -- while a passing test may return a plain
+ metric string that lands in the cell unprefixed. Treating unknown shapes as fail
+ would publish a green table as a red verdict."""
+ got = self._sum(['a'], [('a', {'device_speed': '480.0 MBps'})])
+ self.assertTrue(got[0]['pass'])
+
+ def test_a_declared_variant_of_another_board_is_not_stolen(self):
+ """A declared variant need not start with its own board's name, so it may start with
+ a DIFFERENT board's name plus '-'. The prefix fallback must not attribute it twice."""
+ got = self._sum(['alpha', 'beta'],
+ [('beta-x', {'usbtest': 'fail'})],
+ cfg_boards=[{'name': 'alpha', 'variant': [{'name': 'beta-x'}]},
+ {'name': 'beta'}])
+ self.assertTrue(got[0]['ran'])
+ self.assertFalse(got[0]['pass'])
+ self.assertFalse(got[1]['ran'], "beta must not inherit alpha's row")
+
+
class CaveatSurvivesAccumulate(unittest.TestCase):
"""CI reruns with --accumulate: the sidecar keeps every earlier attempt's cells, but the
banner was recomputed per attempt. A first attempt on a degraded rig and a clean rerun