diff options
| author | hathach <[email protected]> | 2026-08-20 16:43:49 +0700 |
|---|---|---|
| committer | hathach <[email protected]> | 2026-08-20 16:43:49 +0700 |
| commit | 6905639b07c69fec68e9ebc77f7d27ac2775ee41 (patch) | |
| tree | 2d46b0b6ae45b27df4874583c5830997df5829bd | |
| parent | 7800876bf151a239046521232dc4603b156061be (diff) | |
hil: run every board in one hil_test.py and hand results across as JSON
hil-validate ran one hil-operator per board. That parallelizes at the wrong layer:
hil_test.py already schedules boards across host controllers and budgets concurrent
flashes and usbtest batteries per controller (FLASH_PARALLEL/USBTEST_PARALLEL), and
those permits live in one process - N parallel runs multiply the budget onto the same
uPD720201 cards for no wall-clock gain over one run that already parallelizes. The
workflow now spawns ONE operator with every board as repeated -b.
The operator no longer retypes the report table. Four consecutive max-effort review
rounds found ~15 defects in this file and every one was in reconstructing board
identity from transcribed prose: report rows are named per VARIANT (nanoch32v203 only
ever produces -fsdev/-usbfs rows), a variant need not start with its board's name,
lock contention is a `board-locked` cell rather than a phrase, and each fix introduced
the next round's bug - including a fake-green test that asserted an invariant with the
one input shape that could not break it. The new helper test/hil/helper/hil_summary.py
does the join where the roster lives and emits one machine verdict per board
({board, ran, pass, locked, detail}); the operator returns that JSON verbatim plus
`wedged`, the only field it authors, and the workflow reads fields, never parses a
string. Its cell classifier mirrors hil_test.py's own tally exactly: failures are
always marked ('fail' or a ❌ prefix, TestFail's contract), everything unmarked is a
pass - a passing test may return a plain metric cell like '13443 KB/s', and the
mirrored rule is what keeps a green table from becoming a red verdict.
hil_ci.sh kept only the LAST -b, so multi-board remote runs staged one board's
binaries and every other board died on the rig after its lock and flash slot were
spent. It now parses every -b spelling argparse accepts (with the -bt arms ordered
first, longest-match, so the <config>.failed retry form is never read as a board named
"t..."), pre-flights roster membership and build dirs for ALL boards before anything
is wiped or staged, warns per declared variant with no build dir (which hil_test.py
would silently green-skip), forwards HIL_* knobs as export lines in one %q word the
remote evals ('; '-joined so it round-trips under dash - an authorized
HIL_NO_BOARD_LOCK force must not silently no-op), keeps HIL_REPORT_DIR local because
the copy-backs look in REMOTE_DIR, and copies hil_report.json and the .failed re-run
spec back beside the markdown, deleting stale local copies first so a green run cannot
leave last run's spec looking current.
Retries preserve the fleet: the documented path is the <config>.failed spec, which
already begins with --accumulate; a fresh scoped re-run would unlink the report and
collapse the whole-fleet table to the retried boards alone.
The risky logic is executable, not argued about: .claude/workflows/test-hil-validate.mjs
pins the lookup/verdict helpers and runs in pre-commit (hil-validate-logic); nine
staging tests drive hil_ci.sh through an ssh stub that models the real thing (argv
joined into one string the remote re-splits, heredoc on stdin - the naive echo-stub
passed while the feature was broken); and deliberate mutations of the verdict logic
are all caught.
Validated on the rig: a 2-board run (usbtest 30/30 on both; the pre-fix classifier,
replayed against that run's real report, fails the fully-green stm32f723disco on its
two passing '13443 KB/s' cells), the .failed --accumulate retry (merged report kept
every earlier row), and a 10-run soak over random subsets of a 22-board pool - 43
board-slots, every failure signature matched pre-existing CI state or known flake,
zero tooling failures, no locks left behind.
| -rw-r--r-- | .claude/agents/hil-operator.md | 49 | ||||
| -rw-r--r-- | .claude/skills/hil/SKILL.md | 18 | ||||
| -rw-r--r-- | .claude/workflows/hil-validate.js | 134 | ||||
| -rw-r--r-- | .claude/workflows/test-hil-validate.mjs | 59 | ||||
| -rw-r--r-- | .gitignore | 1 | ||||
| -rw-r--r-- | .pre-commit-config.yaml | 9 | ||||
| -rw-r--r-- | test/hil/helper/hil_summary.py | 115 | ||||
| -rw-r--r-- | test/hil/hil_ci.sh | 250 | ||||
| -rw-r--r-- | test/hil/test/test_hil_bounded.py | 317 |
9 files changed, 854 insertions, 98 deletions
diff --git a/.claude/agents/hil-operator.md b/.claude/agents/hil-operator.md index ebc9251cc..a37501211 100644 --- a/.claude/agents/hil-operator.md +++ b/.claude/agents/hil-operator.md @@ -15,14 +15,22 @@ You operate physical USB test hardware. These repo skills are your source of tru The GitHub Actions runner keeps running during your work. Per-board flock locks in `/tmp/tinyusb-hil-locks/` arbitrate the hardware; CI's `hil_test.py` fails fast on locked boards (re-runnable later). -- `python3 test/hil/hil_test.py ...` runs: do NOT pre-hold those boards — `hil_test.py` self-locks each board for its flash+test and would fail fast with `board locked` against your own hold. +- `python3 test/hil/hil_test.py ...` runs: do NOT pre-hold those boards — `hil_test.py` self-locks each board for its flash+test and would fail fast with `board locked` against your own hold. Several boards go into ONE run as repeated `-b`, never into several runs. - ANY other hardware action (JLinkExe/openocd/GDB, manual flash, usbtest.py, serial poking): hold first, release when done — release is mandatory cleanup (a crashed holder auto-releases via kernel flock, but do not rely on it): ```bash python3 test/hil/helper/hil_lock.py hold <board...> --reason "<task>" # ... hardware work ... python3 test/hil/helper/hil_lock.py release <board...> ``` -- Rig-wide operations (uhubctl power cycling, controller resets — they renumber buses): `python3 test/hil/helper/hil_lock.py hold --all --reason "<why>"` first. +- Rig-wide operations — uhubctl power cycling, `usb_recover.sh root-cycle`, pci-rebind, + controller resets — need `python3 test/hil/helper/hil_lock.py hold --all --config <this host's config> --reason "<why>"` + first, even a single root-port bounce. `--all` is coarse for a bounce, but it is the only + correct reservation available: the affected siblings are sysfs busports, nothing maps a + busport to a board name (the pool check's topology report counts devices per subtree, it does + not name them), and `hil_lock.py hold` validates nothing against the roster — so passing it + `13-1.6` creates a lock file for a board that does not exist and reserves nothing while + reporting success. If `--all` cannot be taken, wait: a partial hold is worse than none, + because it reads as protection. - If a lock is already held by someone else: report holder/reason (`hil_lock.py status`) — never force, never kill the holder. If the holder's reason is `hil_test.py`, that is a concurrent CI job mid-test on the board: waiting a few minutes and retrying once is appropriate when your task allows; otherwise return the holder info so the orchestrator can ask the user. - You cannot ask the user anything. Bypassing a lock (`HIL_NO_BOARD_LOCK=1`, or proceeding with manual hardware work despite a held lock) is allowed ONLY when your prompt explicitly states the user authorized forcing. @@ -33,11 +41,40 @@ The GitHub Actions runner keeps running during your work. Per-board flock locks Bash tool's 10 min foreground cap: run it in the background and wait for the completion notification. A foreground timeout kills the run before hil_test.py can write its report. NEVER cancel early. -- One hardware action at a time. You are never run concurrently with another hil-operator. -- On test failure: retry once with `-v -r 1` appended (one verbose attempt for diagnosis; a usbtest battery that produced per-case verdicts is NOT auto-retried, so its result already stands). If a board/fixture stops enumerating, or a tool of YOURS hangs in D state, consult usb-kernel-recover and capture `dmesg | tail -50` into `detail`; set `wedged` true. A `> **Rig note.**` banner reporting someone else's D-state process is not that — see the hil skill's banner list. +- One hardware action at a time. You are never run concurrently with another hil-operator, and a + multi-board `hil_test.py` run is ONE action: hand it every board as repeated `-b` and let it + schedule them — it round-robins boards across host controllers and budgets simultaneous flashes + and usbtest batteries per controller. Those budgets live in one process, so a second + `hil_test.py` alongside the first does not share them and the rig sees double the configured + width. (Do not read that as the cause of a dead card: hil_lock.py:128-131 records that every + observed uPD720201 death traced to a marginal DUT port bouncing under concurrent batteries, + and that lowering the widths does not fix a bad port — fix the port or pull the board.) +- On test failure, retry ONCE, with `-v` for diagnosis. Retry from the spec the run just wrote — + `<config>.failed`, which already begins with `--accumulate` and restricts each board to its + failed tests via `-bt`. If you compose the retry by hand you MUST pass `--accumulate` yourself: + a fresh run unlinks the report, so a hand-scoped `-b <board>` retry replaces the whole-fleet + table with a one-row table. A usbtest battery that produced per-case verdicts is NOT auto-retried, + so its result already stands. If a board/fixture stops enumerating, or a tool of YOURS hangs in D + state, consult usb-kernel-recover and capture `dmesg | tail -50` into `detail`; set `wedged` true. + A `> **Rig note.**` banner reporting someone else's D-state process is not that — see the hil + skill's banner list. ## Output contract -Your final message is parsed by a program. Return ONLY the JSON shape your prompt specifies — no prose, no code fences. Typical board-run shape: +Your final message is parsed by a program. Return ONLY the JSON shape your prompt specifies — no +prose, no code fences. -{"board": "raspberry_pi_pico", "pass": true, "detail": "<per-test summary or first failure>", "wedged": false} +For a board run, do NOT transcribe the report table. Run the tests, then hand back the machine +output verbatim: + +```bash +python3 test/hil/helper/hil_summary.py <config> -b BOARD [-b BOARD...] # from the report dir +``` + +`{"results": <its results array, verbatim>, "banner": <its banner, verbatim>, "wedged": ["board", ...]}` + +`results` and `banner` are copied, never retyped, reworded or re-ordered: report rows are named +per variant, a variant name need not start with the board name, and lock contention is a cell +rather than a phrase, so re-deriving any of it by hand is how this contract broke before. +`wedged` is yours — the boards your run left unresponsive, usually none — and the only field you +author. diff --git a/.claude/skills/hil/SKILL.md b/.claude/skills/hil/SKILL.md index 093f345b2..f0c449d33 100644 --- a/.claude/skills/hil/SKILL.md +++ b/.claude/skills/hil/SKILL.md @@ -32,7 +32,7 @@ python3 test/hil/helper/hil_lock.py release BOARD [BOARD...] ``` - Never pre-hold boards you are about to run `hil_test.py` on — it self-locks and would treat your own hold as a conflict. -- Rig-wide operations (uhubctl power cycling, controller resets — bus renumbering) affect every board: `hil_lock.py hold --all --reason "..."` first. +- Rig-wide operations (uhubctl power cycling, `usb_recover.sh root-cycle`, pci-rebind, controller resets — bus renumbering) affect every board: `hil_lock.py hold --all --config <this host's config> --reason "..."` first — `--all` defaults to `tinyusb.json`, so on `tusb` it would reserve 27 boards that do not exist there and none of the three that do. Even a single root-port bounce needs `--all`: nothing maps a sysfs busport to a board name, and `hil_lock.py hold` accepts any string, so a "just the siblings" hold reserves nothing while reporting success. - `hil_lock.py status` lists holders. Locks auto-release when the holder process dies (kernel flock); `/tmp` clears on reboot. - Forcing past a lock: `HIL_NO_BOARD_LOCK=1 python3 test/hil/hil_test.py ...` bypasses the guard without killing the holder. Only with the user's explicit go-ahead — they accept the risk of colliding with whatever holds the board. @@ -81,7 +81,7 @@ Examples must be built for the target board(s) — see CLAUDE.md "Build" → "Al ## Arguments -- **Board:** `-b BOARD_NAME` for one board; omit to run all boards in the config. +- **Board:** `-b BOARD_NAME`, repeatable for a subset (`-b a -b b`); omit to run all boards in the config. Give a whole set to ONE run rather than one run per board: it schedules the boards across host controllers and budgets concurrent flashes and usbtest batteries per controller (`hil_lock.py` `FLASH_PARALLEL`/`USBTEST_PARALLEL`). Those permits are in-process semaphores — a second `hil_test.py` running alongside does not share them, it multiplies the load on the same xHCI cards. - **Pass-through:** `-v`, `-r N`, etc. forwarded unchanged. If `local.json` is missing on a dev PC, ask the user to supply one (only fall back to `tinyusb.json` if told to). @@ -108,10 +108,13 @@ python3 test/hil/hil_test.py -b stm32f723disco -B examples "$CONFIG" # All boards: bash test/hil/hil_ci.sh -# A single board, with pass-through flags: -bash test/hil/hil_ci.sh -b raspberry_pi_pico2 -t host/cdc_msc_hid -r 1 +# A subset — repeat -b, ONE invocation for the whole set: +bash test/hil/hil_ci.sh -b raspberry_pi_pico2 -b stm32f723disco -t host/cdc_msc_hid -r 1 ``` +One invocation per board is wrong here, not merely slow: each run `rm -rf`s `REMOTE_DIR` +and rewrites the report, so only the last board's rows survive. + Env overrides: `REMOTE`, `REMOTE_DIR`, `CONFIG`. Fails fast if the build dir/repo layout is missing. ## Timing @@ -155,5 +158,8 @@ PREFIX, since each carries trailing detail and one is a blockquote: "device not found" from the named boards means "could not tell". Do NOT report their red cells as broken boards. -On failure, retry with `-v`; if that's not enough, add temporary debug prints to -`hil_test.py`. +On failure, retry once with `-v` — from the `<config>.failed` spec the run just wrote, which +already begins with `--accumulate` and restricts each board to its failed tests. A hand-scoped +`-b <board>` retry MUST pass `--accumulate` too: a fresh run unlinks the report, replacing the +whole-fleet table with a one-row table. If that is still not enough, add temporary debug prints +to `hil_test.py`. diff --git a/.claude/workflows/hil-validate.js b/.claude/workflows/hil-validate.js index 136f9075e..bc0bda8b9 100644 --- a/.claude/workflows/hil-validate.js +++ b/.claude/workflows/hil-validate.js @@ -1,60 +1,134 @@ export const meta = { name: 'hil-validate', - description: 'Serialized hardware-in-the-loop run: flash+test each board with hil-operator; per-board flock locks arbitrate with concurrent CI (the actions-runner keeps running)', - whenToUse: 'After validate passes, to exercise built firmware on the physical rig. Requires examples/cmake-build-<board> for each board. If the result has non-empty `locked`, ask the user: force (re-invoke with force: true), continue waiting (re-invoke later), or accept the partial result. Pass force: true ONLY with explicit user authorization.', - phases: [{ title: 'HIL', detail: 'strictly serial per-board hil-operator runs' }], + description: 'Hardware-in-the-loop run: one hil-operator flashes and tests every board in a single hil_test.py run; per-board flock locks arbitrate with concurrent CI (the actions-runner keeps running)', + whenToUse: 'After validate passes, to exercise built firmware on the physical rig. Requires the boards to be built (examples/cmake-build-<board>, plus a dir per declared variant). If the result has non-empty `locked`, ask the user: force (re-invoke with force: true), continue waiting (re-invoke later), or accept the partial result. Pass force: true ONLY with explicit user authorization.', + phases: [{ title: 'HIL', detail: 'one hil-operator, every board in one hil_test.py run' }], } // args: { boards: string[], force?: boolean } if (typeof args === 'string') { try { args = JSON.parse(args) } catch { /* not JSON: shape check below reports it */ } } if (!args || !Array.isArray(args.boards) || args.boards.length === 0) { - throw new Error('args must be { boards: string[], force? } with examples/cmake-build-<board> already built') + throw new Error('args must be { boards: string[], force? } with the boards already built') } +// The operator returns hil_summary.py's JSON verbatim plus its own observations. It does NOT +// retype the report table: rows are named per variant, a variant need not start with the board +// name, and lock contention is a cell rather than a phrase — rebuilding board identity from +// prose produced a defect in each of four review rounds. hil_summary.py does that join against +// the roster, so `locked` and `ran` arrive as fields and nothing here parses a detail string. +const BOARD = { + type: 'object', additionalProperties: false, + required: ['board', 'ran', 'pass', 'locked', 'detail'], + properties: { + board: { type: 'string' }, ran: { type: 'boolean' }, pass: { type: 'boolean' }, + locked: { type: 'boolean' }, detail: { type: 'string' }, + }, +} const HIL = { type: 'object', additionalProperties: false, - required: ['board', 'pass', 'detail', 'wedged'], + required: ['results', 'wedged'], properties: { - board: { type: 'string' }, pass: { type: 'boolean' }, - detail: { type: 'string' }, wedged: { type: 'boolean' }, + results: { type: 'array', items: BOARD }, + // the operator's own observation — not derivable from the report + wedged: { type: 'array', items: { type: 'string' } }, + banner: { type: 'string' }, }, } -const runBoard = (b) => agent( - `Run the HIL test for board ${b} per .claude/skills/hil/SKILL.md. Do NOT touch the actions-runner service and do NOT pre-hold the board lock — hil_test.py self-locks the board while testing. ` + +// ONE operator for the whole set, and one hil_test.py inside it. hil_test.py already runs the +// boards concurrently: it round-robins them across host controllers and holds per-controller +// permits (hil_lock.py FLASH_PARALLEL/USBTEST_PARALLEL) that bound simultaneous flashes and +// usbtest batteries. Those permits live in one process, so a second hil_test.py does not share +// them - N parallel single-board runs multiply the budget by N onto the same xHCI cards, for no +// wall-clock gain over one run that already parallelizes them. +const runBoards = (boards, isRetry = false) => agent( + `Run the HIL tests for these boards per .claude/skills/hil/SKILL.md: ${boards.join(', ')}. ` + + `Pass them ALL to ONE hil_test.py invocation as repeated -b flags (${boards.map((b) => `-b ${b}`).join(' ')}) — it schedules them across host controllers and budgets concurrent flashes and usbtest batteries itself. Never start a second hil_test.py alongside it. ` + + (isRetry + ? 'This is a RE-RUN of boards an earlier run could not take: pass --accumulate as well, or hil_test.py unlinks the report and the whole-fleet table collapses to just these boards. ' + : '') + + 'Do NOT touch the actions-runner service and do NOT pre-hold the board locks — hil_test.py self-locks each board for its flash+test. ' + (args.force ? 'THE USER HAS EXPLICITLY AUTHORIZED FORCING: run hil_test.py with HIL_NO_BOARD_LOCK=1 in the environment (bypasses the board lock check; do NOT release or kill the existing holder). ' - : 'If the run fails because the board lock is held (a dev session or concurrent CI job), report pass=false and set detail to start EXACTLY with "board locked:" followed by the holder JSON verbatim — never force the lock. ') + - 'Reserve the phrase "board locked" strictly for lock contention; describe a frozen or non-enumerating board as "unresponsive" instead. ' + - `Firmware is in examples/cmake-build-${b}. Use the config for this host (hostname first), single-board flag -b ${b}. Run hil_test.py as a BACKGROUND Bash task and wait for it (a stuck fleet runs to its pool guard, 60 min by default — beyond any foreground timeout); never cancel it early. ` + - 'On non-lock failures retry once with -v -r 1 (one verbose attempt for diagnosis; note a usbtest battery that produced per-case verdicts is NOT auto-retried, so its result already stands). wedged=true if the board/fixture is unresponsive after the run (capture dmesg | tail -50 into detail).', - { label: `hil:${b}`, phase: 'HIL', agentType: 'hil-operator', schema: HIL }, + : 'A board whose lock is held (a dev session or concurrent CI job) fails fast inside the run without blocking the others — never force the lock. ') + + 'If hil_test.py refuses the run with "board(s) not in <config>", re-run it WITHOUT the unknown names but keep the FULL board list on the hil_summary call below — it emits a ran:false entry for every board you name, so the unknown ones surface as "no report row" instead of costing the whole batch. ' + + 'Use the config for this host (hostname first). Run hil_test.py as a BACKGROUND Bash task and wait for it (a stuck fleet runs to its pool guard, 60 min by default — beyond any foreground timeout); never cancel it early. ' + + 'On non-lock failures retry ONCE from the re-run spec hil_test.py just wrote — `<config>.failed`, which already begins with --accumulate — adding -v. A usbtest battery that produced per-case verdicts is NOT auto-retried, so its result already stands. ' + + 'THEN, from the directory the run wrote its report to, produce the results with:\n' + + ` python3 test/hil/helper/hil_summary.py <the config you used> ${boards.map((b) => `-b ${b}`).join(' ')}\n` + + 'Return its `results` array and `banner` EXACTLY as printed — do not retype, reword, re-order or "correct" them, and never transcribe the markdown table instead. ' + + 'Add `wedged`: the board names whose board or fixture your run left unresponsive (usually none). That is your own observation and the one field you author; put `dmesg | tail -50` in your reply text for any board you list.', + { + label: boards.length === 1 ? `hil:${boards[0]}` : `hil:${boards.length} boards`, + phase: 'HIL', agentType: 'hil-operator', schema: HIL, + }, ) -const results = [] -for (const b of args.boards) { - const r = await runBoard(b) - results.push(r || { board: b, pass: false, detail: 'hil-operator agent died', wedged: false }) - log(`${b}: ${results[results.length - 1].pass ? 'PASS' : 'FAIL'}`) -} +// A lookup, not a reconciliation: hil_summary.py emits exactly one entry per requested board, +// so a missing entry means the operator dropped it rather than that the names disagree. +const byBoard = (out) => new Map((out?.results || []) + .filter((r) => r && typeof r.board === 'string') + .map((r) => [r.board, r])) + +// `wedged` is the one field the operator authors, so it may echo a report row name +// ('nano-fsdev') where the prompt asked for a board name. Accept the variant spelling +// rather than dropping a wedge over it -- only `wedged` sends anyone to usb-kernel-recover. +const wedgedFor = (list, b) => (Array.isArray(list) ? list : []) + .some((w) => typeof w === 'string' && (w === b || w.startsWith(`${b}-`))) + +const first = await runBoards(args.boards) +const firstRows = byBoard(first) +const firstWedged = first?.wedged || [] +const results = args.boards.map((b) => { + const r = firstRows.get(b) + if (!r) { + return { + board: b, pass: false, locked: false, ran: false, wedged: wedgedFor(firstWedged, b), + detail: first ? 'hil-operator returned no entry for this board' : 'hil-operator agent died', + } + } + return { ...r, wedged: wedgedFor(firstWedged, b) } +}) +for (const r of results) log(`${r.board}: ${r.pass ? 'PASS' : r.locked ? 'LOCKED' : 'FAIL'}`) +if (first?.banner) log(`report banner: ${first.banner.trim().split('\n')[0]}`) // A concurrent CI job may have held some boards (its hil_test.py flock). // CI finishes a board in minutes — retry locked boards once, at the end. if (!args.force) { - for (let i = 0; i < results.length; i++) { - if (results[i].pass || !results[i].detail.startsWith('board locked')) continue - log(`${results[i].board}: was locked — retrying once`) - const r = await runBoard(results[i].board) - if (r) results[i] = r - else results[i].detail += ' (retry operator died)' - log(`${results[i].board}: retry ${results[i].pass ? 'PASS' : 'FAIL'}`) + const relock = results.filter((r) => r.locked && !r.pass).map((r) => r.board) + if (relock.length) { + log(`was locked, retrying once: ${relock.join(', ')}`) + const again = await runBoards(relock, true) + const rows = byBoard(again) + const againWedged = again?.wedged || [] + for (let i = 0; i < results.length; i++) { + const b = results[i].board + if (!relock.includes(b)) continue + const r = rows.get(b) + // No entry keeps the board `locked`, so it still reaches the user's force/wait/accept + // decision instead of being published as a hardware failure. + if (r) { + // a passing retry does NOT clear a wedge the first run left behind: only `wedged` + // sends anyone to usb-kernel-recover + results[i] = { ...r, wedged: results[i].wedged || wedgedFor(againWedged, b) } + } else { + results[i].detail += again ? ' (retry returned no entry)' : ' (retry operator died)' + } + log(`${b}: retry ${results[i].pass ? 'PASS' : 'FAIL'}`) + } } } -const wedged = results.filter(r => r.wedged).map(r => r.board) +// pass/wedged/locked in one place so it can be exercised without running an agent +const summarize = (rs, force) => ({ + pass: rs.every((r) => r.pass), + wedged: rs.filter((r) => r.wedged).map((r) => r.board), + locked: force ? [] : rs.filter((r) => !r.pass && r.locked).map((r) => r.board), +}) + +const { pass, wedged, locked } = summarize(results, args.force) if (wedged.length) log(`WEDGED boards needing usb-kernel-recover: ${wedged.join(', ')}`) // Workers cannot prompt the user — surface still-locked boards for the main // session to ask: force (re-invoke with force: true), wait, or accept. -const locked = args.force ? [] : results.filter(r => !r.pass && r.detail.startsWith('board locked')).map(r => r.board) if (locked.length) log(`still locked after retry: ${locked.join(', ')} — ask the user: force / keep waiting / accept`) -return { pass: results.every(r => r.pass), results, wedged, locked } +return { pass, results, wedged, locked } diff --git a/.claude/workflows/test-hil-validate.mjs b/.claude/workflows/test-hil-validate.mjs new file mode 100644 index 000000000..80f37b27d --- /dev/null +++ b/.claude/workflows/test-hil-validate.mjs @@ -0,0 +1,59 @@ +// Executable checks for hil-validate.js's result handling. +// +// The join that used to live here -- matching variant row names to boards, parsing +// `board locked` out of a prose detail, folding rows, keeping a wedged flag alive -- produced +// a defect in each of four review rounds, including a test that asserted an invariant using +// the one input shape that could not break it. That logic now lives in +// test/hil/helper/hil_summary.py, where the roster is, and arrives here as fields. What is +// left is a lookup and a verdict, and this pins both. +// +// Run: node .claude/workflows/test-hil-validate.mjs +import { readFileSync } from 'node:fs' + +const src = readFileSync(new URL('./hil-validate.js', import.meta.url), 'utf8') +const body = src.slice(src.indexOf('const byBoard ='), src.indexOf('const first = await runBoards')) + + src.slice(src.indexOf('const summarize ='), src.indexOf('const { pass, wedged, locked } =')) +const { byBoard, summarize, wedgedFor } = new Function(`${body}; return { byBoard, summarize, wedgedFor }`)() + +let failed = 0 +const check = (name, got, want) => { + const g = JSON.stringify(got), w = JSON.stringify(want) + if (g === w) return console.log(` ok ${name}`) + failed++ + console.log(` FAIL ${name}\n got ${g}\n want ${w}`) +} +const R = (board, pass, locked = false, wedged = false, detail = '') => + ({ board, ran: true, pass, locked, detail, wedged }) + +console.log('byBoard — indexing the operator payload') +check('indexes by board', [...byBoard({ results: [R('a', true)] }).keys()], ['a']) +check('null payload', [...byBoard(null).keys()], []) +check('missing results', [...byBoard({}).keys()], []) +check('rows not an array', [...byBoard({ results: null }).keys()], []) +for (const bad of [[null], [undefined], [{ pass: true }], [{ board: 42 }]]) { + try { check(`malformed row ${JSON.stringify(bad)}`, [...byBoard({ results: bad }).keys()], []) } + catch (e) { failed++; console.log(` FAIL malformed row threw ${e}`) } +} + +console.log('wedgedFor — operator-authored names, variant spellings tolerated') +check('board name matches', wedgedFor(['nano'], 'nano'), true) +check('variant spelling matches', wedgedFor(['nano-fsdev'], 'nano'), true) +check('another board does not', wedgedFor(['other'], 'nano'), false) +check('prefix without dash does not', wedgedFor(['nanoch32'], 'nano'), false) +check('null list', wedgedFor(null, 'nano'), false) +check('non-string entry does not throw', wedgedFor([42, 'nano'], 'nano'), true) + +console.log('summarize — the ship/no-ship verdict') +check('all pass', summarize([R('a', true)], false).pass, true) +check('one fail sinks it', summarize([R('a', true), R('b', false)], false).pass, false) +check('a locked board is not a pass', summarize([R('a', false, true)], false).pass, false) +check('locked is a field, not a prefix', summarize([R('a', false, true)], false).locked, ['a']) +check('a real failure is not locked', summarize([R('a', false, false)], false).locked, []) +check('a PASSING board is never locked', summarize([R('a', true, true)], false).locked, []) +check('force zeroes locked', summarize([R('a', false, true)], true).locked, []) +check('wedged surfaces', summarize([R('a', false, false, true)], false).wedged, ['a']) +check('a wedged board that passed still surfaces', + summarize([R('a', true, false, true)], false).wedged, ['a']) + +console.log(failed ? `\n${failed} FAILED` : '\nall checks passed') +process.exit(failed ? 1 : 0) diff --git a/.gitignore b/.gitignore index 8773322e4..f6c8702b6 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,7 @@ html latex hil_report.md hil_report.json +*.json.failed *.a *.d *.o diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 3d9c8482b..17170b7d6 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -62,6 +62,15 @@ repos: entry: python3 -m unittest discover -s test/hil/test pass_filenames: false language: system + # hil-validate.js decides which boards ship. Its result join has been wrong three times -- + # dropping variant-named rows, letting a PASS erase a FAIL, breaking the `board locked` + # anchor it had just fixed -- each time because the logic was reasoned about instead of run. + - id: hil-validate-logic + name: hil-validate-logic + files: ^\.claude/workflows/ + entry: node .claude/workflows/test-hil-validate.mjs + pass_filenames: false + language: system - id: hil-select-test name: hil-select-test files: ^(hw/bsp/|src/|examples/) diff --git a/test/hil/helper/hil_summary.py b/test/hil/helper/hil_summary.py new file mode 100644 index 000000000..e566bead0 --- /dev/null +++ b/test/hil/helper/hil_summary.py @@ -0,0 +1,115 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +"""Fold hil_report.json into one machine-readable verdict per BOARD. + +A workflow driving hil_test.py through an operator agent has no filesystem access, so the +agent has to carry the results across. It must carry them, not retype them: the previous +design asked the agent to transcribe the markdown table, and every defect found in four +review rounds came from re-parsing that prose -- variant row names vs board names, +`board locked` vs `board-locked`, folding several variant rows into one verdict, rows that +matched no board. All of it is a join, and the join belongs here, where the roster is. + +Report rows are named per VARIANT (hil_test.py builds them from `vname`), and a variant name +is not required to start with the board name -- nanoch32v203 produces only `-fsdev`/`-usbfs`, +ch32v307v_r1_1v0 only `-usbhs`/`-usbfs`. The config is what maps them back. + +Emits, on stdout: + {"results": [{"board", "ran", "pass", "locked", "detail"}...], "banner": str} + +`locked` is a field, not a prefix to grep for. `ran` false means the board produced no row at +all, which is not the same as failing. + +Usage: hil_summary.py <config.json> [-b BOARD]... [--report-dir DIR] +""" +import argparse +import json +import sys +from pathlib import Path + +FAIL_ICON, SKIP_ICON = '❌', '⚪' # a pass needs no icon: unmarked = pass +LOCKED_CELL = 'board-locked' + + +def cell_state(v: str) -> str: + """'pass' | 'fail' | 'skip' -- the EXACT classifier hil_test.py's own tally uses + (cell_kind in render_matrix): 'fail' or a ❌ prefix is a failure, 'skip' or a ⚪ + prefix is a skip, and EVERYTHING ELSE is a pass. That last arm is load-bearing: a + passing test may return a plain metric string ('480.0 MBps') that lands in the cell + unprefixed, while failures are guaranteed marked -- TestFail's docstring pins that its + metric is icon-prefixed precisely so render/tally treat it as a failure. Classifying + unknown shapes as fail here would publish a green table as a red verdict.""" + if v == 'fail' or v.startswith(FAIL_ICON): + return 'fail' + if v == 'skip' or v.startswith(SKIP_ICON): + return 'skip' + return 'pass' + + +def variants_of(cfg: dict, board: str) -> list: + for b in cfg.get('boards', []): + if b['name'] == board: + return [v['name'] for v in (b.get('variant') or [])] or [board] + return [board] + + +def summarize(cfg: dict, boards: list, report: dict) -> dict: + rows = {r['board']: r.get('cells') or {} for r in report.get('rows', [])} + owner = {v['name']: b['name'] for b in cfg.get('boards', []) + for v in (b.get('variant') or [])} + results = [] + for board in boards: + names = variants_of(cfg, board) + mine = {n: rows[n] for n in names if n in rows} + # a variant name that is neither declared nor prefixed cannot be attributed; the + # `<board>-` fallback only helps ad-hoc builds, it is not the primary path. It must + # also never steal a row DECLARED by another board: a declared variant need not start + # with its own board's name, so it may happen to start with this board's name plus '-'. + mine.update({n: c for n, c in rows.items() + if n.startswith(f'{board}-') and n not in mine + and owner.get(n, board) == board}) + if not mine: + results.append({'board': board, 'ran': False, 'pass': False, 'locked': False, + 'detail': 'no report row for this board'}) + continue + locked = any(LOCKED_CELL in cells for cells in mine.values()) + bad = [] + for vname, cells in sorted(mine.items()): + for test, val in sorted(cells.items()): + if test == LOCKED_CELL: + continue + if cell_state(str(val)) == 'fail': + bad.append(f'{vname} {test}: {val}') + ok = not bad and not locked + if locked: + detail = 'held by another holder; not flashed' + elif bad: + detail = '; '.join(bad) + else: + detail = f'{len(mine)} variant(s), {sum(len(c) for c in mine.values())} cell(s) ok' + results.append({'board': board, 'ran': True, 'pass': ok, 'locked': locked, + 'detail': detail}) + return {'results': results, 'banner': report.get('banner', '')} + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument('config_file') + ap.add_argument('-b', '--board', action='append', default=[], + help='boards to report on; default: every board in the config') + ap.add_argument('--report-dir', default='.', help='where hil_report.json lives (default: cwd)') + a = ap.parse_args() + + cfg = json.loads(Path(a.config_file).read_text()) + boards = a.board or [b['name'] for b in cfg.get('boards', [])] + jpath = Path(a.report_dir) / 'hil_report.json' + if not jpath.is_file(): + print(f'error: {jpath} not found -- did hil_test.py run in this directory?', + file=sys.stderr) + return 1 + json.dump(summarize(cfg, boards, json.loads(jpath.read_text())), sys.stdout, indent=2) + print() + return 0 + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/test/hil/hil_ci.sh b/test/hil/hil_ci.sh index c7dfa95df..f474c3055 100644 --- a/test/hil/hil_ci.sh +++ b/test/hil/hil_ci.sh @@ -1,8 +1,9 @@ #!/usr/bin/env bash # Run HIL test remotely on ci.lan -# Usage: test/hil/hil_ci.sh [-b BOARD] [-t TEST] [extra hil_test.py args...] +# Usage: test/hil/hil_ci.sh [-b BOARD]... [-t TEST] [extra hil_test.py args...] # Example: # test/hil/hil_ci.sh -b stm32f723disco +# test/hil/hil_ci.sh -b stm32f723disco -b raspberry_pi_pico # test/hil/hil_ci.sh -b stm32f723disco -t host/cdc_msc_hid -r 1 # # Env overrides: REMOTE, REMOTE_DIR, CONFIG (path to HIL config json), @@ -44,17 +45,46 @@ for a in "$@"; do exit 1 done -# Parse -b BOARD from arguments to know which build to copy -BOARD="" +# Parse -b BOARD from arguments to know which builds to copy. Repeatable: hil_test.py +# takes the whole board set in ONE run (it schedules them across host controllers and +# budgets the flashes itself), so every -b needs its binaries staged, not just the last. +BOARDS=() ARGS=() while [[ $# -gt 0 ]]; do case "$1" in - -b) - [[ $# -ge 2 ]] || { echo "error: -b requires a BOARD argument" >&2; exit 1; } - BOARD="$2" + # hil_test.py declares `-b, --board` with action='append', so argparse also accepts + # --board=X and -bX. Recognising only the bare `-b X` forwarded the others to the rig + # while never staging them: the board ran with no firmware and reported a green row. + -b|--board) + [[ $# -ge 2 ]] || { echo "error: $1 requires a BOARD argument" >&2; exit 1; } + BOARDS+=("$2") ARGS+=("$1" "$2") shift 2 ;; + --board=*) + BOARDS+=("${1#--board=}") + ARGS+=("$1") + shift + ;; + # -bt (--board-test) BEFORE the glued -b?* arm, mirroring argparse's longest-match: it is + # the form <config>.failed uses, and a bare -b?* would register a board named "t..." that + # the roster check below rejects -- killing every documented retry. + -bt|--board-test) + [[ $# -ge 2 ]] || { echo "error: $1 requires NAME:tests" >&2; exit 1; } + ARGS+=("$1" "$2") + shift 2 + ;; + -bt?*|--board-test=*) + ARGS+=("$1") + shift + ;; + # glued short form: argparse resolves -bNAME to --board NAME, so staging must too -- + # unparsed it fell through to the all-boards branch and silently staged everything built + -b?*) + BOARDS+=("${1#-b}") + ARGS+=("$1") + shift + ;; *) ARGS+=("$1") shift @@ -62,6 +92,107 @@ while [[ $# -gt 0 ]]; do esac done +# Resolve a board to its build dirs: its own dir, the cmake-build-<board>-* glob (ad-hoc +# local builds), and the variant dirs named in $CONFIG -- variant names are NOT required to +# be prefixed with the board name, so the glob alone is not enough. Prints one dir per line. +variant_names() { + python3 -c ' +import json, sys +cfg = json.load(open(sys.argv[1])) +for b in cfg.get("boards", []): + if b["name"] == sys.argv[2]: + for v in b.get("variant") or []: + print(v["name"]) +' "$CONFIG" "$1" +} + +resolve_build_dirs() { + local board="$1" d v + declare -A seen=() + shopt -s nullglob + for d in "$ROOT_DIR"/examples/cmake-build-"$board" "$ROOT_DIR"/examples/cmake-build-"$board"-*; do + [[ -d $d && -z ${seen[$d]:-} ]] && { seen[$d]=1; printf '%s\n' "$d"; } + done + shopt -u nullglob + # to a file, not a process substitution: `set -e`/pipefail cannot see the exit status of + # the latter, so a malformed roster silently yielded zero variant dirs + local vf; vf=$(mktemp) + variant_names "$board" > "$vf" || { rm -f "$vf"; echo "Error: could not read variants for $board from $CONFIG" >&2; exit 1; } + while IFS= read -r v; do + d="$ROOT_DIR/examples/cmake-build-$v" + [[ -d $d && -z ${seen[$d]:-} ]] && { seen[$d]=1; printf '%s\n' "$d"; } + done < "$vf" + rm -f "$vf" +} + +# Pre-flight: EVERY board must resolve to at least one build dir before anything is wiped or +# copied. This check used to live in the copy loop, so an unbuilt board late in the list +# aborted the run after the remote tree had been rm -rf'd and earlier boards fully rsynced -- +# zero coverage, a half-staged rig, and a stale local hil_report.md left in place. Report all +# missing boards at once so one build round fixes them. +MANIFEST=$(mktemp) +trap 'rm -f "$MANIFEST"' EXIT +# Roster membership first: hil_test.py rejects an unknown -b with sys.exit(1) for the WHOLE +# run (hil_test.py:2297), and it does so AFTER this script has wiped REMOTE_DIR and staged +# every board -- one typo then costs the entire batch. We already parse $CONFIG here, so +# catch it before anything is touched. Note -b matches board names only, never variant names. +if [ ${#BOARDS[@]} -gt 0 ]; then +ROSTER=$(python3 -c ' +import json, sys +print("\n".join(b["name"] for b in json.load(open(sys.argv[1])).get("boards", []))) +' "$CONFIG") || { echo "error: could not read the board roster from $CONFIG" >&2; exit 1; } +notinroster=() +for b in ${BOARDS[@]+"${BOARDS[@]}"}; do + grep -qxF -- "$b" <<< "$ROSTER" || notinroster+=("$b") +done +if [ ${#notinroster[@]} -gt 0 ]; then + echo "error: not in $(basename "$CONFIG"): ${notinroster[*]}" >&2 + echo " (-b takes board names, not variant names)" >&2 + exit 1 +fi +fi # BOARDS non-empty: nothing to validate for an all-boards run + +missing=() +for b in ${BOARDS[@]+"${BOARDS[@]}"}; do + dirs=$(resolve_build_dirs "$b") + if [ -z "$dirs" ]; then + missing+=("$b") + else + while IFS= read -r d; do printf '%s\t%s\n' "$b" "$d" >> "$MANIFEST"; done <<< "$dirs" + # A declared variant with no build dir is NOT an error -- no cmake preset is + # variant-suffixed, so this is the normal state for e.g. the -DMA variants. It is worth + # saying out loud: hil_test.py logs `Skip (no binary)` and counts zero errors for it, so + # the run exits 0 and the operator reads a green table for cells that never ran. + while IFS= read -r v; do + [ -z "$v" ] && continue + # whole lines: a substring match lets cmake-build-<v>-DMA silence the warning for <v> + grep -qxF -- "$ROOT_DIR/examples/cmake-build-$v" <<< "$dirs" \ + || echo "warning: $b variant '$v' has no build dir -- its cells will be skipped, not tested" >&2 + done < <(variant_names "$b") + fi +done +if [ ${#missing[@]} -gt 0 ]; then + echo "Error: no build directory under $ROOT_DIR/examples/ for: ${missing[*]}" >&2 + for b in "${missing[@]}"; do + echo " cd examples && cmake --preset $b && cmake --build --preset $b" >&2 + done + exit 1 +fi + +# The all-boards form needs its emptiness check HERE too: below the setup ssh it fired after +# the remote tree was already rm -rf'd, destroying the previous run's report and re-run spec +# on the rig before deciding there was nothing to do. +if [ ${#BOARDS[@]} -eq 0 ]; then + shopt -s nullglob + allbuilds=("$ROOT_DIR"/examples/cmake-build-*/) + shopt -u nullglob + if [ ${#allbuilds[@]} -eq 0 ]; then + echo "error: no examples/cmake-build-* directories under $ROOT_DIR -- nothing to test" >&2 + echo " build first, e.g.: cd examples && cmake --preset <board> && cmake --build --preset <board>" >&2 + exit 1 + fi +fi + # Setup remote directory. `bash -s` + heredoc so REMOTE_DIR arrives as a positional # parameter, keeping the `rm -rf` target out of the command string the heredoc runs. echo "==> Setting up remote $REMOTE:$REMOTE_DIR" @@ -89,6 +220,7 @@ scp -q "$ROOT_DIR/test/hil/helper/__init__.py" \ "$ROOT_DIR/test/hil/helper/hil_util.py" \ "$ROOT_DIR/test/hil/helper/hil_health.py" \ "$ROOT_DIR/test/hil/helper/hil_lock.py" \ + "$ROOT_DIR/test/hil/helper/hil_summary.py" \ "$ROOT_DIR/test/hil/helper/hil_select.py" \ "$REMOTE:$REMOTE_DIR/test/hil/helper/" @@ -103,52 +235,23 @@ copy_board_binaries() { "$src" "$REMOTE:$REMOTE_DIR/examples/" } -if [ -n "$BOARD" ]; then - # Copy the board's build dir plus its variant dirs. Variant names come from - # $CONFIG (they are not required to be prefixed with the board name); the - # cmake-build-<BOARD>-* glob is kept as a fallback for ad-hoc local builds. - # Collect only dirs that actually exist, deduplicated. - declare -A SEEN_DIRS=() - BUILD_DIRS=() - add_build_dir() { - [[ -d "$1" && -z "${SEEN_DIRS[$1]:-}" ]] || return 0 - SEEN_DIRS[$1]=1 - BUILD_DIRS+=("$1") - } - shopt -s nullglob - for d in "$ROOT_DIR"/examples/cmake-build-"$BOARD" "$ROOT_DIR"/examples/cmake-build-"$BOARD"-*; do - add_build_dir "$d" - done - shopt -u nullglob - # to a file, not a process substitution: `set -e`/pipefail cannot see the exit - # status of the latter, so a malformed roster silently yielded zero variant dirs - VARIANTS_FILE=$(mktemp) - python3 -c ' -import json, sys -cfg = json.load(open(sys.argv[1])) -for b in cfg.get("boards", []): - if b["name"] == sys.argv[2]: - for v in b.get("variant") or []: - print(v["name"]) -' "$CONFIG" "$BOARD" > "$VARIANTS_FILE" || { - echo "Error: could not read variants for $BOARD from $CONFIG" - rm -f "$VARIANTS_FILE" - exit 1 - } - while IFS= read -r v; do - add_build_dir "$ROOT_DIR/examples/cmake-build-$v" - done < "$VARIANTS_FILE" - rm -f "$VARIANTS_FILE" - if [ ${#BUILD_DIRS[@]} -eq 0 ]; then - echo "Error: no build directory found for $BOARD under $ROOT_DIR/examples/" - echo "Build first with: cd examples && cmake --preset $BOARD && cmake --build --preset $BOARD" - exit 1 - fi - echo "==> Copying binaries for $BOARD (${#BUILD_DIRS[@]} build dir(s))" - for d in "${BUILD_DIRS[@]}"; do - copy_board_binaries "$d" +if [ ${#BOARDS[@]} -gt 0 ]; then + # Replay the pre-flight manifest: the dirs were already resolved and proved non-empty + # for every board, so nothing here can abort mid-staging. Plain reads of the manifest -- + # a process substitution would hide a reader failure from set -e (the comment in + # resolve_build_dirs is about exactly that trap). + for b in "${BOARDS[@]}"; do + dirs=() + while IFS=$'\t' read -r bb d; do + [ "$bb" = "$b" ] && [ -n "$d" ] && dirs+=("$d") + done < "$MANIFEST" + echo "==> Copying binaries for $b (${#dirs[@]} build dir(s))" + for d in ${dirs[@]+"${dirs[@]}"}; do + copy_board_binaries "$d" + done done else + # emptiness was already refused in pre-flight, before the remote wipe echo "==> Copying all built binaries" # Use `%/` parameter expansion to strip the trailing slash from the glob — # rsync needs the bare dir name so the per-board cmake-build-<BOARD>/ subdir @@ -170,14 +273,38 @@ for a in ${ARGS[@]+"${ARGS[@]}"}; do ARGS_Q+=("$(printf '%q' "$a")"); done CONFIG_Q="$(printf '%q' "test/hil/$(basename "$CONFIG")")" echo "==> Running HIL test on $REMOTE" rc=0 -# --retry 1 FIRST, before the user's args: this targets the same shared rig CI uses, and -# the pool guard is a flat constant that does not scale with max_retry -- argparse's -# default of 3 lets a few flaky boards re-pay 510s each until the 3600s guard fires, -# abandoning the pool and holding board flocks against concurrent CI. Placed first, not -# appended, so argparse's last-wins means `hil_ci.sh -r 3` still gets 3. -ssh "$REMOTE" bash -s -- "$REMOTE_DIR" --retry 1 ${ARGS_Q[@]+"${ARGS_Q[@]}"} "$CONFIG_Q" <<'REMOTE' || rc=$? +# --retry 1 FIRST, before the user's args: this targets the same shared rig CI uses, and the +# pool guard is a flat constant that does not scale with max_retry, so a few flaky boards can +# re-pay ~510s each until the 3600s guard fires, abandoning the pool and holding board flocks +# against concurrent CI. hil_test.py's own default is already 1; passing it explicitly keeps +# that true if the default ever moves. Placed first, not appended, so argparse's last-wins +# means `hil_ci.sh -r 3` still gets 3. +# Forward the HIL_* knobs (HIL_NO_BOARD_LOCK for an authorized force, the parallel widths, +# HIL_POOL_TIMEOUT). ssh passes no environment and joins its argv into one string the remote +# shell re-splits, so a bare NAME=value element would arrive as a positional argument to +# hil_test.py and argparse would exit 2. Build `export` lines instead and hand them over as a +# single %q-quoted word for the remote to eval. +# Joined with '; ', NOT newlines: %q renders a newline as bash-only $'...' quoting, which the +# remote LOGIN shell must parse from the joined command string -- under dash the force arrives +# as garbage and silently does nothing. Backslash escaping round-trips in both shells. +# HIL_REPORT_DIR stays local: where the report lands on the rig is this script's contract +# (REMOTE_DIR, where all three copy-backs below look), so forwarding it would relocate the +# report and every copy-back would come home empty. +HIL_EXPORTS="" +while IFS= read -r v; do + [ -z "$v" ] && continue + HIL_EXPORTS+="export $(printf '%s=%q' "$v" "${!v}"); " +done < <(compgen -v | grep -x 'HIL_[A-Z0-9_]*' | grep -vxE 'HIL_EXPORTS|HIL_REPORT_DIR' || true) +[ -n "$HIL_EXPORTS" ] && echo "==> Forwarding: $HIL_EXPORTS" +# One %q-quoted word, so ssh's argv join and the remote shell's re-split hand it back +# byte-for-byte, and the remote evals it. Empty stays `''` -- a real, shiftable argument -- +# rather than vanishing from the joined string and shifting the run's own flags out of place. +HIL_EXPORTS_Q=$(printf '%q' "$HIL_EXPORTS") + +ssh "$REMOTE" bash -s -- "$REMOTE_DIR" "$HIL_EXPORTS_Q" --retry 1 ${ARGS_Q[@]+"${ARGS_Q[@]}"} "$CONFIG_Q" <<'REMOTE' || rc=$? cd -- "$1" shift +eval "$1"; shift # HIL_* exports, %q-quoted locally into one word # Flasher CLIs live in the user bin dirs on ci.lan (esptool/idf in ~/.local/bin, # STM32CubeProgrammer's STM32_Programmer_CLI in ~/bin); the non-interactive shell # subprocess used for flashing doesn't source profile/rc, so add them explicitly. @@ -191,4 +318,15 @@ scp -q "$REMOTE:$REMOTE_DIR/hil_report.md" "$ROOT_DIR/hil_report.md" \ && echo "==> Report copied to $ROOT_DIR/hil_report.md" \ || echo "==> warning: no hil_report.md copied back" >&2 +# The re-run spec and the JSON sidecar live in the run's cwd on the rig (REMOTE_DIR), and the +# next invocation rm -rf's it. Without copying them back, the `--accumulate` retry every doc on +# this branch prescribes has nothing to read and nothing to merge onto. Delete the local copies +# FIRST: a green run writes no .failed, so a silent no-op scp would leave last run's spec in +# the checkout looking current, and "retry from the spec" would re-flash boards that passed. +for extra in "$(basename "$CONFIG").failed" hil_report.json; do + rm -f "$ROOT_DIR/$extra" + scp -q "$REMOTE:$REMOTE_DIR/$extra" "$ROOT_DIR/$extra" 2>/dev/null \ + && echo "==> $extra copied to $ROOT_DIR/$extra" || true +done + exit $rc diff --git a/test/hil/test/test_hil_bounded.py b/test/hil/test/test_hil_bounded.py index 908a142d5..c6d454f0e 100644 --- a/test/hil/test/test_hil_bounded.py +++ b/test/hil/test/test_hil_bounded.py @@ -1364,7 +1364,14 @@ class RemoteDirIsScreened(unittest.TestCase): rc = 0 if keep_going else 77 for tool in ('ssh', 'scp', 'rsync'): write_script(Path(td) / tool, f'echo "stub-{tool} $*" >&2; exit {rc}') + # hil_ci.sh now refuses an all-boards run with nothing built, so this arg-quoting + # test needs a checkout stub with one build dir to reach the run invocation + root = Path(td) / 'root' + (root / 'test' / 'hil').mkdir(parents=True) + (root / 'test' / 'hil' / 'hil_test.py').touch() + (root / 'examples' / 'cmake-build-alpha').mkdir(parents=True) env = {**os.environ, 'REMOTE_DIR': remote_dir, 'REMOTE': 'stub', + 'ROOT_DIR': str(root), 'PATH': td + os.pathsep + os.environ['PATH']} return subprocess.run( ['bash', str(Path(TEST_DIR).parents[0] / 'hil_ci.sh'), *args], @@ -1414,6 +1421,316 @@ class RemoteDirIsScreened(unittest.TestCase): self.assertIn(r'host/cdc\ msc', run_line) +class EveryBoardIsStaged(unittest.TestCase): + """One hil_test.py run takes several `-b` flags, and hil-operator hands it the whole board + set that way. The `-b` parse loop kept a single BOARD, so only the LAST board's binaries + were rsynced and every other board died on the rig with a missing firmware path -- after + its flash slot and lock were already spent. + + Three of these five fail against the pre-fix script (the discriminating unbuilt case + puts the board FIRST, because the old single-BOARD parse happened to handle a trailing + one correctly); the run-line and variant-dir tests are characterization -- the old script + already forwarded ARGS whole and read variants from the config for its one board.""" + + def _run(self, boards, cfg_boards=None, variants=None): + import json + import subprocess + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + root = Path(td.name) / 'root' + (root / 'test' / 'hil').mkdir(parents=True) + (root / 'test' / 'hil' / 'hil_test.py').touch() + built = cfg_boards if cfg_boards is not None else boards + (root / 'examples').mkdir(parents=True, exist_ok=True) + for b in built: + (root / 'examples' / f'cmake-build-{b}').mkdir(parents=True) + roster = [{'name': b} for b in boards] + for entry in roster: + for v in (variants or {}).get(entry['name'], []): + entry.setdefault('variant', []).append({'name': v}) + cfg = root / 'test' / 'hil' / 'cfg.json' + cfg.write_text(json.dumps({'boards': roster})) + stubs = Path(td.name) / 'bin' + stubs.mkdir() + # real ssh/scp/rsync would reach the rig; these just record the argv + for tool in ('ssh', 'scp', 'rsync'): + write_script(stubs / tool, f'echo "stub-{tool} $*" >&2; exit 0') + env = {**os.environ, 'REMOTE': 'stub', 'ROOT_DIR': str(root), 'CONFIG': str(cfg), + 'PATH': str(stubs) + os.pathsep + os.environ['PATH']} + args = [a for b in boards for a in ('-b', b)] + r = subprocess.run(['bash', str(Path(TEST_DIR).parents[0] / 'hil_ci.sh'), *args], + capture_output=True, text=True, timeout=60, env=env) + r.rsyncs = [l for l in r.stderr.splitlines() if l.startswith('stub-rsync')] + # the RUN ssh is the one carrying hil_test.py's args; the setup ssh is not + r.run_lines = [l for l in r.stderr.splitlines() if '--retry 1' in l] + return r + + def test_binaries_for_every_requested_board_are_copied(self): + r = self._run(['alpha', 'beta', 'gamma']) + self.assertEqual(r.returncode, 0, r.stderr) + for b in ('alpha', 'beta', 'gamma'): + self.assertTrue(any(f'cmake-build-{b} ' in l for l in r.rsyncs), + f'{b} binaries never staged: {r.rsyncs}') + + def test_every_board_reaches_hil_test(self): + r = self._run(['alpha', 'beta']) + self.assertEqual(len(r.run_lines), 1, r.stderr) + self.assertIn('-b alpha', r.run_lines[0]) + self.assertIn('-b beta', r.run_lines[0]) + + def test_an_unbuilt_board_aborts_before_anything_is_staged(self): + """The discriminating case: the unbuilt board is FIRST. The pre-fix script kept only + the last -b, found it built, and ran happily while silently testing one board. It also + has to fail BEFORE staging -- the old in-loop check fired after the remote tree was + wiped and earlier boards were rsynced, costing a run and leaving a half-staged rig.""" + r = self._run(['alpha', 'beta'], cfg_boards=['beta']) + self.assertNotEqual(r.returncode, 0, 'unbuilt first board was accepted') + self.assertIn('alpha', r.stdout + r.stderr) + self.assertEqual(r.rsyncs, [], f'staged despite an unbuilt board: {r.rsyncs}') + self.assertEqual(r.run_lines, [], 'reached the run despite an unbuilt board') + + def test_a_board_whose_firmware_is_only_a_variant_dir_is_accepted(self): + """Variant names are not required to be prefixed with the board name, so a board can + own no `cmake-build-<board>` dir at all. A pre-flight that only globs the board name + rejects it and tells the user to build firmware that is already there.""" + r = self._run(['alpha', 'beta'], cfg_boards=['alpha', 'odd-name-v'], + variants={'beta': ['odd-name-v']}) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertTrue(any('cmake-build-odd-name-v ' in l for l in r.rsyncs), + f"beta's variant dir never staged: {r.rsyncs}") + + def test_all_unbuilt_boards_are_named_at_once(self): + """One build round should fix every complaint, so the guard reports the whole set.""" + r = self._run(['alpha', 'beta', 'gamma'], cfg_boards=['beta']) + self.assertNotEqual(r.returncode, 0) + out = r.stdout + r.stderr + self.assertIn('alpha', out) + self.assertIn('gamma', out) + + +class StagingCoversEveryBoardForm(unittest.TestCase): + """hil_test.py declares `-b, --board` with action='append', so argparse accepts --board X, + --board=X and -bX too. Staging only the bare form sent boards to the rig with no firmware, + where every test logs `Skip (no binary)` and counts zero errors -- a green row for a board + that was never flashed. Also covers the roster check, which has to fire BEFORE the remote + tree is wiped, since hil_test.py rejects an unknown -b for the whole run.""" + + def _run(self, argv, built, roster=None, variants=None, env_extra=None, stale=None): + import json + import subprocess + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + root = Path(td.name) / 'root' + (root / 'test' / 'hil').mkdir(parents=True) + (root / 'test' / 'hil' / 'hil_test.py').touch() + (root / 'examples').mkdir(parents=True, exist_ok=True) + for b in built: + (root / 'examples' / f'cmake-build-{b}').mkdir(parents=True) + entries = [{'name': b} for b in (roster if roster is not None else built)] + for e in entries: + for v in (variants or {}).get(e['name'], []): + e.setdefault('variant', []).append({'name': v}) + cfg = root / 'test' / 'hil' / 'cfg.json' + cfg.write_text(json.dumps({'boards': entries})) + stubs = Path(td.name) / 'bin' + stubs.mkdir() + # ssh joins its argv into ONE string that the REMOTE shell re-splits, and feeds the + # heredoc on stdin. A stub that echoes "$*" hides exactly that, which is how a + # completely broken env-forwarding change once passed its own test -- so this stub + # re-splits like the real thing and reports the script body separately. + write_script(stubs / 'ssh', 'shift; printf "REMOTE-ARGV: %s\\n" "$*" >&2; ' + 'body=$(cat); printf "REMOTE-BODY: %s\\n" "$body" >&2; exit 0') + for tool in ('scp', 'rsync'): + write_script(stubs / tool, f'echo "stub-{tool} $*" >&2; exit 0') + for name, content in (stale or {}).items(): + (root / name).write_text(content) + env = {**os.environ, 'REMOTE': 'stub', 'ROOT_DIR': str(root), 'CONFIG': str(cfg), + 'PATH': str(stubs) + os.pathsep + os.environ['PATH'], **(env_extra or {})} + r = subprocess.run(['bash', str(Path(TEST_DIR).parents[0] / 'hil_ci.sh'), *argv], + capture_output=True, text=True, timeout=60, env=env) + r.rsyncs = [l for l in r.stderr.splitlines() if l.startswith('stub-rsync')] + r.run_lines = [l for l in r.stderr.splitlines() if '--retry 1' in l] + r.body = '\n'.join(l for l in r.stderr.splitlines() if l.startswith('REMOTE-BODY')) + r.stale_left = {name: (root / name).exists() for name in (stale or {})} + return r + + def test_long_board_forms_are_staged_and_only_that_board(self): + """Two boards are built so the pre-fix 'copy all built binaries' else-branch cannot + stage the right one by accident -- that is what made the first version of this test + pass against master while the feature was broken. -balpha is the glued short form + argparse resolves to --board alpha; unparsed it fell through to the all-boards branch + and silently staged everything built with no roster check.""" + for argv in (['--board', 'alpha'], ['--board=alpha'], ['-balpha']): + with self.subTest(argv=argv): + r = self._run(argv, built=['alpha', 'beta'], roster=['alpha', 'beta']) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertTrue(any('cmake-build-alpha ' in l for l in r.rsyncs), + f'{argv} never staged: {r.rsyncs}') + self.assertFalse(any('cmake-build-beta ' in l for l in r.rsyncs), + f'{argv} staged an unrequested board: {r.rsyncs}') + + def test_board_test_flag_is_not_mistaken_for_a_board(self): + """-bt is hil_test.py's --board-test and is exactly what <config>.failed contains, so + a glued -b?* pattern turns the documented retry into 'not in the roster: t'.""" + for argv in (['-b', 'alpha', '-bt', 'alpha:device/cdc_msc'], + ['-b', 'alpha', '-btalpha:device/cdc_msc']): + with self.subTest(argv=argv): + r = self._run(argv, built=['alpha']) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertNotIn('not in', r.stderr) + self.assertTrue(any('alpha:device/cdc_msc' in l for l in r.run_lines), + f'-bt never reached the rig: {r.run_lines}') + + def test_a_board_outside_the_roster_is_refused_before_staging(self): + r = self._run(['-b', 'alpha', '-b', 'ghost'], built=['alpha', 'ghost'], roster=['alpha']) + self.assertNotEqual(r.returncode, 0) + self.assertIn('ghost', r.stdout + r.stderr) + self.assertEqual(r.rsyncs, [], 'staged despite an unknown board') + self.assertEqual(r.run_lines, [], 'reached the run despite an unknown board') + + def test_a_variant_with_no_build_dir_warns_instead_of_passing_silently(self): + r = self._run(['-b', 'alpha'], built=['alpha'], variants={'alpha': ['alpha-DMA']}) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertIn('alpha-DMA', r.stderr) + self.assertIn('skipped, not tested', r.stderr) + + def test_no_build_dirs_at_all_aborts_the_all_boards_form(self): + """`hil_ci.sh` with no -b stages everything built. With nothing built it used to wipe + the rig, stage nothing, and return a green all-skip table.""" + r = self._run([], built=[], roster=['alpha']) + self.assertNotEqual(r.returncode, 0) + self.assertIn('nothing to test', r.stdout + r.stderr) + self.assertEqual(r.run_lines, [], 'reached the run with nothing staged') + self.assertNotIn('Setting up remote', r.stdout + r.stderr, + 'the guard fired only after the remote tree was already wiped') + + def test_hil_env_reaches_the_rig_as_environment_not_argv(self): + """An authorized force is HIL_NO_BOARD_LOCK=1. Passed through ssh's argv it arrives as a + positional argument and argparse exits 2, so it has to travel in the script body.""" + r = self._run(['-b', 'alpha'], built=['alpha'], env_extra={'HIL_NO_BOARD_LOCK': '1'}) + self.assertEqual(r.returncode, 0, r.stderr) + # one %q-quoted word of `export NAME=value; ` fragments, evaluated by the remote — + # NOT a bare NAME=value element, which hil_test.py's argparse takes as a positional. + # %q backslash-escapes the spaces, so match the pieces rather than the plain phrase. + run = '\n'.join(r.run_lines) + self.assertIn('HIL_NO_BOARD_LOCK=1', run) + self.assertIn('export', run) + self.assertFalse(any(' HIL_NO_BOARD_LOCK=1 ' in l for l in r.run_lines), + 'env reached argv unquoted, where hil_test.py sees a positional') + + def test_a_value_with_spaces_survives_forwarding(self): + r = self._run(['-b', 'alpha'], built=['alpha'], + env_extra={'HIL_SCRATCH': '/tmp/my scratch'}) + self.assertEqual(r.returncode, 0, r.stderr) + run = '\n'.join(r.run_lines) + self.assertIn('HIL_SCRATCH', run) + self.assertIn('scratch', run) + + def test_hil_report_dir_is_never_forwarded(self): + """Where the report lands on the rig is this script's contract (REMOTE_DIR, where all + three copy-backs look); forwarding a local HIL_REPORT_DIR relocates it there and every + copy-back comes home empty -- two of the three silently.""" + r = self._run(['-b', 'alpha'], built=['alpha'], + env_extra={'HIL_REPORT_DIR': '/tmp/elsewhere'}) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertFalse(any('HIL_REPORT_DIR' in l for l in r.run_lines), + f'HIL_REPORT_DIR reached the rig: {r.run_lines}') + + def test_a_stale_local_failed_spec_does_not_survive_a_green_run(self): + """A green run writes no .failed on the rig, so the copy-back scp no-ops; the local + spec from a previous FAILED run must not survive it looking current -- a later + "retry from the spec" would re-flash boards that already passed.""" + r = self._run(['-b', 'alpha'], built=['alpha'], + stale={'cfg.json.failed': '--accumulate -b alpha'}) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertFalse(r.stale_left['cfg.json.failed'], + "last run's re-run spec survived a green run") + + +class SummaryFoldsReportToBoards(unittest.TestCase): + """hil_summary.py replaces the agent retyping the markdown table. Report rows are named per + VARIANT and a variant need not start with the board name, so the config is what maps them + back -- the previous string-matching design produced a defect in each of four review rounds.""" + + def _sum(self, boards, rows, cfg_boards=None, banner=''): + import json + import subprocess + td = TemporaryDirectory() + self.addCleanup(td.cleanup) + d = Path(td.name) + (d / 'hil_report.json').write_text(json.dumps( + {'rows': [{'board': b, 'cells': c, 'duration': '1s'} for b, c in rows], + 'banner': banner})) + cfg = d / 'cfg.json' + cfg.write_text(json.dumps({'boards': cfg_boards or [{'name': b} for b in boards]})) + args = [a for b in boards for a in ('-b', b)] + r = subprocess.run(['python3', str(Path(TEST_DIR).parents[0] / 'helper' / 'hil_summary.py'), + str(cfg), *args, '--report-dir', str(d)], + capture_output=True, text=True, timeout=60) + self.assertEqual(r.returncode, 0, r.stderr) + return json.loads(r.stdout)['results'] + + def test_variant_rows_fold_onto_their_board(self): + """nanoch32v203 never produces a row named after the board.""" + got = self._sum(['nanoch32v203'], + [('nanoch32v203-fsdev', {'usbtest': 'pass'}), + ('nanoch32v203-usbfs', {'usbtest': 'pass'})], + cfg_boards=[{'name': 'nanoch32v203', + 'variant': [{'name': 'nanoch32v203-fsdev'}, + {'name': 'nanoch32v203-usbfs'}]}]) + self.assertEqual([r['board'] for r in got], ['nanoch32v203']) + self.assertTrue(got[0]['pass']) + self.assertTrue(got[0]['ran']) + + def test_one_failing_variant_fails_the_board(self): + got = self._sum(['nano'], + [('nano-a', {'usbtest': 'pass'}), ('nano-b', {'usbtest': '❌ 29/30'})], + cfg_boards=[{'name': 'nano', 'variant': [{'name': 'nano-a'}, + {'name': 'nano-b'}]}]) + self.assertFalse(got[0]['pass']) + self.assertIn('29/30', got[0]['detail']) + + def test_lock_contention_is_a_field_not_a_prefix(self): + got = self._sum(['alpha'], [('alpha', {'board-locked': 'fail'})]) + self.assertTrue(got[0]['locked']) + self.assertFalse(got[0]['pass']) + + def test_a_board_with_no_row_is_marked_not_run(self): + got = self._sum(['alpha', 'beta'], [('alpha', {'usbtest': 'pass'})]) + self.assertTrue(got[0]['ran']) + self.assertFalse(got[1]['ran']) + self.assertFalse(got[1]['pass']) + + def test_a_metric_cell_counts_by_its_icon(self): + got = self._sum(['a', 'b'], [('a', {'cdc_msc_throughput': '✅ C 1.2 M 3.4'}), + ('b', {'cdc_msc_throughput': '❌ C 0.0 M 0.0'})]) + self.assertTrue(got[0]['pass']) + self.assertFalse(got[1]['pass']) + + def test_skipped_cells_do_not_fail_a_board(self): + got = self._sum(['a'], [('a', {'usbtest': 'skip', 'cdc_msc': 'pass'})]) + self.assertTrue(got[0]['pass']) + + def test_a_plain_metric_cell_is_a_pass(self): + """Mirrors hil_test.py's own tally (cell_kind): failures are ALWAYS marked -- 'fail' + or a ❌ prefix, per TestFail's docstring -- while a passing test may return a plain + metric string that lands in the cell unprefixed. Treating unknown shapes as fail + would publish a green table as a red verdict.""" + got = self._sum(['a'], [('a', {'device_speed': '480.0 MBps'})]) + self.assertTrue(got[0]['pass']) + + def test_a_declared_variant_of_another_board_is_not_stolen(self): + """A declared variant need not start with its own board's name, so it may start with + a DIFFERENT board's name plus '-'. The prefix fallback must not attribute it twice.""" + got = self._sum(['alpha', 'beta'], + [('beta-x', {'usbtest': 'fail'})], + cfg_boards=[{'name': 'alpha', 'variant': [{'name': 'beta-x'}]}, + {'name': 'beta'}]) + self.assertTrue(got[0]['ran']) + self.assertFalse(got[0]['pass']) + self.assertFalse(got[1]['ran'], "beta must not inherit alpha's row") + + class CaveatSurvivesAccumulate(unittest.TestCase): """CI reruns with --accumulate: the sidecar keeps every earlier attempt's cells, but the banner was recomputed per attempt. A first attempt on a degraded rig and a clean rerun |
