| Age | Commit message (Collapse) | Author |
|---|
|
hil_report.json and hil_report.md were written independently. Four writers
produced the markdown and three wrote no JSON at all -- and those three are the
paths where a run died, so a JSON consumer saw nothing exactly when it mattered:
the per-board verdicts an agent hands back reported the whole fleet as "no
report row" while a human read the real story from the markdown.
Every writer now goes through render_report(), so a table can never contain
something the JSON does not. The document gains `scope` (a three-board PR run
and a full run that lost 24 boards were indistinguishable) and `caveat` (how the
run ended). `banner` carries rig health across an --accumulate retry; `caveat`
records how a run ended and must not -- conflating them made a clean retry
publish an abandonment that never happened.
helper/hil_report.py owns the document end to end, dissolving the import cycle
that forced write_timeout_report to compose its own markdown and removing a
duplicate cell classifier kept in sync by hand. hil_summary.py is deleted; its
CLI moves there. hil_ci.sh uploads the sidecar so a remote --accumulate has a
merge bas
|
|
hil-validate ran one hil-operator per board. That parallelizes at the wrong layer:
hil_test.py already schedules boards across host controllers and budgets concurrent
flashes and usbtest batteries per controller (FLASH_PARALLEL/USBTEST_PARALLEL), and
those permits live in one process - N parallel runs multiply the budget onto the same
uPD720201 cards for no wall-clock gain over one run that already parallelizes. The
workflow now spawns ONE operator with every board as repeated -b.
The operator no longer retypes the report table. Four consecutive max-effort review
rounds found ~15 defects in this file and every one was in reconstructing board
identity from transcribed prose: report rows are named per VARIANT (nanoch32v203 only
ever produces -fsdev/-usbfs rows), a variant need not start with its board's name,
lock contention is a `board-locked` cell rather than a phrase, and each fix introduced
the next round's bug - including a fake-green test that asserted an invariant with the
one input shape that could not break it. The new helper test/hil/helper/hil_summary.py
does the join where the roster lives and emits one machine verdict per board
({board, ran, pass, locked, detail}); the operator returns that JSON verbatim plus
`wedged`, the only field it authors, and the workflow reads fields, never parses a
string. Its cell classifier mirrors hil_test.py's own tally exactly: failures are
always marked ('fail' or a ❌ prefix, TestFail's contract), everything unmarked is a
pass - a passing test may return a plain metric cell like '13443 KB/s', and the
mirrored rule is what keeps a green table from becoming a red verdict.
hil_ci.sh kept only the LAST -b, so multi-board remote runs staged one board's
binaries and every other board died on the rig after its lock and flash slot were
spent. It now parses every -b spelling argparse accepts (with the -bt arms ordered
first, longest-match, so the <config>.failed retry form is never read as a board named
"t..."), pre-flights roster membership and build dirs for ALL boards before anything
is wiped or staged, warns per declared variant with no build dir (which hil_test.py
would silently green-skip), forwards HIL_* knobs as export lines in one %q word the
remote evals ('; '-joined so it round-trips under dash - an authorized
HIL_NO_BOARD_LOCK force must not silently no-op), keeps HIL_REPORT_DIR local because
the copy-backs look in REMOTE_DIR, and copies hil_report.json and the .failed re-run
spec back beside the markdown, deleting stale local copies first so a green run cannot
leave last run's spec looking current.
Retries preserve the fleet: the documented path is the <config>.failed spec, which
already begins with --accumulate; a fresh scoped re-run would unlink the report and
collapse the whole-fleet table to the retried boards alone.
The risky logic is executable, not argued about: .claude/workflows/test-hil-validate.mjs
pins the lookup/verdict helpers and runs in pre-commit (hil-validate-logic); nine
staging tests drive hil_ci.sh through an ssh stub that models the real thing (argv
joined into one string the remote re-splits, heredoc on stdin - the naive echo-stub
passed while the feature was broken); and deliberate mutations of the verdict logic
are all caught.
Validated on the rig: a 2-board run (usbtest 30/30 on both; the pre-fix classifier,
replayed against that run's real report, fails the fully-green stm32f723disco on its
two passing '13443 KB/s' cells), the .failed --accumulate retry (merged report kept
every earlier row), and a 10-run soak over random subsets of a 22-board pool - 43
board-slots, every failure signature matched pre-existing CI state or known flake,
zero tooling failures, no locks left behind.
|
|
Two things the rig taught us that the old guidance got wrong.
A usbfs ioctl wedged in D state cannot be freed on a running kernel. It holds
the device lock, so usb_disconnect() blocks behind it; reboot(2) walks
device_shutdown() and takes the same lock, so every userspace reboot stalls too.
Only sysrq b (emergency_restart, which skips device_shutdown) or hypervisor
action clears it -- all cited to the kernel source. The recovery ladder is
generic across rigs now (ci.lan, hifiphile, a bench PC) and ends at hypervisor
escalation only where host access exists. Two claims are corrected outright:
JLinkExe is NOT convoy-safe, and a park-flash cannot free a device-lock owner.
The hil skill's banner list is what an operator agent matches a report against,
so it enumerates the banners that actually exist, including the D-state note --
which is explicitly NOT a wedge, since a healthy in-flight testusb is
uninterruptible for most of every case and a concurrent CI battery would
otherwise turn a clean run red.
|
|
Rename usb-target-debug -> target-debug, usb-debug -> usb-kernel-debug,
usb-recover -> usb-kernel-recover (script filenames unchanged), and make all
debug skills/agents decide tool applicability by which end of the link runs
Linux: TinyUSB may run the device or host stack, and its peer may be a Linux
PC, another TinyUSB board, or a Linux gadget (e.g. Raspberry Pi UDC).
- usbmon: exists only when a Linux PC is the link's host
- usb-kernel-debug: either Linux end; allowlist gains dwc3/libcomposite/udc_core
for the gadget side of a Linux peer
- usb-sniffer: the only full-visibility capture when TinyUSB is the host
- target-debug: covers dcd_* and hcd_*/tuh_ debugging; channel choice by topology
- update target-debugger/hil-operator agents, pre-pr, hil-validate.js, and the
USB_RECOVER path constant in test/hil/usbtest.py
- CLAUDE.md: fold the dcd/hcd datasheet cross-check rule into the read-doc line
|
|
Confirmed by a 10-finder / 28-verifier adversarial review pass:
board_lock.py — the flock is now the sole authority: drop cmd_hold's
pid-liveness pre-gate (a live hil_test.py pool worker's stale record no
longer blocks a genuinely free board); cmd_release probes the flock and
only signals a verified holder, refuses to kill hil_test.py holders
(CI mid-test), handles PermissionError; the holder daemon truncates its
lock records on SIGTERM and keeps the success pipe clear of fds 0-2
(closed-stdio hold used to leave an orphan holder while reporting
failure); --config default resolves beside the script.
hil_test.py — truncate the lock record on per-board release (pool
workers outlive their flocks); warn instead of silently failing open
when the lock dir is unusable; error out on -b names absent from the
config (was a silent zero-test exit 0, readable as a green HIL run);
drop an emptied board row in accumulate_report (variant boards left a
blank ghost row).
workflows — remove the stray positional arg that made the validate size
stage exit 2 on every run; wrap JSON.parse(args) in all six scripts;
factor pr-babysit's drifted reply recipe into postReplyRecipe and dedup
refutation replies across cycles; validate args.pr and maxCycles;
driver-review rejects an empty dimensions list; hil-validate drops a
dead guard clause and retries diagnostics with -v -r 1.
agents/docs — port-dev scopes git clang-format to its own files
(concurrent workers reformatted each other in shared checkouts);
hil-operator/hil skill wording matches actual fail-fast output; the
implementation plan is now a DO-NOT-EXECUTE historical record (banner +
checked boxes) so plan-executing agents cannot revert shipped files.
Verified: lock storm 1-winner-in-10, stale-record hold, closed-stdio
hold, dead-pid cleanup, CI-holder refusal, ghost-row 4-scenario merge,
unknown-board exit 1, py_compile + check.sh on all six workflows,
pre-commit clean.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upj4hta5TNoAbidqeC1zZ6
|
|
Add worker agents (builder, port-dev, driver-reviewer, hil-operator,
pr-monitor), deterministic workflows (validate, fanout-dev, driver-review,
hil-validate, full-check, pr-babysit) and a /pre-pr gate skill, so sessions
can fan build/test/review/PR-triage work out to tiered subagents. pr-babysit
drives a PR to green: triage CI + bot reviews, fix validated findings, verify,
push, and reply-to + resolve each inline review thread (fixed or refuted).
Replace the stop-the-runner HIL discipline with per-board flock locks:
test/hil/board_lock.py plus a fail-open guard in hil_test.py let CI and dev
sessions share the rig per board (locked boards fail fast and re-run;
HIL_NO_BOARD_LOCK=1 is a user-authorized bypass). The actions-runner is
never stopped.
Design spec, implementation plan, and real-rig smoke evidence under
docs/superpowers/.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Rn1AN5DsTdFhRwhugfgKZi
|
