| Age | Commit message (Collapse) | Author |
|---|
|
|
|
The modules were 21% comment, much of it review-cycle argument rather than
guidance -- _kill_kids stated 'descendant by construction, no argv check needed'
twice, eight lines apart. Deleting such comments outright makes maintenance
worse: the next reader simplifies the thing the comment was defending. So the
history moves to the 2026-07-30 fleet-wedge design doc, which gains a trim
addendum recording what was removed, what was deliberately kept, and the rule
that decided each -- the CI ceiling bounds how long a run burns, and does nothing
about state that outlives it.
One comment was not merely long but WRONG: the report wipe carried 'The unlink is
DEFERRED to inside the pool try/except below', which is the opposite of what the
code does -- it sits before Manager() with its own comment explaining why. That
is the failure mode this pass is about, so it is deleted rather than reworded.
Kept everywhere: citations that refute a plausible wrong reading. That
usb_lock_device_interruptible is why the readers are killable, that usblp_mutex
is driver-global, that rawmidi honours O_NONBLOCK where usblp does not.
Two follow-ups are retired with them: pr3803-hil-blindness-reporting.md (there
is no blindness to report any more) and pr3803-usbtest-recovery-reserve.md (the
reserve is derived now). Kept: pr3803-flasher-recover.md, which PR #3832
implements, plus pr3803-pci-rebind-stranding.md and pr3803-hil-iar-rerun-spec.md,
both independent of this work.
|
|
Two layers whose cost was a contract to reason about rather than an outcome.
SYSFS_UNKNOWN was a three-valued return five files had to keep apart, and
misreading unknown as absence was silent: a healthy board reported as a firmware
regression. What it guarded is real -- `serial` is served by usb_string_attr,
which takes usb_lock_device_interruptible (v6.12.96 sysfs.c:141-143), the same
lock a wedged usbfs ioctl holds -- so the BOUND stays, on every caller by
default. usb_scan reads `serial` on every device matching the VID, and hil_lock's
controller_of does that on essentially every board, so one wedged DUT would
otherwise stall every worker, not one. What goes is the third value.
read_sysfs now returns str or None, and the question the third value existed to
answer is asked directly instead, by two predicates that say which question they
answer: sysfs_stranded() is process-wide and sticky, for hil_pool_check's footer
("could anything here be the tool losing sight of healthy hardware?"), and
path_stranded(path) is per-device, which is what usbtest needs to tell a DUT
whose `serial` is held under device_lock from one that genuinely left the bus --
that difference decides whether it performs driver-registry writes that take the
uninterruptible device_lock.
Gone: _SysfsUnknown, SYSFS_UNKNOWN, sysfs_blind, sysfs_blind_note,
note_sysfs_strand, the cross-process blindness publishing and its report banner,
usb_scan's (list, bool) return, usbtest's inconclusive abort, _blind_note's slot
in the result tuple, and bounded_open, whose last caller went in the previous
commit.
The strand memo is rewritten around the one invariant that makes it safe to
reuse: it is keyed by the path's kernfs inode, captured BEFORE the read. A
busport does not change when a board returns to the same physical port, so a
path-only blacklist outlives the wedge and hil_pool_check's own recovery flow --
reset, reflash, wait_device polling that busport -- would never look at the
board again. A re-enumeration destroys the kernfs node and makes a new one, so a
changed inode is the all-clear. Two ceilings bound different things: per path
(_PATH_STRAND_MAX) for a board that flaps while still wedged, and per process
(_STRAND_MAX) as a backstop against RLIMIT_NOFILE, counted per PATH rather than
per reader because hil_pool_check runs four poll threads over one bus.
A board the pool guard never reached is now reported as run-aborted rather than
pool-timed-out, and outranks a stale board-locked cell for the same reason the
pool-timeout cell does.
Both predicates answer conservatively where they are consulted before something
irreversible. path_stranded() covers the paths read_sysfs answered None for
WITHOUT reading -- past _STRAND_MAX it declines to start another reader, and
vouching for a path nobody looked at hands usbtest's fail-CLOSED guard a
fabricated all-clear, running remove_id/unbind against a wedged device. usbtest's
startup lookup carries the same caveat hil_test's absent arm already did, because
its stderr is relayed verbatim into the report cell.
strand_note() survives the removal for the same reason master had it: every
caller that can say "not found" needs the same sentence, and the one site left to
re-invent it got missed -- a wedged-but-enumerated printer was reported as an
enumeration failure, sending a maintainer after firmware.
The two predicates are not interchangeable, and usbtest needs both. Its per-case
verdict is per-DUT -- a peer that stranded at case 2 must not make our board
report wedged at case 29 -- but the finally block's cleanup is process-wide:
remove_id plus an unbind of EVERY interface under the driver, including that
peer's, each taking the uninterruptible device_lock. So the verdict uses
path_stranded() and the global cleanup stays gated on sysfs_stranded().
USBTEST_RECOVERY_BUDGET was an independent 250s that could not actually contain
the ladder it reserved for, which is why usbtest.py carried a _time_left() gate
re-deciding before every step -- with a bare '- 35' for costs paid downstream
that nobody could re-derive. Between them the two produced a recovery that
skipped its own steps for most real hangs.
The reserve is now derived from the bounds usbtest itself declares, per flasher
and per target: a probe reset, a reflash, and the Rescue-DP POR plus retry a
wedged RP DAP needs, plus the settles and hil_util.REAP_GRACE for each bounded
step. The Rescue-DP legs are openocd-only and gated on the RP target cfg, and
a stub reset is screened out, so the reserve tracks each board's real ladder
instead of one fleet number: 390s for the two RP boards -- whose ladder the old
250 could not contain, which is exactly why the gates skipped their steps --
190s for the other seventeen probe-reset boards, and 150s for esptool and
lm4flash, whose reset is a no-op. Changing a bound in usbtest moves the
reserve with it, and a unit test asserts it covers the ladder.
With the room actually reserved, the child runs the ladder straight through:
recovery_steps, _time_left, the three per-step gates and the parsed-but-unused
--outer-timeout are gone. What stays is what decides outcomes -- the convoy_safe
gate, reset-before-reflash, the no_op screen so a stub that resets nothing is not
claimed, and wedged_pids() as the arbiter, because a clean flash only proves the
probe wrote the MCU.
hil_util.py 616 -> 514 lines.
|
|
support (#3853)
Promote SEGGER RTT from an inline debugging technique to a standalone skill
backed by one stdlib-only implementation in tools/rtt.py: a CLI and importable
module for console/capture over J-Link (RTTTelnetPort) and OpenOCD (rtt server)
probes, with probe selection by serial or VID:PID, control-block address via
--elf or --addr, bidirectional console, post-mortem ring dump, and
--reset-before-attach for boot-time capture. The HIL harness reads a board's
console over RTT when its probe has no VCOM ("logger": "rtt" plus a LOGGER=rtt
variant define), covering device_info, pool-check aliveness, and CI wiring.
Validated on 22 boards across both backends; 26 unit tests run in pre-commit.
|
|
hil_report.json and hil_report.md were written independently. Four writers
produced the markdown and three wrote no JSON at all -- and those three are the
paths where a run died, so a JSON consumer saw nothing exactly when it mattered:
the per-board verdicts an agent hands back reported the whole fleet as "no
report row" while a human read the real story from the markdown.
Every writer now goes through render_report(), so a table can never contain
something the JSON does not. The document gains `scope` (a three-board PR run
and a full run that lost 24 boards were indistinguishable) and `caveat` (how the
run ended). `banner` carries rig health across an --accumulate retry; `caveat`
records how a run ended and must not -- conflating them made a clean retry
publish an abandonment that never happened.
helper/hil_report.py owns the document end to end, dissolving the import cycle
that forced write_timeout_report to compose its own markdown and removing a
duplicate cell classifier kept in sync by hand. hil_summary.py is deleted; its
CLI moves there. hil_ci.sh uploads the sidecar so a remote --accumulate has a
merge bas
|
|
Every PR built all 74 legs (2494 example builds on GHA cmake alone) and flashed
all 30 rig boards, whatever it touched. One classifier now walks the PR diff twice
and answers three questions: which families to build, which examples per family,
and which boards run which tests. Fail-open throughout - anything no rule
classifies, any exception, any unusable output falls back to the full matrix, and
a master push always builds everything.
test/hil/helper/hil_select.py moves to tools/ci_select.py: it is no longer HIL-only,
and tools/ is where the build side can import it. test_hil_select.py follows it as
test_ci_select.py.
Rules (docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md holds the
full table): a port selects the families whose family.cmake references it, and its
role - a dcd change skips host examples and vice versa; a class selects only the
examples whose tusb_config.h enables its CFG_TU[DH]_ macro, following cross-class
includes; an example selects itself; hw/bsp selects its family or board; hw/mcu and
lib select whoever references them. CMake is the reference for all of it - make
follows whatever cmake decides, family.mk is never scanned.
Empty means empty (maintainer ruling): a rule that classifies a path to nothing
selects nothing. Ports no family references, classes no config enables, libs no
example builds and hw/mcu paths that resolve nowhere are all real - nothing
compiles them, so nothing can validate them, and the master-push build is the net.
Structural tests pin each such case with an explicit allowlist, so the day one
stops being empty it fails pre-commit instead of silently narrowing CI.
Per-example builds: build.py grows a repeatable -e, resolved against the targets
CMake actually registered and batched into one `cmake --build --target a b c`.
build_utils mirrors CMake's family_filter (the whole FAMILY_MCUS list, ${...} and
string(TOUPPER ...) resolved) for the cmake side, while the make side keeps
master's algorithm verbatim - the two build systems answer differently and a shared
answer breaks lpc54's make link. hil-build gains this even on a full selection:
1702 example builds become 515.
Transport: the selection travels as a file, never an argv or env var - a mass-sweep
diff selects 261 KB against a 128 KiB exec limit, and E2BIG would fail the step
before its own fallback could run. CircleCI carries the example map inside the
generated config (pipeline parameters cap at 512 chars), swapped into the parameter
defaults by sentinel match, and drops the scoping wholesale if that rewrite fails.
Every PR-derived value written to $GITHUB_ENV/$GITHUB_OUTPUT is character-screened.
Code metrics follow the scoping: metrics.py emits per-example totals, and
metrics_pair_compare compares the (board, example) pairs present on both sides
instead of a scoped run against a full-matrix average.
The selector's own suite gates it in both providers: a selector that exits 0 with
valid-but-wrong JSON is the one failure fail-open cannot catch, so a red suite
means the full matrix.
|
|
The doc carried its own rename instruction for when the branch gained a number;
the branch is PR #3836.
|
|
Four writers produce hil_report.md and three of them write no JSON - the no-boards
exit, the pool-guard fallback and _abandon_exit's text prepend. Those are exactly the
runs that failed, so hil_summary.py, which builds an agent's per-board verdicts from
the sidecar, sees nothing while a human reading the markdown sees the real story. The
scope note is markdown-only too, so a three-board PR run and a full run that lost 24
boards are indistinguishable in JSON.
Five tasks: put scope in the sidecar, render the markdown from the document, give the
two early-exit paths a document, make _abandon_exit set a caveat field instead of
prepending to a file it did not write, then pin the invariant that re-rendering the
JSON reproduces the markdown byte for byte. Split out because it is a hil_test.py
reporting refactor, and the abandon path runs while the interpreter is being torn
down - it deserves its own review.
|
|
Records the convention in CLAUDE.md -- deferred work is a SEPARATE scope that deserves
its own PR, written by another session, so it is handed off as a writing-plans doc in
docs/superpowers/followup/pr<NNN>-<topic>.md rather than accumulated in the PR that
found it.
Five handoffs from #3803: flasher_recover (convoy-safe recovery for J-Link boards, seven
validated on the rig), the blindness reporting gaps, the usbtest recovery reserve, the
IAR re-run spec, and the pci-rebind stranding question. Each carries what is already
established with its citations and measurements, what remains, and why it was split out.
One doc per follow-up, not one per PR: a per-PR file invites unrelated work into the
same document and rots as a unit.
|
