| Age | Commit message (Collapse) | Author |
|---|
|
head SHA
A cycle running before the bots posted saw zero findings and reported done;
with a fast-green CI the babysit loop could exit unreviewed. done now needs
every reviewer settled for the current head: Copilot's verdict review
(commit_id), Codex's verdict comment (Reviewed-commit line), its thumbs-up
reaction on the PR body, or the named claude-review check run — with quota/
error notices and the reaction freshness-gated on push time (check-suite
creation, not committer date) and every lookup paginated. pr-babysit re-arms
with backoff on a pending reviewer instead of exiting unactionable, skipping
the pointless final-cycle wait.
|
|
port-dev/driver-reviewer to code-writer/code-verifier; pin model+effort on every agent
|
|
hil_report.json and hil_report.md were written independently. Four writers
produced the markdown and three wrote no JSON at all -- and those three are the
paths where a run died, so a JSON consumer saw nothing exactly when it mattered:
the per-board verdicts an agent hands back reported the whole fleet as "no
report row" while a human read the real story from the markdown.
Every writer now goes through render_report(), so a table can never contain
something the JSON does not. The document gains `scope` (a three-board PR run
and a full run that lost 24 boards were indistinguishable) and `caveat` (how the
run ended). `banner` carries rig health across an --accumulate retry; `caveat`
records how a run ended and must not -- conflating them made a clean retry
publish an abandonment that never happened.
helper/hil_report.py owns the document end to end, dissolving the import cycle
that forced write_timeout_report to compose its own markdown and removing a
duplicate cell classifier kept in sync by hand. hil_summary.py is deleted; its
CLI moves there. hil_ci.sh uploads the sidecar so a remote --accumulate has a
merge bas
|
|
~/code/pico/pico-sdk and $HOME/code/esp-idf/export.sh are the ci rig's private layout;
written into instructions they silently stop being true on tusb, a dev PC, or any
future rig. The docs now use the variables the vendors define -- PICO_SDK_PATH for the
Pico SDK and IDF_PATH for ESP-IDF, activated explicitly as `. "$IDF_PATH/export.sh"` --
and leave where the checkouts live to each host's profile.
The variables are only useful if the shells that agents actually get can see them, and
`ssh <rig> 'cmd'` is non-interactive AND non-login: it reads no profile, and Debian's
sshd-sourced ~/.bashrc returns at the interactive guard before most of the file. The ci
rig already keeps its exports in the section ABOVE that early-return; IDF_PATH now sits
there beside PICO_SDK_PATH, and the whole chain is verified from a plain non-interactive
ssh: both variables visible, `. "$IDF_PATH/export.sh"` activates ESP-IDF v5.5.3 with
idf.py on PATH -- no login shell, no alias, no hard-coded path. hil-pool-check documents
that placement so the next rig is set up the same way.
|
|
hil-validate ran one hil-operator per board. That parallelizes at the wrong layer:
hil_test.py already schedules boards across host controllers and budgets concurrent
flashes and usbtest batteries per controller (FLASH_PARALLEL/USBTEST_PARALLEL), and
those permits live in one process - N parallel runs multiply the budget onto the same
uPD720201 cards for no wall-clock gain over one run that already parallelizes. The
workflow now spawns ONE operator with every board as repeated -b.
The operator no longer retypes the report table. Four consecutive max-effort review
rounds found ~15 defects in this file and every one was in reconstructing board
identity from transcribed prose: report rows are named per VARIANT (nanoch32v203 only
ever produces -fsdev/-usbfs rows), a variant need not start with its board's name,
lock contention is a `board-locked` cell rather than a phrase, and each fix introduced
the next round's bug - including a fake-green test that asserted an invariant with the
one input shape that could not break it. The new helper test/hil/helper/hil_summary.py
does the join where the roster lives and emits one machine verdict per board
({board, ran, pass, locked, detail}); the operator returns that JSON verbatim plus
`wedged`, the only field it authors, and the workflow reads fields, never parses a
string. Its cell classifier mirrors hil_test.py's own tally exactly: failures are
always marked ('fail' or a ❌ prefix, TestFail's contract), everything unmarked is a
pass - a passing test may return a plain metric cell like '13443 KB/s', and the
mirrored rule is what keeps a green table from becoming a red verdict.
hil_ci.sh kept only the LAST -b, so multi-board remote runs staged one board's
binaries and every other board died on the rig after its lock and flash slot were
spent. It now parses every -b spelling argparse accepts (with the -bt arms ordered
first, longest-match, so the <config>.failed retry form is never read as a board named
"t..."), pre-flights roster membership and build dirs for ALL boards before anything
is wiped or staged, warns per declared variant with no build dir (which hil_test.py
would silently green-skip), forwards HIL_* knobs as export lines in one %q word the
remote evals ('; '-joined so it round-trips under dash - an authorized
HIL_NO_BOARD_LOCK force must not silently no-op), keeps HIL_REPORT_DIR local because
the copy-backs look in REMOTE_DIR, and copies hil_report.json and the .failed re-run
spec back beside the markdown, deleting stale local copies first so a green run cannot
leave last run's spec looking current.
Retries preserve the fleet: the documented path is the <config>.failed spec, which
already begins with --accumulate; a fresh scoped re-run would unlink the report and
collapse the whole-fleet table to the retried boards alone.
The risky logic is executable, not argued about: .claude/workflows/test-hil-validate.mjs
pins the lookup/verdict helpers and runs in pre-commit (hil-validate-logic); nine
staging tests drive hil_ci.sh through an ssh stub that models the real thing (argv
joined into one string the remote re-splits, heredoc on stdin - the naive echo-stub
passed while the feature was broken); and deliberate mutations of the verdict logic
are all caught.
Validated on the rig: a 2-board run (usbtest 30/30 on both; the pre-fix classifier,
replayed against that run's real report, fails the fully-green stm32f723disco on its
two passing '13443 KB/s' cells), the .failed --accumulate retry (merged report kept
every earlier row), and a 10-run soak over random subsets of a 22-board pool - 43
board-slots, every failure signature matched pre-existing CI state or known flake,
zero tooling failures, no locks left behind.
|
|
Finding documents by walking the library tree misses anything the filename
does not carry - Calibre stores only a truncated title and the author there, so
the tags, series, publisher and description that hold most part numbers and
errata IDs are invisible to it. A zero-result tree search then reads as "the
document does not exist" rather than as a bad search; that happened here, and
led to a confident claim that a fully populated 14,000-file library was empty.
search.py queries metadata.db, ANDs its keywords across every metadata field
(including the stored filename), and prints the best matches first with the
exact path to read. Matching is NFKC + casefold, so a typed ASCII apostrophe or
mu reaches the titles that store the typographic ones.
Every printed path is checked on disk. Calibre renames <author>/<title> (<id>)
when metadata is edited and leaves the old directory behind, so a miss retries
by the stable book id before reporting MISSING - which distinguishes "the file
is not here right now" from "no such document". The gate tests for metadata.db
rather than the directory, since an unmounted or half-synced mountpoint is
still a directory.
Consumers that prescribed their own tree search - driver-reviewer, port-dev,
the driver-review workflow, and the calibre-library references in CLAUDE.md,
usbtest, etm-trace and target-debug - now point at the skill, which owns the
library's location.
|
|
Two things the rig taught us that the old guidance got wrong.
A usbfs ioctl wedged in D state cannot be freed on a running kernel. It holds
the device lock, so usb_disconnect() blocks behind it; reboot(2) walks
device_shutdown() and takes the same lock, so every userspace reboot stalls too.
Only sysrq b (emergency_restart, which skips device_shutdown) or hypervisor
action clears it -- all cited to the kernel source. The recovery ladder is
generic across rigs now (ci.lan, hifiphile, a bench PC) and ends at hypervisor
escalation only where host access exists. Two claims are corrected outright:
JLinkExe is NOT convoy-safe, and a park-flash cannot free a device-lock owner.
The hil skill's banner list is what an operator agent matches a report against,
so it enumerates the banners that actually exist, including the D-state note --
which is explicitly NOT a wedge, since a healthy in-flight testusb is
uninterruptible for most of every case and a concurrent CI battery would
otherwise turn a clean run red.
|
|
probes (#3794)
test/hil: add board-pool health check, split hil_test into focused modules (#3794)
Add test/hil/hil_pool_check.py: per-board rig health scan — probe presence,
light-example flash (dfu_runtime; device_info + serial check for host-only
boards), uid re-enumeration, safe recovery (probe authorized-toggle, board
reset), verified board_test re-park, USB topology report, and a markdown
summary table. Missing firmware is built on the spot (tools/build.py, idf.py
for espressif, one get_deps retry); row statuses: ok, flash-failed, failed,
locked. Board locks are always respected, never bypassed.
Refactor hil_test.py into hil_lock.py (flock protocol, controller permits,
hold/release/status CLI; replaces board_lock.py) and hil_flash.py (flashers,
find_firmware, run_cmd). Update WCH probe uids and the board roster in
tinyusb.json; add the hil-pool-check skill.
|
|
Instruction-level trace outranks PC-sampling when samples cannot resolve a
mechanism, but the J-Trace is exclusive per-board hardware: the agent uses
it only when its prompt says the board is trace-wired or the user asked,
and otherwise proposes it in notes - mirroring the lock-force consent rule.
|
|
by board family
- esp-target-debug: de-narrated (~1000 -> 847 words) — session-diary phrasing
('this unit', dates, 'we measured') replaced with durable reference facts;
restructured per-board (P4 vs S3 notes); new Scripted-session gotchas
section consolidates telnet-halt pattern, ESP_ONLYCPU, and ROM-frame
guidance; UART-reset recipe stated once
- target-debug: fix run-on seam from the -singlerun insertion
- target-debugger agent: charter now resolves the board family FIRST and
routes Espressif boards to esp-target-debug as primary playbook; skills
table re-aligned
- retrieval regression: 4/4 (agent routing, S3 keep-alive quirk, OpenOCD RTT
on ST-Link, ROM-frame guidance)
|
|
rig-verified
P4 (hand-wired USB-SJ breakout, GPIO24/25 from header J1):
- COEXISTENCE verified: 303a:1001 + cafe:4008 enumerated simultaneously;
gdb attach/halt/bt during live CDC traffic; symbolized app backtrace
(tud_task_ext <- usb_device_task <- vPortTaskWrapper)
- set ESP_RTOS FreeRTOS before board cfg -> full dual-core task list;
without it, bare 'Remote target'
- attach-may-reset nuance flagged (post-mortem autopsy caution)
- console = UART0 (CP2102 flasher tty) on stock builds; D+/D- swap symptom
documented (low-speed + error -71 vs full-speed)
S3 (same-port PHY swap):
- boundary captured live in dmesg: same hub port flips 303a:1001 ->
cafe:4008 as the app boots; openocd 'esp_usb_jtag: could not find or open
device!' verbatim
- attach/halt/symbol resolution verified via board_test (usb_new_phy absent
from ELF when CFG_TUD/TUH=0); app-context keep-alive quirk (~4 s
unattended drop, -71 half-dead, UART esptool reset recovers); cpu1
OCD_ID=0 -> ESP_ONLYCPU=1; telnet-halt + gdb-read scripted pattern;
RTC_CNTL PHY-mux reference (0x60008120) + esptool read_mem/write_mem
- target-debug pointer + target-debugger agent table row
|
|
aligned columns
- target-debugger: skill list is now a table referencing skills by name only
(path pattern stated once).
- target-debug: probe-mapping run-on paragraph split into bullets; drop the
GDB Ninth-Edition caveat (calibre now holds the Tenth-Edition PDF, id 2264).
- Align markdown table columns across target-debug, usb-sniffer, usbmon, hil,
usbtest and the agent (7 tables); tables with paragraph-length cells left
unpadded (usbmon symptom map, usbtest case map).
|
|
table integration
- Vector catch + Cortex-M fault autopsy, verified with a deliberate bad-load
on stm32f407disco: CFSR=0x8200 (BFARVALID|PRECISERR), BFAR = exact bad
address, stacked pc addr2lined to the faulting line; gotchas recorded
(stale FPB comparators fire phantom SIGTRAPs — scrub first; arm DEMCR
after reset; loads precise / stores imprecise; ARMv6-M has no CFSR/BFAR)
- SWO exception trace + hw PC sampling gate PASSED on F407: 680 KB of
packets in 3 s (0x17 PC samples in flash range, 0x0E SysTick enter/exit);
JLinkSWOViewerCL decodes stimulus only — raw SWORead is the recipe;
SWOStart needs an explicit speed headless
- verifybin 'Verify successful.'; FreeRTOS -rtos plugin lists all 6
cdc_msc_freertos tasks after a run->stop cycle (plain attach = 0xDEAD
placeholder); semihosting anti-note; monitor-mode pointer (untested)
- Intrusiveness table gains the new rows; agent playbook bullet updated;
retrieval gate 5/5 with a fresh reader; executed plan committed
|
|
Rename usb-target-debug -> target-debug, usb-debug -> usb-kernel-debug,
usb-recover -> usb-kernel-recover (script filenames unchanged), and make all
debug skills/agents decide tool applicability by which end of the link runs
Linux: TinyUSB may run the device or host stack, and its peer may be a Linux
PC, another TinyUSB board, or a Linux gadget (e.g. Raspberry Pi UDC).
- usbmon: exists only when a Linux PC is the link's host
- usb-kernel-debug: either Linux end; allowlist gains dwc3/libcomposite/udc_core
for the gadget side of a Linux peer
- usb-sniffer: the only full-visibility capture when TinyUSB is the host
- target-debug: covers dcd_* and hcd_*/tuh_ debugging; channel choice by topology
- update target-debugger/hil-operator agents, pre-pr, hil-validate.js, and the
USB_RECOVER path constant in test/hil/usbtest.py
- CLAUDE.md: fold the dcd/hcd datasheet cross-check rule into the read-doc line
|
|
Opus-tier agent charter for backgrounding a long hardware debug session:
instrument -> build -> flash under one held board lock -> dual-side
capture -> correlate -> refine, strictly one instance, skills as source
of truth (usb-target-debug, usbmon, usb-debug, usb-sniffer, usb-recover,
hil). The charter encodes what dogfooding established:
- diagnosis standard: evidence must show the mechanism, or a fix must
flip the ORIGINAL failing case on hardware; stop after two
evidence-free cycles and hand back a partial diagnosis
- lock cadence: hold for the whole session, release around hil_test.py
runs (it self-locks per board)
- revert semantics: "fix stays, probe goes, re-verify clean" —
instrumentation reverted, candidate fix left uncommitted and
re-verified on a clean build, pristine firmware reflashed before
lock release
Returns a machine-parseable diagnosis report including ruledOut[] —
disproven hypotheses are deliverables. Spec roster updated (opus/xhigh,
effort requested per agent() call).
|
|
Confirmed by a 10-finder / 28-verifier adversarial review pass:
board_lock.py — the flock is now the sole authority: drop cmd_hold's
pid-liveness pre-gate (a live hil_test.py pool worker's stale record no
longer blocks a genuinely free board); cmd_release probes the flock and
only signals a verified holder, refuses to kill hil_test.py holders
(CI mid-test), handles PermissionError; the holder daemon truncates its
lock records on SIGTERM and keeps the success pipe clear of fds 0-2
(closed-stdio hold used to leave an orphan holder while reporting
failure); --config default resolves beside the script.
hil_test.py — truncate the lock record on per-board release (pool
workers outlive their flocks); warn instead of silently failing open
when the lock dir is unusable; error out on -b names absent from the
config (was a silent zero-test exit 0, readable as a green HIL run);
drop an emptied board row in accumulate_report (variant boards left a
blank ghost row).
workflows — remove the stray positional arg that made the validate size
stage exit 2 on every run; wrap JSON.parse(args) in all six scripts;
factor pr-babysit's drifted reply recipe into postReplyRecipe and dedup
refutation replies across cycles; validate args.pr and maxCycles;
driver-review rejects an empty dimensions list; hil-validate drops a
dead guard clause and retries diagnostics with -v -r 1.
agents/docs — port-dev scopes git clang-format to its own files
(concurrent workers reformatted each other in shared checkouts);
hil-operator/hil skill wording matches actual fail-fast output; the
implementation plan is now a DO-NOT-EXECUTE historical record (banner +
checked boxes) so plan-executing agents cannot revert shipped files.
Verified: lock storm 1-winner-in-10, stale-record hold, closed-stdio
hold, dead-pid cleanup, CI-holder refusal, ghost-row 4-scenario merge,
unknown-board exit 1, py_compile + check.sh on all six workflows,
pre-commit clean.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Upj4hta5TNoAbidqeC1zZ6
|
|
Review-fix batch (owner-confirmed) on the multi-agent harness:
- board_lock: detach holder stdio so a captured `hold` cannot hang on the
daemon's inherited pipe; probe locks by holder-pid liveness instead of a
momentary flock, which could spuriously fail a concurrent acquirer
(storm-tested: 1 winner in 10, 0/15 acquire failures under probe storm)
- hil_test: locked board renders a visible board-locked fail row so the
report matches the exit code; stale marker cleared on a real re-run
- pr-babysit: autoPush now opt-in (default dry run); resolve recipe
paginates reviewThreads; post-push resolve gets issue-comment fallback
- validate: size stage honors non-default base via --base-branch; pvs
stage delegated to the new agent
- new static-analyzer agent (sonnet): PVS-Studio SAST+MISRA for one
board, structured findings gated on files changed vs base
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Rn1AN5DsTdFhRwhugfgKZi
|
|
Add worker agents (builder, port-dev, driver-reviewer, hil-operator,
pr-monitor), deterministic workflows (validate, fanout-dev, driver-review,
hil-validate, full-check, pr-babysit) and a /pre-pr gate skill, so sessions
can fan build/test/review/PR-triage work out to tiered subagents. pr-babysit
drives a PR to green: triage CI + bot reviews, fix validated findings, verify,
push, and reply-to + resolve each inline review thread (fixed or refuted).
Replace the stop-the-runner HIL discipline with per-board flock locks:
test/hil/board_lock.py plus a fail-open guard in hil_test.py let CI and dev
sessions share the rig per board (locked boards fail fast and re-run;
HIL_NO_BOARD_LOCK=1 is a user-authorized bypass). The actions-runner is
never stopped.
Design spec, implementation plan, and real-rig smoke evidence under
docs/superpowers/.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Rn1AN5DsTdFhRwhugfgKZi
|
