diff options
| -rw-r--r-- | .claude/skills/usb-target-debug/SKILL.md | 198 | ||||
| -rw-r--r-- | .gitignore | 1 | ||||
| -rw-r--r-- | docs/superpowers/plans/2026-07-14-usb-target-debug-handoff.md | 125 |
3 files changed, 324 insertions, 0 deletions
diff --git a/.claude/skills/usb-target-debug/SKILL.md b/.claude/skills/usb-target-debug/SKILL.md new file mode 100644 index 000000000..c664bf8fc --- /dev/null +++ b/.claude/skills/usb-target-debug/SKILL.md @@ -0,0 +1,198 @@ +--- +name: usb-target-debug +description: Use when a TinyUSB device misbehaves on real hardware and host-side capture can't explain it — a HIL test fails but usbmon shows only Submits with no Completes, the device silently NAKs, wedges, STALLs, babbles, or drops data, EP0 starves, an ISR or DCD/HCD state bug is suspected — and you need device-side evidence: TU_LOG/RTT logs, GDB state dumps, a RAM ring-buffer event trace, or PC-sampling of where the core spins. +--- + +# usb-target-debug — device-side capture & debugging on the HIL rig + +Completes the debugging trio (the `usb-sniffer` skill adds a fourth, +wire-level view when hardware tapping is available): + +| Skill | Answers | +|---|---| +| `usbmon` | what the host actually exchanged (URBs) | +| `usb-debug` | why the host acted (dmesg / dynamic debug) | +| **`usb-target-debug`** | **what the device did** (logs, driver state, PC) | +| `usb-sniffer` | what crossed the wire (PIDs, handshakes, resets — hardware tap) | + +For enumeration/transfer bugs the default posture is **dual-side capture** — +usbmon on the host *and* a target-side channel, simultaneously — not +host-first-then-escalate. + +## Rig discipline — lock first, always + +Hold the board lock for the WHOLE manual session; never stop the +actions-runner (see the `hil` skill for the full lock protocol): + +```bash +python3 test/hil/board_lock.py hold <board> --reason "target debug: <bug>" +# ... instrument / build / flash / capture / GDB ... +python3 test/hil/board_lock.py release <board> +``` + +Board → probe mapping: `test/hil/tinyusb.json` — `flasher.name` is the probe +family, `flasher.uid` the **probe serial** (many identical probes on the rig: +J-Link needs `-SelectEmuBySN <uid>` / GDB server `-select usb=<uid>`; OpenOCD +`-c 'adapter serial <uid>'`). `JLINK_DEVICE` / `OPENOCD_OPTION` come from +`hw/bsp/<family>/boards/<board>/board.cmake` (or `board.mk`); find the family +with `ls -d hw/bsp/*/boards/<board>`. Run on the host that owns the probe — +config is `test/hil/tinyusb.json` on ci, `local.json` on htpc (`hil` skill). + +## Pick the least intrusive technique that can answer the question + +Observation can mask the bug — the ch32v307 Heisenbug changed behavior under +logging *and* under the debugger. If the bug disappears when instrumented, +that IS a finding (timing-sensitive): move down in intrusiveness, not up. + +| Technique | Intrusiveness | Reach for it when | +|---|---|---| +| PC-sampling | none — no halt, no code change | core wedged/spinning somewhere unknown (rusb2 FRDY) | +| RAM ring-buffer | ~tens of cycles per event | ISR ordering/timing bugs (musb babble) | +| TU_LOG (RTT) | µs per line | logic bugs that survive logging | +| TU_LOG (UART) | ms per line — blocking write | same, when no J-Link on the board | +| GDB halt / breakpoints | stops USB service entirely | post-mortem state autopsy once wedged | + +## TU_LOG capture + +Build with `LOG=2` (`LOG=3` adds per-transfer noise and much more timing skew). +`LOGGER=rtt` routes it over the debug probe (J-Link only) — no UART wiring: + +```bash +# RTT: JLinkGDBServer from AGENTS.md "GDB Debugging" + -RTTTelnetPort, then: +timeout 20s JLinkRTTClient > /tmp/rtt.log # non-interactive capture +# UART (board's debug serial, if wired): +stty -F /dev/ttyACM<N> 115200 raw && timeout 20s cat /dev/ttyACM<N> | tee /tmp/uart.log +``` + +An RTT-built firmware that has since wedged still holds a log tail in RAM — +but ONLY what fits the drain model: the default SEGGER mode (NO_BLOCK_SKIP) +**drops** writes once the ring fills with no reader, so an undrained target +holds the first KB after boot, not the wedge tail. There is no overwrite mode +in stock SEGGER RTT (only SKIP/TRIM/BLOCK): post-mortem RTT is evidence only +if a live drain was running — otherwise instrument with the RAM ring below. +Use `JLinkGDBServer -RTTTelnetPort 19021` + `JLinkRTTClient` for the drain +(proven; note the server briefly halts the core on connect). `JLinkRTTLogger` +fails to find the control block on some parts (LPC4088) even when it exists +and even given `-RTTAddress`; don't fight it — `nm` the ELF for `_SEGGER_RTT`, +read the aUp[0] descriptor (`mem32`), `savebin` the buffer — debug-AP RAM +reads don't halt the target. + +## GDB — state autopsy and watchpoints + +Connect/load recipes per probe family (J-Link, OpenOCD for ST-Link / +CMSIS-DAP / WCH-Link) are in AGENTS.md "GDB Debugging". Release builds keep +DWARF (`MinSizeRel`), so `p`/struct access works on HIL firmware. + +**Autopsy of a wedged board: attach and halt ONLY** — skip AGENTS.md's +`monitor reset halt` + `load` (those are for fresh starts; a reset destroys +the evidence). Symbolize with the ELF that is actually flashed — +`<build root>/cmake-build-<board>/<example>/<example>.elf` from the run that +wedged; do not rebuild while the wedge is still on the board. The debug-loop +specifics: + +```gdb +p/x _usbd_dev.ep_status # usbd core [epnum][dir] (1=IN): busy/stalled/claimed +p/x <port's private state> # per-port names — read the board's dcd_*.c first +x/32wx <USB peripheral base> # raw EP/FIFO regs; base = the macro the dcd uses +watch xfer_status[2][1].total_len # HW watchpoint (Cortex-M: ~4); dwc2 names shown +break dcd_int_handler # works, but see warning below +``` + +While halted the device answers **nothing**: host control transfers time out +in ~5 s and the OS may reset/re-enumerate — after `continue`, the bus traffic +shows recovery, not the original bug. Prefer one halt for a post-mortem dump +over stepping through live USB traffic. + +## RAM ring-buffer trace + +The zero-print instrument (cracked the musb babble): a small event ring in the +dcd/hcd, dumped over GDB after the failure. Single-writer (ISR) — no locking: + +```c +typedef struct { uint16_t ev; uint16_t a; uint32_t b; } dbg_ev_t; +#define DBG_N 512 // power of two +static volatile dbg_ev_t dbg_ring[DBG_N]; // volatile REQUIRED: -Os dead-store- +static volatile uint32_t dbg_wr; // eliminates a write-only static array +static inline void DBG_EV(uint16_t ev, uint16_t a, uint32_t b) { + uint32_t i = dbg_wr++; + dbg_ring[i & (DBG_N - 1)] = (dbg_ev_t){ ev, a, b }; +} +// call sites: DBG_EV(__LINE__, ep_addr, count); — __LINE__ as event id +``` + +After building, `nm` the ELF for `dbg_ring`/`dbg_wr` — if they're missing the +compiler deleted your instrument and the run will "reproduce" with an empty ring. + +Order is the index; if durations matter add a `uint32_t t = DWT->CYCCNT` field +(enable once: `CoreDebug->DEMCR |= CoreDebug_DEMCR_TRCENA_Msk; DWT->CTRL |= 1;` +RISC-V: read `mcycle`). Let the failure happen, halt, then: + +```gdb +p dbg_wr # total events; oldest slot = dbg_wr & (DBG_N-1) once wrapped +p dbg_ring +dump binary memory /tmp/ring.bin &dbg_ring[0] &dbg_ring[512] +``` + +## PC-sampling (J-Link) — find where the core spins, without halting + +`DWT_PCSR` (0xE000101C) returns the current PC on every read, target running +(Cortex-M3+; optional on M0+, reads 0 if absent; 0xFFFFFFFF = core halted or +WFI-asleep — `mem32 E000EDF0, 1`, DHCSR bit 17 S_HALT, tells which). One +probe serves one client: quit JLinkExe before starting JLinkGDBServer on the +same probe. Nailed the rusb2 FRDY wedge: + +```bash +for i in $(seq 300); do echo 'mem32 E000101C, 1'; done \ + | JLinkExe -device $JLINK_DEVICE -SelectEmuBySN <uid> -if swd -speed 4000 -autoconnect 1 -nogui 1 \ + | awk '/E000101C = /{print $3}' | sort | uniq -c | sort -rn | head +arm-none-eabi-addr2line -e <firmware.elf> -f -a 0x<hot-pc> ... # PCs → functions +``` + +OpenOCD variant: repeat `mdw 0xE000101C` over telnet :4444. The histogram's +top entries are the spin site; a flat histogram = core is servicing normally. + +## Dual-side capture — the default for enumeration/transfer bugs + +Start both channels, then trigger the failing test: + +```bash +.claude/skills/usbmon/scripts/usbcap.sh cafe: 30 /tmp/host.pcapng & # host URBs (usbmon skill) +timeout 30s JLinkRTTClient > /tmp/target.rtt & # target (or ring dump after) +wait +``` + +RTT lines and ring events carry no wall-clock: correlate on unambiguous +anchors — bus reset, SET_ADDRESS, the first transfer on the failing EP — then +lay device events between anchors in host-URB order. Logging the SOF/frame +number on the target gives a shared clock when you need finer alignment. +When host and target evidence disagree, or the host sees nothing at all, add +the wire itself: `usb-sniffer` skill (hardware tap, PID-level). + +## Warnings + +- **Halting/resetting via the probe does NOT disconnect the device**: a DWC2 + soft-connect pullup stays up through core halt *and* reset, so the host's + stuck URBs stay stuck and a wedged DUT stays wedged — recover the host side + with the `usb-recover` skill. +- **A bug that vanishes under LOG=2 is a timing bug**, not fixed: switch to + the ring buffer; if it vanishes under GDB too, PC-sampling only. +- **UART TU_LOG blocks in the write path** (worst perturbation, including + inside the ISR); RTT is much cheaper but not free; `LOG=3` multiplies both. +- Flash/GDB only with the board lock held; a `hold` refused with reason + `hil_test.py` means CI is mid-test on that board — wait, don't force. +- **Instrumentation is temporary**: before `release`, reflash pristine + firmware (the next CI run must not inherit a debug build) and revert the + instrumentation diff — or hand it over explicitly with the diagnosis. +- **A register snapshot without a validity anchor lies**: J-Link tool sessions + can reset or briefly halt the DUT as a side effect, and a snapshot of a + freshly-reset chip (e.g. NVIC ISER = 0) reads like a smoking gun. Read DHCSR + (0xE000EDF0: bit 17 S_HALT, bit 25 S_RESET_ST) with every snapshot, and + cross-check against something the device demonstrably still does. +- **A marginal link can fake a deterministic firmware bug** — down to failing + the same test at the same iteration twice. "USB disconnect" in dmesg on a + freshly re-cabled port (high devnum = churn) means the plug, not the code: + first sustained bulk traffic is when a bad contact drops. Before declaring a + regression, re-run the OLD build on the SAME link state — and if a bisect + exonerates every hunk, believe it: re-test the exact failing binary. +- **Release your manual lock before `hil_test.py`** — it self-locks each board + and fails immediately on your own hold (`hil` skill). diff --git a/.gitignore b/.gitignore index 61358d118..145069e72 100644 --- a/.gitignore +++ b/.gitignore @@ -60,6 +60,7 @@ BrowseInfo README_processed.rst docs/examples/ .worktrees +.claude/worktrees/ cmake-metrics/ # Directories fetched by tools/get_deps.py - not to be committed lib/CMSIS_5/ diff --git a/docs/superpowers/plans/2026-07-14-usb-target-debug-handoff.md b/docs/superpowers/plans/2026-07-14-usb-target-debug-handoff.md new file mode 100644 index 000000000..b56d035d9 --- /dev/null +++ b/docs/superpowers/plans/2026-07-14-usb-target-debug-handoff.md @@ -0,0 +1,125 @@ +# Hand-off: `usb-target-debug` skill + `target-debugger` agent + +**Status: agreed but NOT started.** Design discussion happened 2026-07-13 in session +`c31a4617-43b1-491d-9865-3e35f393996b` (post-merge of the agents/workflows harness, +PR #3762 / `ac595bc5c`). This document is the implementation brief for a fresh session. + +**Agreed sequencing: skill first → dogfood on 1-2 real HIL failures → then the agent +as its own small PR.** Do not build both at once — the agent charter's hard parts are +exactly what dogfooding the skill answers. + +## The gap being filled + +When HIL fails today, *what failed* is covered (hil-validate workflow, hil-operator +agent) but the deep *why* loop — instrument the target, capture on both sides, +correlate — has no skill and no agent. Every hard case so far (musb babble, rusb2 +FRDY wedge, ch32v307 Heisenbug) fell back to interactive main-session work. + +Why no existing agent can do it: + +- **hil-operator** (sonnet) is deliberately mechanical: lock → flash → `hil_test.py` + → recover. It never edits source, so it cannot inject instrumentation. +- **port-dev** can edit source but its charter is scoped changes verified by a + *build*; it has no hardware mandate. +- The host-side capture knowledge lives in skills (`usbmon`, `usb-debug`); the + device-side half exists only as CLAUDE.md recipes plus session memory. + +The skill completes the debugging trio: + +| Skill | Answers | Status | +|---|---|---| +| `usbmon` | what the host actually exchanged (URBs) | on master | +| `usb-debug` | why the host acted (dmesg / dynamic debug) | ships in PR #3758 (untracked copy in tree) | +| `usb-target-debug` | what the device did | **this hand-off** | + +## Part 1 — `usb-target-debug` skill (do this first) + +Create `.claude/skills/usb-target-debug/SKILL.md`. Match the style of +`.claude/skills/usbmon/SKILL.md` and `usb-debug/SKILL.md`: frontmatter `name` + +`description` where the description states concretely *when* to reach for it +(HIL test fails and host-side capture can't explain it; device silently NAKs, +wedges, or misbehaves; need TU_LOG/device-state evidence from real hardware). + +Playbook to codify — all techniques already proven on this rig: + +1. **TU_LOG capture** — build with `LOG=2` (add `LOGGER=rtt` for RTT); UART capture + from the board's debug serial; RTT via `JLinkGDBServer -RTTTelnetPort 19021` + + `JLinkRTTClient` (non-interactive: `timeout 20s JLinkRTTClient > rtt.log`). + Note which log level perturbs timing (see warning #6). +2. **GDB recipes per probe family** — J-Link, OpenOCD (ST-Link / CMSIS-DAP / + WCH-Link). Base connect/load recipes already exist in CLAUDE.md "GDB Debugging"; + the skill adds the debug-loop specifics: breakpoints in ISR context, dumping + endpoint/FIFO registers, watchpoints on driver state variables. +3. **RAM ring-buffer trace pattern** (used to crack the musb babble): instrument + the dcd/hcd with a small RAM ring of event records instead of TU_LOG when + printing perturbs timing; let the failure happen; halt and dump the ring via + GDB. Include a minimal C snippet (fixed-size struct ring, no allocation, + ISR-safe single-writer). +4. **J-Link PC-sampling** (nailed the rusb2 FRDY wedge): statistically sample PC + without halting to find where the core spins — the non-intrusive option when + halting or logging masks the bug. +5. **Dual-side capture**: usbmon on the host + RTT/ring-buffer on the target, + simultaneously; correlate host URBs against device events on one timeline. + This is the default posture for enumeration/transfer bugs, not an escalation. +6. **Warnings**: observation can mask the bug (the ch32v307 case changed behavior + under logging/debug — prefer ring-buffer over TU_LOG, PC-sampling over halting, + and say so explicitly); a J-Link core reset does NOT drop a DWC2 soft-connect + pullup, so a wedged DUT stays wedged on the host side (cross-ref + `usb-recover/SKILL.md`). +7. **Rig discipline**: hold the board lock for the whole manual session — + `python3 test/hil/board_lock.py hold <board> --reason "target debug: <bug>"` + … work … `release <board>`. Never stop the actions-runner. Board → probe + mapping via `test/hil/tinyusb.json`; `JLINK_DEVICE`/`OPENOCD_OPTION` via + `hw/bsp/*/boards/*/board.cmake` or `board.mk`. + +**Where to ship**: its own small PR (usb-recover/usb-debug already belong to +PR #3758 — don't grow that one), or fold into #3758 if it is still open and being +rebased anyway. User's call at the time. + +## Part 2 — `target-debugger` agent (later, after dogfooding) + +Create `.claude/agents/target-debugger.md` as its own PR once the skill has been +through at least one real debug session. + +Agreed charter outline: + +- **Frontmatter**: `model: opus`; omit `tools:` (= all tools — it must edit source + AND drive hardware). Note the registry supports no `effort` field — the agreed + opus/**xhigh** tier is requested per `agent()` call by whichever workflow or + session spawns it. +- **Loop**: instrument → build → flash under one held board lock → dual-side + capture (host usbmon + target RTT/ring-buffer/GDB) → correlate → refine + hypothesis → repeat. Deliberately serial: no fan-out win; the value is + backgrounding a long debug session and the codified playbook. +- **Strictly one instance**, holds the board lock for the entire session — its work + is exactly the "hardware work outside hil_test.py" case in the lock protocol. +- **Skills are its source of truth** (mirror hil-operator's pattern): read + `usb-target-debug`, `usbmon`, `usb-debug`, `usb-recover`, `hil` SKILL.md files + before acting. +- **Hard rule — instrumentation is temporary**: the instrumentation diff must be + reverted (or explicitly listed in the hand-back report) at session end; the *fix* + itself goes to port-dev. Keeps charters clean: this agent produces a diagnosis + and evidence, not a merged patch. + +Questions dogfooding must answer before the charter is written (do NOT guess these +now — that was the whole reason for skill-first): + +1. When to stop instrumenting and report a partial diagnosis vs keep digging. +2. Maximum board-lock hold time / check-in cadence for a backgrounded session. +3. What "revert instrumentation" means when a partial fix emerged mid-debug + (revert + attach diff? keep on a branch?). + +## Conventions and references for the implementing session + +- Skill style exemplars: `.claude/skills/usbmon/SKILL.md`, `usb-debug/SKILL.md`, + `usb-recover/SKILL.md` (the latter two are #3758's copies, present untracked). +- Agent style exemplars: `.claude/agents/hil-operator.md` (lock discipline, + skills-as-source-of-truth), `port-dev.md` (source-edit + verify charter). +- When the agent lands, update the harness spec's agent roster: + `docs/superpowers/specs/2026-07-09-claude-agents-workflows-design.md` + (convention: spec evolves in-repo; plans like this file are per-effort records). +- Agents register from `.claude/agents/*.md` at session start — a new agent file + is only visible to sessions launched after it exists. +- Past cases to mine for the skill's examples: musb babble (ring-buffer trace), + rusb2 FRDY wedge (J-Link PC-sampling), ch32v307 Heisenbug (observation + sensitivity) — details in session memory and the referenced session transcript. |