From 8918c4fec4a99b174d480c043000fd4405678282 Mon Sep 17 00:00:00 2001 From: hathach Date: Thu, 23 Jul 2026 10:18:11 +0700 Subject: docs(skills): rename debug skills, drop the PC-host/TinyUSB-device assumption Rename usb-target-debug -> target-debug, usb-debug -> usb-kernel-debug, usb-recover -> usb-kernel-recover (script filenames unchanged), and make all debug skills/agents decide tool applicability by which end of the link runs Linux: TinyUSB may run the device or host stack, and its peer may be a Linux PC, another TinyUSB board, or a Linux gadget (e.g. Raspberry Pi UDC). - usbmon: exists only when a Linux PC is the link's host - usb-kernel-debug: either Linux end; allowlist gains dwc3/libcomposite/udc_core for the gadget side of a Linux peer - usb-sniffer: the only full-visibility capture when TinyUSB is the host - target-debug: covers dcd_* and hcd_*/tuh_ debugging; channel choice by topology - update target-debugger/hil-operator agents, pre-pr, hil-validate.js, and the USB_RECOVER path constant in test/hil/usbtest.py - CLAUDE.md: fold the dcd/hcd datasheet cross-check rule into the read-doc line --- .claude/agents/hil-operator.md | 6 +- .claude/agents/target-debugger.md | 31 ++-- .claude/skills/pre-pr/SKILL.md | 2 +- .claude/skills/target-debug/SKILL.md | 206 +++++++++++++++++++++ .claude/skills/usb-debug/SKILL.md | 36 ---- .claude/skills/usb-debug/scripts/usb_dyndbg.sh | 46 ----- .claude/skills/usb-kernel-debug/SKILL.md | 47 +++++ .../skills/usb-kernel-debug/scripts/usb_dyndbg.sh | 47 +++++ .claude/skills/usb-kernel-recover/SKILL.md | 107 +++++++++++ .../usb-kernel-recover/scripts/usb_recover.sh | 140 ++++++++++++++ .claude/skills/usb-recover/SKILL.md | 107 ----------- .claude/skills/usb-recover/scripts/usb_recover.sh | 140 -------------- .claude/skills/usb-sniffer/SKILL.md | 14 +- .claude/skills/usb-target-debug/SKILL.md | 198 -------------------- .claude/skills/usbmon/SKILL.md | 4 +- .claude/skills/usbtest/SKILL.md | 2 +- .claude/workflows/hil-validate.js | 2 +- CLAUDE.md | 2 +- test/hil/usbtest.py | 6 +- 19 files changed, 586 insertions(+), 557 deletions(-) create mode 100644 .claude/skills/target-debug/SKILL.md delete mode 100644 .claude/skills/usb-debug/SKILL.md delete mode 100755 .claude/skills/usb-debug/scripts/usb_dyndbg.sh create mode 100644 .claude/skills/usb-kernel-debug/SKILL.md create mode 100755 .claude/skills/usb-kernel-debug/scripts/usb_dyndbg.sh create mode 100644 .claude/skills/usb-kernel-recover/SKILL.md create mode 100755 .claude/skills/usb-kernel-recover/scripts/usb_recover.sh delete mode 100644 .claude/skills/usb-recover/SKILL.md delete mode 100755 .claude/skills/usb-recover/scripts/usb_recover.sh delete mode 100644 .claude/skills/usb-target-debug/SKILL.md diff --git a/.claude/agents/hil-operator.md b/.claude/agents/hil-operator.md index c48ceb8bd..d19eca047 100644 --- a/.claude/agents/hil-operator.md +++ b/.claude/agents/hil-operator.md @@ -8,8 +8,8 @@ model: sonnet You operate physical USB test hardware. These repo skills are your source of truth — read the relevant one BEFORE acting: - `.claude/skills/hil/SKILL.md` — run `hostname` first (host `ci` = local mode with `test/hil/tinyusb.json`; host `htpc` = local `local.json` or remote via `test/hil/hil_ci.sh`); the board lock protocol; exact `hil_test.py` invocations. -- `.claude/skills/usb-recover/SKILL.md` — only when a device/fixture is wedged or processes hang in D state. -- `.claude/skills/usb-debug/SKILL.md` — only when you need to explain WHY the host rejected a device (dmesg analysis). +- `.claude/skills/usb-kernel-recover/SKILL.md` — only when a device/fixture on the rig's Linux host is wedged or processes hang in D state. +- `.claude/skills/usb-kernel-debug/SKILL.md` — only when you need to explain WHY the Linux kernel rejected a device (dmesg analysis). ## Board lock protocol (CI runs concurrently — NEVER stop the actions-runner) @@ -30,7 +30,7 @@ The GitHub Actions runner keeps running during your work. Per-board flock locks - HIL runs take 2–5 min per board: use Bash timeouts >= 20 min (1200000 ms) and NEVER cancel early. - One hardware action at a time. You are never run concurrently with another hil-operator. -- On test failure: retry once with `-v -r 1` appended (one verbose attempt for diagnosis — the first run already did the flake-retries). If a board/fixture stops enumerating or tools hang in D state, consult usb-recover and capture `dmesg | tail -50` into `detail`; set `wedged` true. +- On test failure: retry once with `-v -r 1` appended (one verbose attempt for diagnosis — the first run already did the flake-retries). If a board/fixture stops enumerating or tools hang in D state, consult usb-kernel-recover and capture `dmesg | tail -50` into `detail`; set `wedged` true. ## Output contract diff --git a/.claude/agents/target-debugger.md b/.claude/agents/target-debugger.md index c1df47cb2..861600428 100644 --- a/.claude/agents/target-debugger.md +++ b/.claude/agents/target-debugger.md @@ -1,25 +1,32 @@ --- name: target-debugger -description: Root-cause one USB misbehavior on real HIL hardware by instrumenting the TinyUSB device side — TU_LOG/RTT, RAM ring-buffer trace, GDB autopsy, J-Link PC-sampling — correlated with host-side and wire-level capture. Long serial debug loop under one held board lock; strictly one instance. Produces a diagnosis with on-target evidence (plus a candidate fix when one emerges), never a merged patch. +description: Root-cause one USB misbehavior on real HIL hardware by instrumenting the TinyUSB target — device or host stack — with TU_LOG/RTT, RAM ring-buffer trace, GDB autopsy, J-Link PC-sampling, correlated with capture from the link's other end (Linux PC host, another TinyUSB board, or a Linux gadget peer) and the wire. Long serial debug loop under one held board lock; strictly one instance. Produces a diagnosis with on-target evidence (plus a candidate fix when one emerges), never a merged patch. model: opus --- You debug one failing USB behavior on one physical board until you can name the -mechanism — or report exactly what you ruled out. These repo skills are your -source of truth; read the relevant SKILL.md BEFORE acting: +mechanism — or report exactly what you ruled out. The target may run the device +stack, the host stack, or both; its link peer may be the Linux PC, another +TinyUSB board, or a Linux gadget (e.g. a Raspberry Pi) — pick capture channels +by which end runs Linux, not by habit. These repo skills are your source of +truth; read the relevant SKILL.md BEFORE acting: -- `.claude/skills/usb-target-debug/SKILL.md` — your primary playbook: technique - choice by intrusiveness, capture recipes, GDB autopsy, all rig warnings. +- `.claude/skills/target-debug/SKILL.md` — your primary playbook: technique + choice by intrusiveness, channel choice by link topology, capture recipes, + GDB autopsy, all rig warnings. - `.claude/skills/hil/SKILL.md` — host/config selection, board lock protocol, `hil_test.py` invocation. -- `.claude/skills/usbmon/SKILL.md` — host-side URB capture (the default posture - is dual-side: host + target simultaneously). +- `.claude/skills/usbmon/SKILL.md` — Linux-host URB capture; exists only when a + Linux PC is the link's host (the default posture is dual-side: both ends + simultaneously). - `.claude/skills/usb-sniffer/SKILL.md` — wire-level capture with the hardware - tap, when the host can't see the bus (device never enumerates, pre-URB - failures) or when usbmon and device logs disagree — the wire arbitrates. -- `.claude/skills/usb-debug/SKILL.md` — why the host acted (dmesg/dynamic debug). -- `.claude/skills/usb-recover/SKILL.md` — only when the DUT or fixture wedges - the host stack. + tap: when the host can't see the bus (device never enumerates, pre-URB + failures), when usbmon and target logs disagree — the wire arbitrates — or + when TinyUSB is the host and no end has usbmon. +- `.claude/skills/usb-kernel-debug/SKILL.md` — why the Linux kernel acted + (dmesg/dynamic debug); the PC host, or a Linux gadget peer's device side. +- `.claude/skills/usb-kernel-recover/SKILL.md` — only when the DUT or fixture + wedges the rig PC's Linux host stack. ## The loop (deliberately serial — no fan-out) diff --git a/.claude/skills/pre-pr/SKILL.md b/.claude/skills/pre-pr/SKILL.md index d4c35f7e5..3f062db78 100644 --- a/.claude/skills/pre-pr/SKILL.md +++ b/.claude/skills/pre-pr/SKILL.md @@ -38,5 +38,5 @@ Invoke the Workflow tool: - Per-stage table: unit / build: / size / pvs, then HIL per board — pass/fail with the first error for each failure. - If the hardware result has non-empty `locked` (a CI job held those boards): ask the user with AskUserQuestion — **Force now** (re-invoke `hil-validate` with `force: true` for those boards; user accepts the risk of colliding with a mid-test CI job), **Keep waiting** (re-invoke `hil-validate` for them after a few minutes; ask again if still locked), or **Accept** the partial verdict. Never force without the user's answer. -- Wedged boards: point at `.claude/skills/usb-recover/SKILL.md`. +- Wedged boards: point at `.claude/skills/usb-kernel-recover/SKILL.md`. - End with a clear ship / no-ship verdict and what to fix first. diff --git a/.claude/skills/target-debug/SKILL.md b/.claude/skills/target-debug/SKILL.md new file mode 100644 index 000000000..c82237ea3 --- /dev/null +++ b/.claude/skills/target-debug/SKILL.md @@ -0,0 +1,206 @@ +--- +name: target-debug +description: Use when TinyUSB firmware — device or host stack — misbehaves on real hardware and capture from the other end can't explain it: a HIL test fails but usbmon shows only Submits with no Completes, the device silently NAKs, wedges, STALLs, babbles, or drops data, EP0 starves, tuh_ enumeration of an attached device fails, an ISR or DCD/HCD state bug is suspected — and you need target-side evidence: TU_LOG/RTT logs, GDB state dumps, a RAM ring-buffer event trace, or PC-sampling of where the core spins. +--- + +# target-debug — target-side capture & debugging on the HIL rig + +The **target** is whichever MCU runs TinyUSB — device stack (`dcd_*`), host +stack (`hcd_*`/`tuh_*`), or both. Its link peer is not always a Linux PC: a +TinyUSB host may face another TinyUSB board or a Linux gadget (e.g. a +Raspberry Pi). Pick capture channels by which end runs Linux, not by habit: + +| Skill | Answers | Exists when | +|---|---|---| +| `usbmon` | what the Linux host exchanged (URBs) | a Linux PC is the link's host | +| `usb-kernel-debug` | why the Linux kernel acted (dmesg / dynamic debug) | Linux on either end: PC host or Linux gadget peer | +| **`target-debug`** | **what the target did** (logs, driver state, PC) | always — either role, needs a debug probe | +| `usb-sniffer` | what crossed the wire (PIDs, handshakes, resets) | hardware tap cabled in — role-agnostic | + +For enumeration/transfer bugs the default posture is **dual-side capture** — +both ends simultaneously, not one-side-first-then-escalate: usbmon plus a +target channel when a Linux PC is the host. When TinyUSB is the host there is +no usbmon on either end — pair the target channel with the wire +(`usb-sniffer`) and, if the peer is a Linux gadget, `usb-kernel-debug` on the +peer. + +## Rig discipline — lock first, always + +Hold the board lock for the WHOLE manual session; never stop the +actions-runner (see the `hil` skill for the full lock protocol): + +```bash +python3 test/hil/board_lock.py hold --reason "target debug: " +# ... instrument / build / flash / capture / GDB ... +python3 test/hil/board_lock.py release +``` + +Board → probe mapping: `test/hil/tinyusb.json` — `flasher.name` is the probe +family, `flasher.uid` the **probe serial** (many identical probes on the rig: +J-Link needs `-SelectEmuBySN ` / GDB server `-select usb=`; OpenOCD +`-c 'adapter serial '`). `JLINK_DEVICE` / `OPENOCD_OPTION` come from +`hw/bsp//boards//board.cmake` (or `board.mk`); find the family +with `ls -d hw/bsp/*/boards/`. Run on the host that owns the probe — +config is `test/hil/tinyusb.json` on ci, `local.json` on htpc (`hil` skill). + +## Pick the least intrusive technique that can answer the question + +Observation can mask the bug — the ch32v307 Heisenbug changed behavior under +logging *and* under the debugger. If the bug disappears when instrumented, +that IS a finding (timing-sensitive): move down in intrusiveness, not up. + +| Technique | Intrusiveness | Reach for it when | +|---|---|---| +| PC-sampling | none — no halt, no code change | core wedged/spinning somewhere unknown (rusb2 FRDY) | +| RAM ring-buffer | ~tens of cycles per event | ISR ordering/timing bugs (musb babble) | +| TU_LOG (RTT) | µs per line | logic bugs that survive logging | +| TU_LOG (UART) | ms per line — blocking write | same, when no J-Link on the board | +| GDB halt / breakpoints | stops USB service entirely | post-mortem state autopsy once wedged | + +## TU_LOG capture + +Build with `LOG=2` (`LOG=3` adds per-transfer noise and much more timing skew). +`LOGGER=rtt` routes it over the debug probe (J-Link only) — no UART wiring: + +```bash +# RTT: JLinkGDBServer from CLAUDE.md "GDB Debugging" + -RTTTelnetPort, then: +timeout 20s JLinkRTTClient > /tmp/rtt.log # non-interactive capture +# UART (board's debug serial, if wired): +stty -F /dev/ttyACM 115200 raw && timeout 20s cat /dev/ttyACM | tee /tmp/uart.log +``` + +An RTT-built firmware that has since wedged still holds a log tail in RAM — +but ONLY what fits the drain model: the default SEGGER mode (NO_BLOCK_SKIP) +**drops** writes once the ring fills with no reader, so an undrained target +holds the first KB after boot, not the wedge tail. There is no overwrite mode +in stock SEGGER RTT (only SKIP/TRIM/BLOCK): post-mortem RTT is evidence only +if a live drain was running — otherwise instrument with the RAM ring below. +Use `JLinkGDBServer -RTTTelnetPort 19021` + `JLinkRTTClient` for the drain +(proven; note the server briefly halts the core on connect). `JLinkRTTLogger` +fails to find the control block on some parts (LPC4088) even when it exists +and even given `-RTTAddress`; don't fight it — `nm` the ELF for `_SEGGER_RTT`, +read the aUp[0] descriptor (`mem32`), `savebin` the buffer — debug-AP RAM +reads don't halt the target. + +## GDB — state autopsy and watchpoints + +Connect/load recipes per probe family (J-Link, OpenOCD for ST-Link / +CMSIS-DAP / WCH-Link) are in CLAUDE.md "GDB Debugging". Release builds keep +DWARF (`MinSizeRel`), so `p`/struct access works on HIL firmware. + +**Autopsy of a wedged board: attach and halt ONLY** — skip CLAUDE.md's +`monitor reset halt` + `load` (those are for fresh starts; a reset destroys +the evidence). Symbolize with the ELF that is actually flashed — +`/cmake-build-//.elf` from the run that +wedged; do not rebuild while the wedge is still on the board. The debug-loop +specifics: + +```gdb +p/x _usbd_dev.ep_status # device stack: usbd [epnum][dir] (1=IN): busy/stalled/claimed +p _usbh_devices[0] # host stack: usbh per-device state (addr, enum/config) +p/x # per-port names — read the board's dcd_*.c first +x/32wx # raw EP/FIFO regs; base = the macro the dcd uses +watch xfer_status[2][1].total_len # HW watchpoint (Cortex-M: ~4); dwc2 names shown +break dcd_int_handler # works, but see warning below +``` + +While halted the device answers **nothing**: host control transfers time out +in ~5 s and the OS may reset/re-enumerate — after `continue`, the bus traffic +shows recovery, not the original bug. Prefer one halt for a post-mortem dump +over stepping through live USB traffic. + +## RAM ring-buffer trace + +The zero-print instrument (cracked the musb babble): a small event ring in the +dcd/hcd, dumped over GDB after the failure. Single-writer (ISR) — no locking: + +```c +typedef struct { uint16_t ev; uint16_t a; uint32_t b; } dbg_ev_t; +#define DBG_N 512 // power of two +static volatile dbg_ev_t dbg_ring[DBG_N]; // volatile REQUIRED: -Os dead-store- +static volatile uint32_t dbg_wr; // eliminates a write-only static array +static inline void DBG_EV(uint16_t ev, uint16_t a, uint32_t b) { + uint32_t i = dbg_wr++; + dbg_ring[i & (DBG_N - 1)] = (dbg_ev_t){ ev, a, b }; +} +// call sites: DBG_EV(__LINE__, ep_addr, count); — __LINE__ as event id +``` + +After building, `nm` the ELF for `dbg_ring`/`dbg_wr` — if they're missing the +compiler deleted your instrument and the run will "reproduce" with an empty ring. + +Order is the index; if durations matter add a `uint32_t t = DWT->CYCCNT` field +(enable once: `CoreDebug->DEMCR |= CoreDebug_DEMCR_TRCENA_Msk; DWT->CTRL |= 1;` +RISC-V: read `mcycle`). Let the failure happen, halt, then: + +```gdb +p dbg_wr # total events; oldest slot = dbg_wr & (DBG_N-1) once wrapped +p dbg_ring +dump binary memory /tmp/ring.bin &dbg_ring[0] &dbg_ring[512] +``` + +## PC-sampling (J-Link) — find where the core spins, without halting + +`DWT_PCSR` (0xE000101C) returns the current PC on every read, target running +(Cortex-M3+; optional on M0+, reads 0 if absent; 0xFFFFFFFF = core halted or +WFI-asleep — `mem32 E000EDF0, 1`, DHCSR bit 17 S_HALT, tells which). One +probe serves one client: quit JLinkExe before starting JLinkGDBServer on the +same probe. Nailed the rusb2 FRDY wedge: + +```bash +for i in $(seq 300); do echo 'mem32 E000101C, 1'; done \ + | JLinkExe -device $JLINK_DEVICE -SelectEmuBySN -if swd -speed 4000 -autoconnect 1 -nogui 1 \ + | awk '/E000101C = /{print $3}' | sort | uniq -c | sort -rn | head +arm-none-eabi-addr2line -e -f -a 0x ... # PCs → functions +``` + +OpenOCD variant: repeat `mdw 0xE000101C` over telnet :4444. The histogram's +top entries are the spin site; a flat histogram = core is servicing normally. + +## Dual-side capture — the default for enumeration/transfer bugs + +Start both channels, then trigger the failing test (Linux-PC-host link shown; +TinyUSB-as-host: swap the usbmon line for a `usb-sniffer` capture, plus +`usb-kernel-debug` on the peer if it is a Linux gadget): + +```bash +.claude/skills/usbmon/scripts/usbcap.sh cafe: 30 /tmp/host.pcapng & # host URBs (usbmon skill) +timeout 30s JLinkRTTClient > /tmp/target.rtt & # target (or ring dump after) +wait +``` + +RTT lines and ring events carry no wall-clock: correlate on unambiguous +anchors — bus reset, SET_ADDRESS, the first transfer on the failing EP — then +lay device events between anchors in host-URB order. Logging the SOF/frame +number on the target gives a shared clock when you need finer alignment. +When host and target evidence disagree, or the host sees nothing at all, add +the wire itself: `usb-sniffer` skill (hardware tap, PID-level). + +## Warnings + +- **Halting/resetting via the probe does NOT disconnect the device**: a DWC2 + soft-connect pullup stays up through core halt *and* reset, so the host's + stuck URBs stay stuck and a wedged DUT stays wedged — recover the Linux + host side with the `usb-kernel-recover` skill. +- **A bug that vanishes under LOG=2 is a timing bug**, not fixed: switch to + the ring buffer; if it vanishes under GDB too, PC-sampling only. +- **UART TU_LOG blocks in the write path** (worst perturbation, including + inside the ISR); RTT is much cheaper but not free; `LOG=3` multiplies both. +- Flash/GDB only with the board lock held; a `hold` refused with reason + `hil_test.py` means CI is mid-test on that board — wait, don't force. +- **Instrumentation is temporary**: before `release`, reflash pristine + firmware (the next CI run must not inherit a debug build) and revert the + instrumentation diff — or hand it over explicitly with the diagnosis. +- **A register snapshot without a validity anchor lies**: J-Link tool sessions + can reset or briefly halt the DUT as a side effect, and a snapshot of a + freshly-reset chip (e.g. NVIC ISER = 0) reads like a smoking gun. Read DHCSR + (0xE000EDF0: bit 17 S_HALT, bit 25 S_RESET_ST) with every snapshot, and + cross-check against something the device demonstrably still does. +- **A marginal link can fake a deterministic firmware bug** — down to failing + the same test at the same iteration twice. "USB disconnect" in dmesg on a + freshly re-cabled port (high devnum = churn) means the plug, not the code: + first sustained bulk traffic is when a bad contact drops. Before declaring a + regression, re-run the OLD build on the SAME link state — and if a bisect + exonerates every hunk, believe it: re-test the exact failing binary. +- **Release your manual lock before `hil_test.py`** — it self-locks each board + and fails immediately on your own hold (`hil` skill). diff --git a/.claude/skills/usb-debug/SKILL.md b/.claude/skills/usb-debug/SKILL.md deleted file mode 100644 index 20ab7d764..000000000 --- a/.claude/skills/usb-debug/SKILL.md +++ /dev/null @@ -1,36 +0,0 @@ ---- -name: usb-debug -description: Use when USB enumeration fails or misbehaves and usbmon alone can't explain WHY the host acted — port reset storms, repeated re-enumeration, address errors, xHCI ring/command errors, "device descriptor read error", babble, or when you need the host driver's own reasoning from dmesg on the ci HIL rig. ---- - -# usb-debug — host-side kernel dynamic debug for USB - -usbmon shows the URBs; kernel **dynamic debug** shows the host driver's -*reasoning* usbmon can't: port resets and their causes, enumeration retries, -address (re)assignment, EP halts, xHCI ring/command errors. - -Run this skill's `scripts/usb_dyndbg.sh` with `sudo` (abbreviated to -`usb_dyndbg.sh` in the examples below). It flips the dynamic-debug print flag -for an allowlisted set of USB host modules only: - -```bash -sudo usb_dyndbg.sh on usbcore xhci_hcd # enable +p; pick modules from `lsusb -t` Driver= -sudo usb_dyndbg.sh status [module] # list enabled print sites -sudo usb_dyndbg.sh off usbcore xhci_hcd # ALWAYS turn off when done — very noisy -``` - -Allowlisted modules: `usbcore xhci_hcd xhci_pci xhci_pci_renesas ehci_hcd -ehci_pci ohci_hcd ohci_pci uhci_hcd dwc2 cdc_acm usb_storage uas`. - -## Workflow - -1. `sudo usb_dyndbg.sh on usbcore ` — `usbcore` for enumeration/hub - logic, plus the controller module (`lsusb -t` shows the driver per bus). -2. Reproduce (replug / re-enumerate / rerun the failing test) while following - `sudo dmesg -w` (or grab `sudo dmesg | tail` afterwards). -3. `sudo usb_dyndbg.sh off ...` — leaving it on floods the log and skews timing. - -Pair with the `usbmon` skill: usbmon for what crossed the bus, dynamic debug for -why the host reacted. For a wedged device/bus use the `usb-recover` skill. - -Requires `CONFIG_DYNAMIC_DEBUG` and mounted debugfs (standard on distro kernels). diff --git a/.claude/skills/usb-debug/scripts/usb_dyndbg.sh b/.claude/skills/usb-debug/scripts/usb_dyndbg.sh deleted file mode 100755 index 0dc880469..000000000 --- a/.claude/skills/usb-debug/scripts/usb_dyndbg.sh +++ /dev/null @@ -1,46 +0,0 @@ -#!/usr/bin/env bash -# usb_dyndbg.sh — toggle kernel dynamic-debug on USB host drivers; run with sudo. -# Flips +p/-p only on an allowlisted set of USB modules, so it can't reach -# arbitrary kernel debug or unrelated subsystems. -# -# Usage: -# sudo usb_dyndbg.sh on ... # enable +p (e.g. usbcore xhci_hcd) -# sudo usb_dyndbg.sh off ... # disable -p -# sudo usb_dyndbg.sh status [module] # show enabled sites (or one module's sites) -set -euo pipefail - -CTL=/sys/kernel/debug/dynamic_debug/control -# Allowlist: USB host-controller + core + common host class drivers. -ALLOW='usbcore xhci_hcd xhci_pci xhci_pci_renesas ehci_hcd ehci_pci ohci_hcd ohci_pci uhci_hcd dwc2 cdc_acm usb_storage uas' - -die() { echo "usb_dyndbg: $*" >&2; exit 1; } -usage() { - echo "usage: usb_dyndbg.sh {on|off} ... modules: $ALLOW" >&2 - echo " usb_dyndbg.sh status [module]" >&2 - exit 2 -} -allowed() { local m; for m in $ALLOW; do [ "$m" = "$1" ] && return 0; done; return 1; } - -[ -e "$CTL" ] || die "dynamic_debug unavailable (need CONFIG_DYNAMIC_DEBUG + debugfs mounted)" - -action=${1:-}; shift || true -case "$action" in - on|off) - [ "$#" -ge 1 ] || usage - flag='+p'; [ "$action" = off ] && flag='-p' - for m in "$@"; do allowed "$m" || die "module not allowlisted: $m"; done - for m in "$@"; do echo "module $m $flag" > "$CTL"; echo "dynamic debug $action: $m"; done - ;; - status) - m=${1:-} - if [ -n "$m" ]; then - allowed "$m" || die "module not allowlisted: $m" - grep -E "\[$m\]" "$CTL" || echo "(no sites for $m)" - else - grep -E '=p( |$)' "$CTL" || echo "(no print sites enabled)" - fi - ;; - *) - usage - ;; -esac diff --git a/.claude/skills/usb-kernel-debug/SKILL.md b/.claude/skills/usb-kernel-debug/SKILL.md new file mode 100644 index 000000000..e4169b049 --- /dev/null +++ b/.claude/skills/usb-kernel-debug/SKILL.md @@ -0,0 +1,47 @@ +--- +name: usb-kernel-debug +description: Use when USB enumeration fails or misbehaves and packet/URB capture can't explain WHY the Linux kernel acted — port reset storms, repeated re-enumeration, address errors, xHCI ring/command errors, "device descriptor read error", babble — on whichever end of the link runs Linux: the PC host when testing a TinyUSB device, or a Linux gadget peer (e.g. Raspberry Pi) when testing the TinyUSB host stack. +--- + +# usb-kernel-debug — Linux kernel dynamic debug for USB + +Kernel **dynamic debug** shows the Linux side's *reasoning* that packet +capture can't: port resets and their causes, enumeration retries, address +(re)assignment, EP halts, xHCI ring/command errors. It applies wherever Linux +sits in the link — the rig PC when it is the host, or a Linux gadget peer +(dwc2/UDC + gadget modules) when TinyUSB is the host. It cannot see inside +the TinyUSB MCU — that is the `target-debug` skill. + +Run this skill's `scripts/usb_dyndbg.sh` with `sudo` (abbreviated to +`usb_dyndbg.sh` in the examples below). It flips the dynamic-debug print flag +for an allowlisted set of USB modules only: + +```bash +sudo usb_dyndbg.sh on usbcore xhci_hcd # enable +p; pick modules from `lsusb -t` Driver= +sudo usb_dyndbg.sh status [module] # list enabled print sites +sudo usb_dyndbg.sh off usbcore xhci_hcd # ALWAYS turn off when done — very noisy +``` + +Allowlisted modules: `usbcore xhci_hcd xhci_pci xhci_pci_renesas ehci_hcd +ehci_pci ohci_hcd ohci_pci uhci_hcd dwc2 dwc3 cdc_acm usb_storage uas +libcomposite udc_core` (`dwc2`/`dwc3` + the last two cover a Linux gadget +peer's device side). + +## Workflow + +1. `sudo usb_dyndbg.sh on usbcore ` — `usbcore` for enumeration/hub + logic, plus the controller module (`lsusb -t` shows the driver per bus). + On a gadget peer: `dwc2` (or `dwc3`) + `udc_core` + `libcomposite` instead — + run on the peer itself (its SSH/serial console); the script is self-contained, + copy it over or use the raw `dynamic_debug/control` writes from the `usbmon` + skill. +2. Reproduce (replug / re-enumerate / rerun the failing test) while following + `sudo dmesg -w` (or grab `sudo dmesg | tail` afterwards). +3. `sudo usb_dyndbg.sh off ...` — leaving it on floods the log and skews timing. + +On a Linux-PC-host link, pair with the `usbmon` skill: usbmon for what crossed +the bus, dynamic debug for why the kernel reacted. A gadget peer's UDC has no +usbmon — pair with `usb-sniffer` on the wire instead. For a wedged device/bus +on the rig PC use the `usb-kernel-recover` skill. + +Requires `CONFIG_DYNAMIC_DEBUG` and mounted debugfs (standard on distro kernels). diff --git a/.claude/skills/usb-kernel-debug/scripts/usb_dyndbg.sh b/.claude/skills/usb-kernel-debug/scripts/usb_dyndbg.sh new file mode 100755 index 000000000..3923cdc6d --- /dev/null +++ b/.claude/skills/usb-kernel-debug/scripts/usb_dyndbg.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# usb_dyndbg.sh — toggle kernel dynamic-debug on USB drivers (host or gadget +# side); run with sudo. Flips +p/-p only on an allowlisted set of USB modules, +# so it can't reach arbitrary kernel debug or unrelated subsystems. +# +# Usage: +# sudo usb_dyndbg.sh on ... # enable +p (e.g. usbcore xhci_hcd) +# sudo usb_dyndbg.sh off ... # disable -p +# sudo usb_dyndbg.sh status [module] # show enabled sites (or one module's sites) +set -euo pipefail + +CTL=/sys/kernel/debug/dynamic_debug/control +# Allowlist: USB core + host-controller + common class drivers, plus the +# gadget/UDC side of a Linux peer (dwc2/dwc3, udc_core, libcomposite). +ALLOW='usbcore xhci_hcd xhci_pci xhci_pci_renesas ehci_hcd ehci_pci ohci_hcd ohci_pci uhci_hcd dwc2 dwc3 cdc_acm usb_storage uas libcomposite udc_core' + +die() { echo "usb_dyndbg: $*" >&2; exit 1; } +usage() { + echo "usage: usb_dyndbg.sh {on|off} ... modules: $ALLOW" >&2 + echo " usb_dyndbg.sh status [module]" >&2 + exit 2 +} +allowed() { local m; for m in $ALLOW; do [ "$m" = "$1" ] && return 0; done; return 1; } + +[ -e "$CTL" ] || die "dynamic_debug unavailable (need CONFIG_DYNAMIC_DEBUG + debugfs mounted)" + +action=${1:-}; shift || true +case "$action" in + on|off) + [ "$#" -ge 1 ] || usage + flag='+p'; [ "$action" = off ] && flag='-p' + for m in "$@"; do allowed "$m" || die "module not allowlisted: $m"; done + for m in "$@"; do echo "module $m $flag" > "$CTL"; echo "dynamic debug $action: $m"; done + ;; + status) + m=${1:-} + if [ -n "$m" ]; then + allowed "$m" || die "module not allowlisted: $m" + grep -E "\[$m\]" "$CTL" || echo "(no sites for $m)" + else + grep -E '=p( |$)' "$CTL" || echo "(no print sites enabled)" + fi + ;; + *) + usage + ;; +esac diff --git a/.claude/skills/usb-kernel-recover/SKILL.md b/.claude/skills/usb-kernel-recover/SKILL.md new file mode 100644 index 000000000..3f03722fe --- /dev/null +++ b/.claude/skills/usb-kernel-recover/SKILL.md @@ -0,0 +1,107 @@ +--- +name: usb-kernel-recover +description: Use when a USB device or fixture attached to the ci HIL rig's Linux host is stuck, hung, not enumerating, or wedged after a failed flash or test, or when processes touching USB (testusb, JLinkExe, uhubctl, libusb tools) start hanging in D state. Linux-kernel-side only — a bus owned by a TinyUSB host is out of reach (reset the target / cycle its VBUS instead); the rig's probes and serial fixtures always remain in scope. +--- + +# USB Recovery on the HIL Rig (Linux kernel side) + +Run this skill's `scripts/usb_recover.sh` with `sudo` (abbreviated to +`usb_recover.sh` in the examples below). It wraps the sysfs reset actions, a +uhubctl power-cycle escalator, and a resolver: + +```bash +sudo usb_recover.sh resolve /dev/ttyACM3 # /dev node -> busport (e.g. 3-4.7); also ttyUSB*, sg* +sudo usb_recover.sh authorized # deauthorize+reauthorize: re-enumerate, no VBUS cut +sudo usb_recover.sh rebind # usb driver unbind+bind: re-probe +sudo usb_recover.sh hub-cycle # uhubctl VBUS cycle of the feeding port, walking parent hub + # -> root port until the device re-enumerates +sudo usb_recover.sh pci-rebind # whole HCD controller unbind+bind, e.g. 0000:02:00.0 +sudo usb_recover.sh pci-reset # PCI function-level reset: kills URBs at HW level, no device lock +sudo usb_recover.sh pci-bind [drv] # re-bind a DRIVERLESS controller (auto-tries xHCI drivers) +``` + +`hub-cycle` caveats: leaf hubs that gang (or fake) port power switching bounce +**all siblings** on that hub when cycled; a **self-powered** leaf hub keeps +downstream VBUS up, so cycling it only resets its uplink — that's why the walk +escalates to the root port, where the Renesas cards' per-port power (ppps) is +real. A device that is wedged but bus-powered from a switching hub gets a true +power cycle; one on a self-powered hub may only get a re-enumeration. + +## Decide first: is anything stuck in D state? + +```bash +ps -eo pid,stat,wchan:30,cmd | awk '$2 ~ /D/' +``` + +**If yes** (uninterruptible sleep, typically a usbfs ioctl — e.g. testusb inside +`usb_sg_wait`): run `pci-reset` and NOTHING ELSE first: + +```bash +sudo usb_recover.sh pci-reset +``` + +FLR kills the URBs at the hardware level without taking the per-device lock; +the ioctl then returns and the convoy unwinds on its own. + +**Not every controller supports FLR.** The Renesas uPD720201 (`0000:01:00.0`) +has no reset method — `pci-reset` fails with `Inappropriate ioctl for device` +(ENOTTY). On those, there is no clean software D-state cure — a VM reboot is NOT +reliable (downstream hubs can latch up across the PCIe reset and need a physical +replug); ask the operator for a full PVE host power cycle instead. Do NOT +fall through to `pci-rebind` (see next). + +**`pci-rebind` can strand the controller driverless.** Its unbind succeeds but, +with a D-state process still holding a URB, the *re-bind* hangs — leaving the +PCI device with **no driver** (`/sys/bus/pci/devices//driver` gone) and the +whole controller's fixtures offline. A second `pci-rebind` then dies with "no +driver bound". Recover with `pci-bind ` (re-attaches the xHCI driver); +if that also hangs because the D-state URB is unkillable, only a full PVE host +power cycle (operator action) recovers. The Renesas binds via `xhci-pci-renesas` (firmware loader), others via +`xhci_hcd` — `pci-bind` auto-tries both, or pass the driver explicitly. + +**Ordering is critical.** `authorized`/`rebind`/`pci-rebind` all take the +per-device lock the stuck ioctl holds — they block and join the convoy, and +soon every libusb tool (uhubctl, JLinkExe) hangs too. Worse, a blocked +`pci-rebind` grabs the PCI device lock on its way in, which `pci-reset` also +needs: once a rebind has been attempted and is stuck, even FLR deadlocks and +**only a full PVE host power cycle recovers**. pci-reset first (if supported), and never +`pci-rebind` a D-state wedge. + +**If no** (device merely dead or silent), escalate gently: + +1. `authorized ` — re-enumerates just that device +2. `rebind ` — re-probe; also worth trying on the parent hub's busport +3. `hub-cycle ` — VBUS cycle of the feeding port, walking up to the + root port; may bounce sibling fixtures on ganged hubs +4. `pci-rebind ` — last resort: bounces every fixture on that controller + +## Finding targets + +```bash +grep -l /sys/bus/usb/devices/*/serial # serial -> busport (dir name) +readlink -f /sys/bus/usb/devices/usb # bus N -> its PCI addr in the path +``` + +Rig layout (2026-07-15, two Renesas uPD720201 cards; bus numbers renumber every +boot — re-derive with `readlink`): AMD `0000:02:00.0` = the debug-probe tree +(J-Links, ST-Links, WCH-Links), no port power switching; Renesas `0000:01:00.0` +and `0000:03:00.0` = DUT device hubs + serial fixtures, and ALL their root-hub +ports have real per-port power (`ppps`, 4+4 each) — `sudo uhubctl -l -p + -a cycle` cuts VBUS to the leaf hub on that port. The 1a40:0201 leaf +hubs themselves claim "ganged" switching but do not actually cut power. + +## Common mistakes + +- `resolve` takes a **/dev node**, not a busport or serial ("no such device node"). +- `authorized`/`rebind` take a **busport** (`3-4.7`); `pci-rebind`/`pci-reset` + take a **PCI addr**. +- Command produces no output and doesn't return → it is blocked on the device + lock: a D-state holder exists; see above. +- Trying `pci-rebind` on a D-state hang — its re-bind hangs and strands the + controller **driverless**; recover with `pci-bind `, or a PVE host power + cycle if the D-state URB is unkillable. Use `pci-reset` (if supported) for D-state, never + `pci-rebind`. +- Running `pci-reset` on a controller without FLR support (Renesas) → ENOTTY; + no software recovery — needs a PVE host power cycle. +- A J-Link reset (`r; go`) does not disconnect a wedged DUT from the host: the + DWC2 soft-connect pullup stays up through a core halt, so stuck URBs stay stuck. diff --git a/.claude/skills/usb-kernel-recover/scripts/usb_recover.sh b/.claude/skills/usb-kernel-recover/scripts/usb_recover.sh new file mode 100755 index 000000000..7652253fa --- /dev/null +++ b/.claude/skills/usb-kernel-recover/scripts/usb_recover.sh @@ -0,0 +1,140 @@ +#!/usr/bin/env bash +# usb_recover.sh — USB recovery helper for the HIL rig; run with sudo. Writes only +# to the specific sysfs control files below; arg regexes block path traversal. +# +# Usage: +# sudo usb_recover.sh authorized # e.g. 3-2 -> deauthorize+reauthorize (re-enumerate, NO VBUS cut) +# sudo usb_recover.sh rebind # e.g. 3-2 -> usb driver unbind+bind (re-probe) +# sudo usb_recover.sh pci-rebind # e.g. 0000:01:00.0 -> HCD unbind+bind (WHOLE controller) +# sudo usb_recover.sh pci-reset # e.g. 0000:01:00.0 -> PCI function-level reset: kills URBs at +# # HW level WITHOUT the device lock; the only cure when a process +# # is stuck in D state (usbfs ioctl) and unbind paths would convoy +# sudo usb_recover.sh pci-bind [driver] # bind a DRIVERLESS controller (e.g. after a pci-rebind +# # whose re-bind hung and left it unbound). Auto-tries the xHCI +# # drivers (xhci-pci-renesas, xhci_hcd) unless one is named. +# sudo usb_recover.sh hub-cycle # e.g. 13-1.6 -> uhubctl power-cycle of the port feeding it, +# # walking upstream (parent hub -> root port) until the device +# # re-enumerates. Ganged/fake-switching hubs may bounce ALL +# # siblings; self-powered hubs only reset their uplink, which +# # is why the walk ends at the root port (real xHCI ppps). +# sudo usb_recover.sh resolve # e.g. /dev/ttyACM3 -> print its (no privilege needed) +set -euo pipefail + +USBPATH_RE='^[0-9]+-[0-9]+(\.[0-9]+)*$' +PCI_RE='^[0-9a-fA-F]{4}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}\.[0-9]$' +DRIVER_RE='^[A-Za-z0-9_-]+$' + +die() { echo "usb_recover: $*" >&2; exit 1; } +usage() { grep -E '^# sudo usb_recover' "$0" >&2; exit 2; } + +# Refuse to touch a PCI function that is not a USB controller (class 0x0c03xx), so a stray or +# mistyped BDF can't unbind/reset an unrelated device (storage, NIC) on a shared HIL host. +require_usb_controller() { + local addr=$1 cls + cls=$(cat "/sys/bus/pci/devices/$addr/class" 2>/dev/null) || die "no such pci device: $addr" + [[ "$cls" =~ ^0x0c03 ]] || die "$addr is not a USB controller (class $cls); refusing" +} + +# Resolve a /dev node (ttyACMx, ttyUSBx, sgN, ...) up to its USB device busport. +resolve() { + local node=$1 syspath dev + [ -e "$node" ] || die "no such device node: $node" + syspath=$(udevadm info -q path -n "$node" 2>/dev/null) || die "udevadm failed for $node" + dev="/sys$syspath" + while [ "$dev" != "/sys" ] && [ -n "$dev" ]; do + if [ -e "$dev/busnum" ] && [ -e "$dev/devnum" ] && [ -e "$dev/authorized" ]; then + basename "$dev"; return 0 + fi + dev=$(dirname "$dev") + done + die "could not find parent USB device for $node" +} + +action=${1:-}; target=${2:-} +[ -n "$action" ] && [ -n "$target" ] || usage + +case "$action" in + resolve) + resolve "$target" + ;; + authorized) + [[ "$target" =~ $USBPATH_RE ]] || die "bad usb path: $target" + d="/sys/bus/usb/devices/$target" + [ -e "$d/authorized" ] || die "no such usb device: $target" + echo 0 > "$d/authorized"; sleep 1; echo 1 > "$d/authorized" + echo "re-authorized $target" + ;; + rebind) + [[ "$target" =~ $USBPATH_RE ]] || die "bad usb path: $target" + [ -e "/sys/bus/usb/devices/$target" ] || die "no such usb device: $target" + echo "$target" > /sys/bus/usb/drivers/usb/unbind; sleep 1 + echo "$target" > /sys/bus/usb/drivers/usb/bind + echo "rebound $target" + ;; + pci-rebind) + [[ "$target" =~ $PCI_RE ]] || die "bad pci addr: $target" + require_usb_controller "$target" + [ -e "/sys/bus/pci/devices/$target/driver" ] || die "no driver bound to $target" + drv=$(basename "$(readlink -f "/sys/bus/pci/devices/$target/driver")") + echo "$target" > "/sys/bus/pci/drivers/$drv/unbind"; sleep 1 + echo "$target" > "/sys/bus/pci/drivers/$drv/bind" + echo "rebound pci $target ($drv)" + ;; + pci-bind) + # Re-attach a driver to a controller left DRIVERLESS (e.g. a pci-rebind whose re-bind hung). + [[ "$target" =~ $PCI_RE ]] || die "bad pci addr: $target" + require_usb_controller "$target" + [ -e "/sys/bus/pci/devices/$target" ] || die "no such pci device: $target" + [ -e "/sys/bus/pci/devices/$target/driver" ] && die "$target already has a driver bound" + drv=${3:-} + if [ -n "$drv" ]; then + [[ "$drv" =~ $DRIVER_RE ]] || die "bad driver name: $drv" + [ -e "/sys/bus/pci/drivers/$drv/bind" ] || die "no such pci driver: $drv" + echo "$target" > "/sys/bus/pci/drivers/$drv/bind" + echo "bound pci $target ($drv)" + else + # Auto-try the xHCI drivers (Renesas uPD720201 uses xhci-pci-renesas; others xhci_hcd). + for cand in xhci-pci-renesas xhci_hcd; do + [ -e "/sys/bus/pci/drivers/$cand/bind" ] || continue + if echo "$target" > "/sys/bus/pci/drivers/$cand/bind" 2>/dev/null; then + echo "bound pci $target ($cand)"; exit 0 + fi + done + die "could not bind $target with a known xHCI driver; pass the driver explicitly" + fi + ;; + hub-cycle) + [[ "$target" =~ $USBPATH_RE ]] || die "bad usb path: $target" + UHUBCTL=$(command -v uhubctl || echo /sbin/uhubctl) + [ -x "$UHUBCTL" ] || die "uhubctl not installed" + dev="$target" + while :; do + if [[ "$dev" =~ ^([0-9]+)-([0-9]+)$ ]]; then # parent is the root hub + loc="${BASH_REMATCH[1]}"; port="${BASH_REMATCH[2]}"; up="" + else # parent is a downstream hub + loc="${dev%.*}"; port="${dev##*.}"; up="$loc" + fi + echo "hub-cycle: power-cycling hub $loc port $port (feeds $dev)" + "$UHUBCTL" -l "$loc" -p "$port" -a cycle -d 5 -f || echo " (uhubctl failed at $loc; walking up)" + for _ in $(seq 1 10); do + sleep 1 + if [ -e "/sys/bus/usb/devices/$target/idVendor" ]; then + echo "recovered: $target re-enumerated"; exit 0 + fi + done + [ -n "$up" ] || break + dev="$up" + done + die "hub-cycle: $target still not enumerated after cycling up to the root port" + ;; + pci-reset) + [[ "$target" =~ $PCI_RE ]] || die "bad pci addr: $target" + require_usb_controller "$target" + [ -e "/sys/bus/pci/devices/$target/reset" ] || die "no reset support on $target" + echo 1 > "/sys/bus/pci/devices/$target/reset" + echo "flr-reset pci $target" + ;; + *) + usage + ;; +esac diff --git a/.claude/skills/usb-recover/SKILL.md b/.claude/skills/usb-recover/SKILL.md deleted file mode 100644 index beb6fd862..000000000 --- a/.claude/skills/usb-recover/SKILL.md +++ /dev/null @@ -1,107 +0,0 @@ ---- -name: usb-recover -description: Use when a USB device or fixture on the ci HIL rig is stuck, hung, not enumerating, or wedged after a failed flash or test, or when processes touching USB (testusb, JLinkExe, uhubctl, libusb tools) start hanging in D state. ---- - -# USB Recovery on the HIL Rig - -Run this skill's `scripts/usb_recover.sh` with `sudo` (abbreviated to -`usb_recover.sh` in the examples below). It wraps the sysfs reset actions, a -uhubctl power-cycle escalator, and a resolver: - -```bash -sudo usb_recover.sh resolve /dev/ttyACM3 # /dev node -> busport (e.g. 3-4.7); also ttyUSB*, sg* -sudo usb_recover.sh authorized # deauthorize+reauthorize: re-enumerate, no VBUS cut -sudo usb_recover.sh rebind # usb driver unbind+bind: re-probe -sudo usb_recover.sh hub-cycle # uhubctl VBUS cycle of the feeding port, walking parent hub - # -> root port until the device re-enumerates -sudo usb_recover.sh pci-rebind # whole HCD controller unbind+bind, e.g. 0000:02:00.0 -sudo usb_recover.sh pci-reset # PCI function-level reset: kills URBs at HW level, no device lock -sudo usb_recover.sh pci-bind [drv] # re-bind a DRIVERLESS controller (auto-tries xHCI drivers) -``` - -`hub-cycle` caveats: leaf hubs that gang (or fake) port power switching bounce -**all siblings** on that hub when cycled; a **self-powered** leaf hub keeps -downstream VBUS up, so cycling it only resets its uplink — that's why the walk -escalates to the root port, where the Renesas cards' per-port power (ppps) is -real. A device that is wedged but bus-powered from a switching hub gets a true -power cycle; one on a self-powered hub may only get a re-enumeration. - -## Decide first: is anything stuck in D state? - -```bash -ps -eo pid,stat,wchan:30,cmd | awk '$2 ~ /D/' -``` - -**If yes** (uninterruptible sleep, typically a usbfs ioctl — e.g. testusb inside -`usb_sg_wait`): run `pci-reset` and NOTHING ELSE first: - -```bash -sudo usb_recover.sh pci-reset -``` - -FLR kills the URBs at the hardware level without taking the per-device lock; -the ioctl then returns and the convoy unwinds on its own. - -**Not every controller supports FLR.** The Renesas uPD720201 (`0000:01:00.0`) -has no reset method — `pci-reset` fails with `Inappropriate ioctl for device` -(ENOTTY). On those, there is no clean software D-state cure — a VM reboot is NOT -reliable (downstream hubs can latch up across the PCIe reset and need a physical -replug); ask the operator for a full PVE host power cycle instead. Do NOT -fall through to `pci-rebind` (see next). - -**`pci-rebind` can strand the controller driverless.** Its unbind succeeds but, -with a D-state process still holding a URB, the *re-bind* hangs — leaving the -PCI device with **no driver** (`/sys/bus/pci/devices//driver` gone) and the -whole controller's fixtures offline. A second `pci-rebind` then dies with "no -driver bound". Recover with `pci-bind ` (re-attaches the xHCI driver); -if that also hangs because the D-state URB is unkillable, only a full PVE host -power cycle (operator action) recovers. The Renesas binds via `xhci-pci-renesas` (firmware loader), others via -`xhci_hcd` — `pci-bind` auto-tries both, or pass the driver explicitly. - -**Ordering is critical.** `authorized`/`rebind`/`pci-rebind` all take the -per-device lock the stuck ioctl holds — they block and join the convoy, and -soon every libusb tool (uhubctl, JLinkExe) hangs too. Worse, a blocked -`pci-rebind` grabs the PCI device lock on its way in, which `pci-reset` also -needs: once a rebind has been attempted and is stuck, even FLR deadlocks and -**only a full PVE host power cycle recovers**. pci-reset first (if supported), and never -`pci-rebind` a D-state wedge. - -**If no** (device merely dead or silent), escalate gently: - -1. `authorized ` — re-enumerates just that device -2. `rebind ` — re-probe; also worth trying on the parent hub's busport -3. `hub-cycle ` — VBUS cycle of the feeding port, walking up to the - root port; may bounce sibling fixtures on ganged hubs -4. `pci-rebind ` — last resort: bounces every fixture on that controller - -## Finding targets - -```bash -grep -l /sys/bus/usb/devices/*/serial # serial -> busport (dir name) -readlink -f /sys/bus/usb/devices/usb # bus N -> its PCI addr in the path -``` - -Rig layout (2026-07-15, two Renesas uPD720201 cards; bus numbers renumber every -boot — re-derive with `readlink`): AMD `0000:02:00.0` = the debug-probe tree -(J-Links, ST-Links, WCH-Links), no port power switching; Renesas `0000:01:00.0` -and `0000:03:00.0` = DUT device hubs + serial fixtures, and ALL their root-hub -ports have real per-port power (`ppps`, 4+4 each) — `sudo uhubctl -l -p - -a cycle` cuts VBUS to the leaf hub on that port. The 1a40:0201 leaf -hubs themselves claim "ganged" switching but do not actually cut power. - -## Common mistakes - -- `resolve` takes a **/dev node**, not a busport or serial ("no such device node"). -- `authorized`/`rebind` take a **busport** (`3-4.7`); `pci-rebind`/`pci-reset` - take a **PCI addr**. -- Command produces no output and doesn't return → it is blocked on the device - lock: a D-state holder exists; see above. -- Trying `pci-rebind` on a D-state hang — its re-bind hangs and strands the - controller **driverless**; recover with `pci-bind `, or a PVE host power - cycle if the D-state URB is unkillable. Use `pci-reset` (if supported) for D-state, never - `pci-rebind`. -- Running `pci-reset` on a controller without FLR support (Renesas) → ENOTTY; - no software recovery — needs a PVE host power cycle. -- A J-Link reset (`r; go`) does not disconnect a wedged DUT from the host: the - DWC2 soft-connect pullup stays up through a core halt, so stuck URBs stay stuck. diff --git a/.claude/skills/usb-recover/scripts/usb_recover.sh b/.claude/skills/usb-recover/scripts/usb_recover.sh deleted file mode 100755 index 7652253fa..000000000 --- a/.claude/skills/usb-recover/scripts/usb_recover.sh +++ /dev/null @@ -1,140 +0,0 @@ -#!/usr/bin/env bash -# usb_recover.sh — USB recovery helper for the HIL rig; run with sudo. Writes only -# to the specific sysfs control files below; arg regexes block path traversal. -# -# Usage: -# sudo usb_recover.sh authorized # e.g. 3-2 -> deauthorize+reauthorize (re-enumerate, NO VBUS cut) -# sudo usb_recover.sh rebind # e.g. 3-2 -> usb driver unbind+bind (re-probe) -# sudo usb_recover.sh pci-rebind # e.g. 0000:01:00.0 -> HCD unbind+bind (WHOLE controller) -# sudo usb_recover.sh pci-reset # e.g. 0000:01:00.0 -> PCI function-level reset: kills URBs at -# # HW level WITHOUT the device lock; the only cure when a process -# # is stuck in D state (usbfs ioctl) and unbind paths would convoy -# sudo usb_recover.sh pci-bind [driver] # bind a DRIVERLESS controller (e.g. after a pci-rebind -# # whose re-bind hung and left it unbound). Auto-tries the xHCI -# # drivers (xhci-pci-renesas, xhci_hcd) unless one is named. -# sudo usb_recover.sh hub-cycle # e.g. 13-1.6 -> uhubctl power-cycle of the port feeding it, -# # walking upstream (parent hub -> root port) until the device -# # re-enumerates. Ganged/fake-switching hubs may bounce ALL -# # siblings; self-powered hubs only reset their uplink, which -# # is why the walk ends at the root port (real xHCI ppps). -# sudo usb_recover.sh resolve # e.g. /dev/ttyACM3 -> print its (no privilege needed) -set -euo pipefail - -USBPATH_RE='^[0-9]+-[0-9]+(\.[0-9]+)*$' -PCI_RE='^[0-9a-fA-F]{4}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}\.[0-9]$' -DRIVER_RE='^[A-Za-z0-9_-]+$' - -die() { echo "usb_recover: $*" >&2; exit 1; } -usage() { grep -E '^# sudo usb_recover' "$0" >&2; exit 2; } - -# Refuse to touch a PCI function that is not a USB controller (class 0x0c03xx), so a stray or -# mistyped BDF can't unbind/reset an unrelated device (storage, NIC) on a shared HIL host. -require_usb_controller() { - local addr=$1 cls - cls=$(cat "/sys/bus/pci/devices/$addr/class" 2>/dev/null) || die "no such pci device: $addr" - [[ "$cls" =~ ^0x0c03 ]] || die "$addr is not a USB controller (class $cls); refusing" -} - -# Resolve a /dev node (ttyACMx, ttyUSBx, sgN, ...) up to its USB device busport. -resolve() { - local node=$1 syspath dev - [ -e "$node" ] || die "no such device node: $node" - syspath=$(udevadm info -q path -n "$node" 2>/dev/null) || die "udevadm failed for $node" - dev="/sys$syspath" - while [ "$dev" != "/sys" ] && [ -n "$dev" ]; do - if [ -e "$dev/busnum" ] && [ -e "$dev/devnum" ] && [ -e "$dev/authorized" ]; then - basename "$dev"; return 0 - fi - dev=$(dirname "$dev") - done - die "could not find parent USB device for $node" -} - -action=${1:-}; target=${2:-} -[ -n "$action" ] && [ -n "$target" ] || usage - -case "$action" in - resolve) - resolve "$target" - ;; - authorized) - [[ "$target" =~ $USBPATH_RE ]] || die "bad usb path: $target" - d="/sys/bus/usb/devices/$target" - [ -e "$d/authorized" ] || die "no such usb device: $target" - echo 0 > "$d/authorized"; sleep 1; echo 1 > "$d/authorized" - echo "re-authorized $target" - ;; - rebind) - [[ "$target" =~ $USBPATH_RE ]] || die "bad usb path: $target" - [ -e "/sys/bus/usb/devices/$target" ] || die "no such usb device: $target" - echo "$target" > /sys/bus/usb/drivers/usb/unbind; sleep 1 - echo "$target" > /sys/bus/usb/drivers/usb/bind - echo "rebound $target" - ;; - pci-rebind) - [[ "$target" =~ $PCI_RE ]] || die "bad pci addr: $target" - require_usb_controller "$target" - [ -e "/sys/bus/pci/devices/$target/driver" ] || die "no driver bound to $target" - drv=$(basename "$(readlink -f "/sys/bus/pci/devices/$target/driver")") - echo "$target" > "/sys/bus/pci/drivers/$drv/unbind"; sleep 1 - echo "$target" > "/sys/bus/pci/drivers/$drv/bind" - echo "rebound pci $target ($drv)" - ;; - pci-bind) - # Re-attach a driver to a controller left DRIVERLESS (e.g. a pci-rebind whose re-bind hung). - [[ "$target" =~ $PCI_RE ]] || die "bad pci addr: $target" - require_usb_controller "$target" - [ -e "/sys/bus/pci/devices/$target" ] || die "no such pci device: $target" - [ -e "/sys/bus/pci/devices/$target/driver" ] && die "$target already has a driver bound" - drv=${3:-} - if [ -n "$drv" ]; then - [[ "$drv" =~ $DRIVER_RE ]] || die "bad driver name: $drv" - [ -e "/sys/bus/pci/drivers/$drv/bind" ] || die "no such pci driver: $drv" - echo "$target" > "/sys/bus/pci/drivers/$drv/bind" - echo "bound pci $target ($drv)" - else - # Auto-try the xHCI drivers (Renesas uPD720201 uses xhci-pci-renesas; others xhci_hcd). - for cand in xhci-pci-renesas xhci_hcd; do - [ -e "/sys/bus/pci/drivers/$cand/bind" ] || continue - if echo "$target" > "/sys/bus/pci/drivers/$cand/bind" 2>/dev/null; then - echo "bound pci $target ($cand)"; exit 0 - fi - done - die "could not bind $target with a known xHCI driver; pass the driver explicitly" - fi - ;; - hub-cycle) - [[ "$target" =~ $USBPATH_RE ]] || die "bad usb path: $target" - UHUBCTL=$(command -v uhubctl || echo /sbin/uhubctl) - [ -x "$UHUBCTL" ] || die "uhubctl not installed" - dev="$target" - while :; do - if [[ "$dev" =~ ^([0-9]+)-([0-9]+)$ ]]; then # parent is the root hub - loc="${BASH_REMATCH[1]}"; port="${BASH_REMATCH[2]}"; up="" - else # parent is a downstream hub - loc="${dev%.*}"; port="${dev##*.}"; up="$loc" - fi - echo "hub-cycle: power-cycling hub $loc port $port (feeds $dev)" - "$UHUBCTL" -l "$loc" -p "$port" -a cycle -d 5 -f || echo " (uhubctl failed at $loc; walking up)" - for _ in $(seq 1 10); do - sleep 1 - if [ -e "/sys/bus/usb/devices/$target/idVendor" ]; then - echo "recovered: $target re-enumerated"; exit 0 - fi - done - [ -n "$up" ] || break - dev="$up" - done - die "hub-cycle: $target still not enumerated after cycling up to the root port" - ;; - pci-reset) - [[ "$target" =~ $PCI_RE ]] || die "bad pci addr: $target" - require_usb_controller "$target" - [ -e "/sys/bus/pci/devices/$target/reset" ] || die "no reset support on $target" - echo 1 > "/sys/bus/pci/devices/$target/reset" - echo "flr-reset pci $target" - ;; - *) - usage - ;; -esac diff --git a/.claude/skills/usb-sniffer/SKILL.md b/.claude/skills/usb-sniffer/SKILL.md index 7c2cd2644..a20aecf05 100644 --- a/.claude/skills/usb-sniffer/SKILL.md +++ b/.claude/skills/usb-sniffer/SKILL.md @@ -1,6 +1,6 @@ --- name: usb-sniffer -description: Use when you need wire-level USB evidence that host-side capture can't provide — a device that never enumerates (usbmon shows nothing or only Submits), suspected NAK storms/STALL/babble/bad handshakes, bus-reset or enumeration timing, split-transaction issues, or a usbmon-vs-device-log disagreement the wire must arbitrate. Captures LS/FS/HS packets (PIDs, tokens, handshakes, SE0/line states) with the ataradov usb-sniffer hardware into Wireshark pcapng. +description: Use when you need wire-level USB evidence that host-side capture can't provide — a device that never enumerates (usbmon shows nothing or only Submits), suspected NAK storms/STALL/babble/bad handshakes, bus-reset or enumeration timing, split-transaction issues, a usbmon-vs-device-log disagreement the wire must arbitrate, or any link where TinyUSB is the host (no Linux PC host to run usbmon on). Captures LS/FS/HS packets (PIDs, tokens, handshakes, SE0/line states) with the ataradov usb-sniffer hardware into Wireshark pcapng. --- # usb-sniffer — wire-level capture with the ataradov hardware analyzer @@ -9,14 +9,16 @@ Extends the debugging trio with the layer below URBs: | Skill | Answers | |---|---| -| `usbmon` | what the host software exchanged (URBs) | -| `usb-debug` | why the host acted (dmesg / dynamic debug) | -| `usb-target-debug` | what the device firmware did | +| `usbmon` | what a Linux PC host exchanged (URBs) | +| `usb-kernel-debug` | why the Linux kernel acted (dmesg / dynamic debug) | +| `target-debug` | what the TinyUSB target did (device or host role) | | **`usb-sniffer`** | **what actually crossed D+/D-** (PIDs, handshakes, resets, timing) | Reach for it when usbmon can't see (device never binds, pre-enumeration -failures) or can't be trusted (URB completed but did the wire really ACK?). -For everything visible in URBs, usbmon is cheaper — no hardware, no locks. +failures), can't be trusted (URB completed but did the wire really ACK?), or +doesn't exist — a link where TinyUSB is the host has no usbmon on either end +(an MCU host runs no kernel; a Linux gadget peer's UDC bypasses usbmon). +Where a Linux PC is the host, usbmon is cheaper — no hardware, no locks. ## Rig inventory — find the sniffer and what it taps diff --git a/.claude/skills/usb-target-debug/SKILL.md b/.claude/skills/usb-target-debug/SKILL.md deleted file mode 100644 index 71fac98f2..000000000 --- a/.claude/skills/usb-target-debug/SKILL.md +++ /dev/null @@ -1,198 +0,0 @@ ---- -name: usb-target-debug -description: Use when a TinyUSB device misbehaves on real hardware and host-side capture can't explain it — a HIL test fails but usbmon shows only Submits with no Completes, the device silently NAKs, wedges, STALLs, babbles, or drops data, EP0 starves, an ISR or DCD/HCD state bug is suspected — and you need device-side evidence: TU_LOG/RTT logs, GDB state dumps, a RAM ring-buffer event trace, or PC-sampling of where the core spins. ---- - -# usb-target-debug — device-side capture & debugging on the HIL rig - -Completes the debugging trio (the `usb-sniffer` skill adds a fourth, -wire-level view when hardware tapping is available): - -| Skill | Answers | -|---|---| -| `usbmon` | what the host actually exchanged (URBs) | -| `usb-debug` | why the host acted (dmesg / dynamic debug) | -| **`usb-target-debug`** | **what the device did** (logs, driver state, PC) | -| `usb-sniffer` | what crossed the wire (PIDs, handshakes, resets — hardware tap) | - -For enumeration/transfer bugs the default posture is **dual-side capture** — -usbmon on the host *and* a target-side channel, simultaneously — not -host-first-then-escalate. - -## Rig discipline — lock first, always - -Hold the board lock for the WHOLE manual session; never stop the -actions-runner (see the `hil` skill for the full lock protocol): - -```bash -python3 test/hil/board_lock.py hold --reason "target debug: " -# ... instrument / build / flash / capture / GDB ... -python3 test/hil/board_lock.py release -``` - -Board → probe mapping: `test/hil/tinyusb.json` — `flasher.name` is the probe -family, `flasher.uid` the **probe serial** (many identical probes on the rig: -J-Link needs `-SelectEmuBySN ` / GDB server `-select usb=`; OpenOCD -`-c 'adapter serial '`). `JLINK_DEVICE` / `OPENOCD_OPTION` come from -`hw/bsp//boards//board.cmake` (or `board.mk`); find the family -with `ls -d hw/bsp/*/boards/`. Run on the host that owns the probe — -config is `test/hil/tinyusb.json` on ci, `local.json` on htpc (`hil` skill). - -## Pick the least intrusive technique that can answer the question - -Observation can mask the bug — the ch32v307 Heisenbug changed behavior under -logging *and* under the debugger. If the bug disappears when instrumented, -that IS a finding (timing-sensitive): move down in intrusiveness, not up. - -| Technique | Intrusiveness | Reach for it when | -|---|---|---| -| PC-sampling | none — no halt, no code change | core wedged/spinning somewhere unknown (rusb2 FRDY) | -| RAM ring-buffer | ~tens of cycles per event | ISR ordering/timing bugs (musb babble) | -| TU_LOG (RTT) | µs per line | logic bugs that survive logging | -| TU_LOG (UART) | ms per line — blocking write | same, when no J-Link on the board | -| GDB halt / breakpoints | stops USB service entirely | post-mortem state autopsy once wedged | - -## TU_LOG capture - -Build with `LOG=2` (`LOG=3` adds per-transfer noise and much more timing skew). -`LOGGER=rtt` routes it over the debug probe (J-Link only) — no UART wiring: - -```bash -# RTT: JLinkGDBServer from CLAUDE.md "GDB Debugging" + -RTTTelnetPort, then: -timeout 20s JLinkRTTClient > /tmp/rtt.log # non-interactive capture -# UART (board's debug serial, if wired): -stty -F /dev/ttyACM 115200 raw && timeout 20s cat /dev/ttyACM | tee /tmp/uart.log -``` - -An RTT-built firmware that has since wedged still holds a log tail in RAM — -but ONLY what fits the drain model: the default SEGGER mode (NO_BLOCK_SKIP) -**drops** writes once the ring fills with no reader, so an undrained target -holds the first KB after boot, not the wedge tail. There is no overwrite mode -in stock SEGGER RTT (only SKIP/TRIM/BLOCK): post-mortem RTT is evidence only -if a live drain was running — otherwise instrument with the RAM ring below. -Use `JLinkGDBServer -RTTTelnetPort 19021` + `JLinkRTTClient` for the drain -(proven; note the server briefly halts the core on connect). `JLinkRTTLogger` -fails to find the control block on some parts (LPC4088) even when it exists -and even given `-RTTAddress`; don't fight it — `nm` the ELF for `_SEGGER_RTT`, -read the aUp[0] descriptor (`mem32`), `savebin` the buffer — debug-AP RAM -reads don't halt the target. - -## GDB — state autopsy and watchpoints - -Connect/load recipes per probe family (J-Link, OpenOCD for ST-Link / -CMSIS-DAP / WCH-Link) are in CLAUDE.md "GDB Debugging". Release builds keep -DWARF (`MinSizeRel`), so `p`/struct access works on HIL firmware. - -**Autopsy of a wedged board: attach and halt ONLY** — skip CLAUDE.md's -`monitor reset halt` + `load` (those are for fresh starts; a reset destroys -the evidence). Symbolize with the ELF that is actually flashed — -`/cmake-build-//.elf` from the run that -wedged; do not rebuild while the wedge is still on the board. The debug-loop -specifics: - -```gdb -p/x _usbd_dev.ep_status # usbd core [epnum][dir] (1=IN): busy/stalled/claimed -p/x # per-port names — read the board's dcd_*.c first -x/32wx # raw EP/FIFO regs; base = the macro the dcd uses -watch xfer_status[2][1].total_len # HW watchpoint (Cortex-M: ~4); dwc2 names shown -break dcd_int_handler # works, but see warning below -``` - -While halted the device answers **nothing**: host control transfers time out -in ~5 s and the OS may reset/re-enumerate — after `continue`, the bus traffic -shows recovery, not the original bug. Prefer one halt for a post-mortem dump -over stepping through live USB traffic. - -## RAM ring-buffer trace - -The zero-print instrument (cracked the musb babble): a small event ring in the -dcd/hcd, dumped over GDB after the failure. Single-writer (ISR) — no locking: - -```c -typedef struct { uint16_t ev; uint16_t a; uint32_t b; } dbg_ev_t; -#define DBG_N 512 // power of two -static volatile dbg_ev_t dbg_ring[DBG_N]; // volatile REQUIRED: -Os dead-store- -static volatile uint32_t dbg_wr; // eliminates a write-only static array -static inline void DBG_EV(uint16_t ev, uint16_t a, uint32_t b) { - uint32_t i = dbg_wr++; - dbg_ring[i & (DBG_N - 1)] = (dbg_ev_t){ ev, a, b }; -} -// call sites: DBG_EV(__LINE__, ep_addr, count); — __LINE__ as event id -``` - -After building, `nm` the ELF for `dbg_ring`/`dbg_wr` — if they're missing the -compiler deleted your instrument and the run will "reproduce" with an empty ring. - -Order is the index; if durations matter add a `uint32_t t = DWT->CYCCNT` field -(enable once: `CoreDebug->DEMCR |= CoreDebug_DEMCR_TRCENA_Msk; DWT->CTRL |= 1;` -RISC-V: read `mcycle`). Let the failure happen, halt, then: - -```gdb -p dbg_wr # total events; oldest slot = dbg_wr & (DBG_N-1) once wrapped -p dbg_ring -dump binary memory /tmp/ring.bin &dbg_ring[0] &dbg_ring[512] -``` - -## PC-sampling (J-Link) — find where the core spins, without halting - -`DWT_PCSR` (0xE000101C) returns the current PC on every read, target running -(Cortex-M3+; optional on M0+, reads 0 if absent; 0xFFFFFFFF = core halted or -WFI-asleep — `mem32 E000EDF0, 1`, DHCSR bit 17 S_HALT, tells which). One -probe serves one client: quit JLinkExe before starting JLinkGDBServer on the -same probe. Nailed the rusb2 FRDY wedge: - -```bash -for i in $(seq 300); do echo 'mem32 E000101C, 1'; done \ - | JLinkExe -device $JLINK_DEVICE -SelectEmuBySN -if swd -speed 4000 -autoconnect 1 -nogui 1 \ - | awk '/E000101C = /{print $3}' | sort | uniq -c | sort -rn | head -arm-none-eabi-addr2line -e -f -a 0x ... # PCs → functions -``` - -OpenOCD variant: repeat `mdw 0xE000101C` over telnet :4444. The histogram's -top entries are the spin site; a flat histogram = core is servicing normally. - -## Dual-side capture — the default for enumeration/transfer bugs - -Start both channels, then trigger the failing test: - -```bash -.claude/skills/usbmon/scripts/usbcap.sh cafe: 30 /tmp/host.pcapng & # host URBs (usbmon skill) -timeout 30s JLinkRTTClient > /tmp/target.rtt & # target (or ring dump after) -wait -``` - -RTT lines and ring events carry no wall-clock: correlate on unambiguous -anchors — bus reset, SET_ADDRESS, the first transfer on the failing EP — then -lay device events between anchors in host-URB order. Logging the SOF/frame -number on the target gives a shared clock when you need finer alignment. -When host and target evidence disagree, or the host sees nothing at all, add -the wire itself: `usb-sniffer` skill (hardware tap, PID-level). - -## Warnings - -- **Halting/resetting via the probe does NOT disconnect the device**: a DWC2 - soft-connect pullup stays up through core halt *and* reset, so the host's - stuck URBs stay stuck and a wedged DUT stays wedged — recover the host side - with the `usb-recover` skill. -- **A bug that vanishes under LOG=2 is a timing bug**, not fixed: switch to - the ring buffer; if it vanishes under GDB too, PC-sampling only. -- **UART TU_LOG blocks in the write path** (worst perturbation, including - inside the ISR); RTT is much cheaper but not free; `LOG=3` multiplies both. -- Flash/GDB only with the board lock held; a `hold` refused with reason - `hil_test.py` means CI is mid-test on that board — wait, don't force. -- **Instrumentation is temporary**: before `release`, reflash pristine - firmware (the next CI run must not inherit a debug build) and revert the - instrumentation diff — or hand it over explicitly with the diagnosis. -- **A register snapshot without a validity anchor lies**: J-Link tool sessions - can reset or briefly halt the DUT as a side effect, and a snapshot of a - freshly-reset chip (e.g. NVIC ISER = 0) reads like a smoking gun. Read DHCSR - (0xE000EDF0: bit 17 S_HALT, bit 25 S_RESET_ST) with every snapshot, and - cross-check against something the device demonstrably still does. -- **A marginal link can fake a deterministic firmware bug** — down to failing - the same test at the same iteration twice. "USB disconnect" in dmesg on a - freshly re-cabled port (high devnum = churn) means the plug, not the code: - first sustained bulk traffic is when a bad contact drops. Before declaring a - regression, re-run the OLD build on the SAME link state — and if a bisect - exonerates every hunk, believe it: re-test the exact failing binary. -- **Release your manual lock before `hil_test.py`** — it self-locks each board - and fails immediately on your own hold (`hil` skill). diff --git a/.claude/skills/usbmon/SKILL.md b/.claude/skills/usbmon/SKILL.md index 85ec33248..a2e7c1ac1 100644 --- a/.claude/skills/usbmon/SKILL.md +++ b/.claude/skills/usbmon/SKILL.md @@ -1,11 +1,11 @@ --- name: usbmon -description: Use when capturing, analyzing, or debugging USB bus traffic for TinyUSB device development on Linux — enumeration failures, STALLed control transfers, missing/short bulk or interrupt transfers, isochronous/audio dropouts, or descriptor problems. Captures host-side URBs with usbmon + tshark into a Wireshark pcapng and decodes them. Use whenever you need to see what the host actually exchanged with a device on real hardware, even if the user just says "sniff USB", "capture the enumeration", or "why won't my device enumerate". +description: Use when capturing, analyzing, or debugging USB bus traffic on a link where a Linux PC is the host (TinyUSB in device role) — enumeration failures, STALLed control transfers, missing/short bulk or interrupt transfers, isochronous/audio dropouts, or descriptor problems. Captures host-side URBs with usbmon + tshark into a Wireshark pcapng and decodes them. Not applicable when TinyUSB is the host — no URBs traverse the PC (use usb-sniffer / target-debug). Use whenever you need to see what the Linux host actually exchanged with a device on real hardware, even if the user just says "sniff USB", "capture the enumeration", or "why won't my device enumerate". --- # usbmon — capture & debug USB traffic -`usbmon` records host-side **URBs** — control / bulk / interrupt / isochronous transfers, descriptors, class requests, STALLs, short packets — i.e. exactly what the host exchanged with a device. Use it to debug a TinyUSB device on real hardware. (It's host/URB-level, not wire-level; for SOF/ACK/electrical use a hardware analyzer.) +`usbmon` records host-side **URBs** — control / bulk / interrupt / isochronous transfers, descriptors, class requests, STALLs, short packets — i.e. exactly what the host exchanged with a device. Use it to debug a TinyUSB device on real hardware. (It's host/URB-level, not wire-level; for SOF/ACK/electrical use a hardware analyzer.) It exists only on the Linux host side of a link: when TinyUSB runs the *host* stack (peer = another TinyUSB board or a Linux gadget, e.g. a Raspberry Pi), neither end has usbmon — capture the wire (`usb-sniffer` skill) or instrument the target (`target-debug` skill). **Setup (assumed in place):** `usbmon` loaded and a udev rule `SUBSYSTEM=="usbmon", GROUP="wireshark", MODE="0640"` with your user in the `wireshark` group — so `tshark` captures with no `sudo`. Freshly added to the group? The running shell doesn't have it yet (group adds need a new login) — wrap captures in `sg wireshark -c 'tshark -i usbmon3 -s 128 -a duration:30 -w /tmp/cap.pcapng'`; reading a finished `.pcapng` (`tshark -r`) needs no group. `-s 128` (snaplen) keeps only URB headers/status, not payloads — use it for long/high-throughput captures. diff --git a/.claude/skills/usbtest/SKILL.md b/.claude/skills/usbtest/SKILL.md index 8146d94e4..850781d6e 100644 --- a/.claude/skills/usbtest/SKILL.md +++ b/.claude/skills/usbtest/SKILL.md @@ -1,6 +1,6 @@ --- name: usbtest -description: Use when running, debugging, or porting the Linux usbtest/testusb battery (examples/device/usbtest, cafe:4010) — device "did not bind", SET_CONFIGURATION fails, a case fails with errno 110/32/5/71, toggle-clear/halt/unlink/iso failures, iso packets dropped, or a new MCU/DCD needs the full 30/30 sign-off. +description: Use when running, debugging, or porting the Linux usbtest/testusb battery (examples/device/usbtest, cafe:4010) — device "did not bind", SET_CONFIGURATION fails, a case fails with errno 110/32/5/71, toggle-clear/halt/unlink/iso failures, iso packets dropped, or a new MCU/DCD needs the full 30/30 sign-off. Needs a Linux PC as the link's host driving TinyUSB in device role — it exercises the DCD, not the TinyUSB host stack. --- # usbtest — porting & debugging the Linux kernel USB battery diff --git a/.claude/workflows/hil-validate.js b/.claude/workflows/hil-validate.js index aa0556abc..50559135f 100644 --- a/.claude/workflows/hil-validate.js +++ b/.claude/workflows/hil-validate.js @@ -52,7 +52,7 @@ if (!args.force) { } const wedged = results.filter(r => r.wedged).map(r => r.board) -if (wedged.length) log(`WEDGED boards needing usb-recover: ${wedged.join(', ')}`) +if (wedged.length) log(`WEDGED boards needing usb-kernel-recover: ${wedged.join(', ')}`) // Workers cannot prompt the user — surface still-locked boards for the main // session to ask: force (re-invoke with force: true), wait, or accept. const locked = args.force ? [] : results.filter(r => !r.pass && r.detail.startsWith('board locked')).map(r => r.board) diff --git a/CLAUDE.md b/CLAUDE.md index 2acdc3a63..77dab4565 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -116,7 +116,7 @@ Cutting a release — version bump, regenerated files, the per-release changelog ## References -- MCU reference manuals, datasheets, schematics: before answering register/bitfield/pinout/errata/timing questions from memory or the web, use the `read-doc` skill (`.claude/skills/read-doc/SKILL.md`) to search and read them from `$HOME/Documents/calibre-library` (skill no-ops if the library is absent). +- MCU reference manuals, datasheets, schematics: before answering register/bitfield/pinout/errata/timing questions from memory or the web — or changing a specific dcd/hcd driver — use the `read-doc` skill (`.claude/skills/read-doc/SKILL.md`) to cross-check against docs in `$HOME/Documents/calibre-library`; tell the user if the needed document is missing (skill no-ops if the library is absent). - Supported MCUs/boards: `hw/bsp/` and `docs/reference/boards.rst`. - USB classes: `src/class/{cdc,hid,msc,audio,…}/` — each has `*_device.c` and `*_host.c`. - Key files: `src/tusb.h`, `src/tusb_config.h`, `tools/get_deps.py`, `tools/build.py`, `test/unit-test/project.yml`. diff --git a/test/hil/usbtest.py b/test/hil/usbtest.py index a2841f0b6..e17705a48 100755 --- a/test/hil/usbtest.py +++ b/test/hil/usbtest.py @@ -38,7 +38,7 @@ PID = '4010' GZ_REF = '0525 a4a0' # copy Gadget Zero's capability profile (ctrl_out+iso+intr) SYS_USB = Path('/sys/bus/usb/devices') DRIVER = Path('/sys/bus/usb/drivers/usbtest') -USB_RECOVER = Path(__file__).resolve().parents[2] / '.claude/skills/usb-recover/scripts/usb_recover.sh' +USB_RECOVER = Path(__file__).resolve().parents[2] / '.claude/skills/usb-kernel-recover/scripts/usb_recover.sh' PATTERN_PARAM = Path('/sys/module/usbtest/parameters/pattern') # Battery per tier, in run order: control sanity first, then simple bulk, @@ -391,7 +391,7 @@ def main(): if pci: print(f'aborting battery: kernel-side hang, device wedged mid-transfer.\n' f'auto-recovering: sudo {USB_RECOVER} pci-reset {pci} ' - f'(see .claude/skills/usb-recover)', file=sys.stderr) + f'(see .claude/skills/usb-kernel-recover)', file=sys.stderr) # FLR frees the D-state ioctl without the device lock; must run BEFORE # any unbind/remove_id, which would deadlock the bus otherwise if sudo([str(USB_RECOVER), 'pci-reset', pci]).returncode != 0: @@ -418,7 +418,7 @@ def main(): try: if unrecovered_hang: # testusb is still stuck in a usbfs ioctl holding the device lock; remove_id/unbind - # would join the convoy and deadlock the bus (see usb-recover skill) — leave it be + # would join the convoy and deadlock the bus (see usb-kernel-recover skill) — leave it be print('skipping cleanup after unrecovered hang: reboot required to release the bus', file=sys.stderr) elif not args.keep_binding: -- cgit v1.3.1