skills: usbtest/usb-debug/usb-recover docs, hil runner notes

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HeF2gZ1M7GWkz6Av4BpKPg

6 files changed, 421 insertions, 0 deletions

diff --git a/.claude/skills/hil/SKILL.md b/.claude/skills/hil/SKILL.md
index a7a916907..18e59c060 100644
--- a/.claude/skills/hil/SKILL.md
+++ b/.claude/skills/hil/SKILL.md
@@ -14,6 +14,20 @@ Run TinyUSB HIL tests on real boards. **Run `hostname` first** — it tells you
 
 Default to **local**. Use **remote** only when on `htpc` and the user says `remote`/`ci.lan`. Never attempt remote on `ci`.
 
+## Stop the CI runner first (on `ci`)
+
+The `ci` rig also hosts a GitHub Actions runner that flashes boards and runs HIL as part of CI. If it fires while you are driving the hardware yourself — any HIL run, flashing, `test/hil/usbtest.py`, GDB, raw USB — it reflashes boards mid-test and churns the bus, producing spurious failures and even wedged devices.
+
+**Before touching hardware on `ci`, stop the runner; restart it when done.** `svc.sh` is run with `sudo` but must be run **from the runner root** (`~/actions-runner`, plural), else it errors "Must run from runner root":
+
+```bash
+(cd ~/actions-runner && sudo ./svc.sh stop)     # before any hardware/HIL action
+# ... flash / run hil_test.py / usbtest.py / GDB ...
+(cd ~/actions-runner && sudo ./svc.sh start)    # ALWAYS restart when finished
+```
+
+Treat the restart as mandatory cleanup — leaving the runner stopped silently disables CI for the whole repo. Only applies on `ci` (htpc has no runner). Check state with `(cd ~/actions-runner && sudo ./svc.sh status)`.
+
 ## Prerequisites
 
 Examples must be built for the target board(s) — see AGENTS.md "Build" → "All examples for a board" (produces `examples/cmake-build-<board>/`). `-B examples` points `hil_test.py` at that parent folder.
diff --git a/.claude/skills/usb-debug/SKILL.md b/.claude/skills/usb-debug/SKILL.md
new file mode 100644
index 000000000..20ab7d764
--- /dev/null
+++ b/.claude/skills/usb-debug/SKILL.md
@@ -0,0 +1,36 @@
+---
+name: usb-debug
+description: Use when USB enumeration fails or misbehaves and usbmon alone can't explain WHY the host acted — port reset storms, repeated re-enumeration, address errors, xHCI ring/command errors, "device descriptor read error", babble, or when you need the host driver's own reasoning from dmesg on the ci HIL rig.
+---
+
+# usb-debug — host-side kernel dynamic debug for USB
+
+usbmon shows the URBs; kernel **dynamic debug** shows the host driver's
+*reasoning* usbmon can't: port resets and their causes, enumeration retries,
+address (re)assignment, EP halts, xHCI ring/command errors.
+
+Run this skill's `scripts/usb_dyndbg.sh` with `sudo` (abbreviated to
+`usb_dyndbg.sh` in the examples below). It flips the dynamic-debug print flag
+for an allowlisted set of USB host modules only:
+
+```bash
+sudo usb_dyndbg.sh on  usbcore xhci_hcd   # enable +p; pick modules from `lsusb -t` Driver=
+sudo usb_dyndbg.sh status [module]        # list enabled print sites
+sudo usb_dyndbg.sh off usbcore xhci_hcd   # ALWAYS turn off when done — very noisy
+```
+
+Allowlisted modules: `usbcore xhci_hcd xhci_pci xhci_pci_renesas ehci_hcd
+ehci_pci ohci_hcd ohci_pci uhci_hcd dwc2 cdc_acm usb_storage uas`.
+
+## Workflow
+
+1. `sudo usb_dyndbg.sh on usbcore <hcd-module>` — `usbcore` for enumeration/hub
+   logic, plus the controller module (`lsusb -t` shows the driver per bus).
+2. Reproduce (replug / re-enumerate / rerun the failing test) while following
+   `sudo dmesg -w` (or grab `sudo dmesg | tail` afterwards).
+3. `sudo usb_dyndbg.sh off ...` — leaving it on floods the log and skews timing.
+
+Pair with the `usbmon` skill: usbmon for what crossed the bus, dynamic debug for
+why the host reacted. For a wedged device/bus use the `usb-recover` skill.
+
+Requires `CONFIG_DYNAMIC_DEBUG` and mounted debugfs (standard on distro kernels).
diff --git a/.claude/skills/usb-debug/scripts/usb_dyndbg.sh b/.claude/skills/usb-debug/scripts/usb_dyndbg.sh
new file mode 100755
index 000000000..0dc880469
--- /dev/null
+++ b/.claude/skills/usb-debug/scripts/usb_dyndbg.sh
@@ -0,0 +1,46 @@
+#!/usr/bin/env bash
+# usb_dyndbg.sh — toggle kernel dynamic-debug on USB host drivers; run with sudo.
+# Flips +p/-p only on an allowlisted set of USB modules, so it can't reach
+# arbitrary kernel debug or unrelated subsystems.
+#
+# Usage:
+#   sudo usb_dyndbg.sh on  <module>...   # enable  +p  (e.g. usbcore xhci_hcd)
+#   sudo usb_dyndbg.sh off <module>...   # disable -p
+#   sudo usb_dyndbg.sh status [module]   # show enabled sites (or one module's sites)
+set -euo pipefail
+
+CTL=/sys/kernel/debug/dynamic_debug/control
+# Allowlist: USB host-controller + core + common host class drivers.
+ALLOW='usbcore xhci_hcd xhci_pci xhci_pci_renesas ehci_hcd ehci_pci ohci_hcd ohci_pci uhci_hcd dwc2 cdc_acm usb_storage uas'
+
+die() { echo "usb_dyndbg: $*" >&2; exit 1; }
+usage() {
+  echo "usage: usb_dyndbg.sh {on|off} <module>...   modules: $ALLOW" >&2
+  echo "       usb_dyndbg.sh status [module]" >&2
+  exit 2
+}
+allowed() { local m; for m in $ALLOW; do [ "$m" = "$1" ] && return 0; done; return 1; }
+
+[ -e "$CTL" ] || die "dynamic_debug unavailable (need CONFIG_DYNAMIC_DEBUG + debugfs mounted)"
+
+action=${1:-}; shift || true
+case "$action" in
+  on|off)
+    [ "$#" -ge 1 ] || usage
+    flag='+p'; [ "$action" = off ] && flag='-p'
+    for m in "$@"; do allowed "$m" || die "module not allowlisted: $m"; done
+    for m in "$@"; do echo "module $m $flag" > "$CTL"; echo "dynamic debug $action: $m"; done
+    ;;
+  status)
+    m=${1:-}
+    if [ -n "$m" ]; then
+      allowed "$m" || die "module not allowlisted: $m"
+      grep -E "\[$m\]" "$CTL" || echo "(no sites for $m)"
+    else
+      grep -E '=p( |$)' "$CTL" || echo "(no print sites enabled)"
+    fi
+    ;;
+  *)
+    usage
+    ;;
+esac
diff --git a/.claude/skills/usb-recover/SKILL.md b/.claude/skills/usb-recover/SKILL.md
new file mode 100644
index 000000000..7f3e86632
--- /dev/null
+++ b/.claude/skills/usb-recover/SKILL.md
@@ -0,0 +1,91 @@
+---
+name: usb-recover
+description: Use when a USB device or fixture on the ci HIL rig is stuck, hung, not enumerating, or wedged after a failed flash or test, or when processes touching USB (testusb, JLinkExe, uhubctl, libusb tools) start hanging in D state.
+---
+
+# USB Recovery on the HIL Rig
+
+Run this skill's `scripts/usb_recover.sh` with `sudo` (abbreviated to
+`usb_recover.sh` in the examples below). It wraps four sysfs reset actions plus
+a resolver:
+
+```bash
+sudo usb_recover.sh resolve    /dev/ttyACM3    # /dev node -> busport (e.g. 3-4.7); also ttyUSB*, sg*
+sudo usb_recover.sh authorized <busport>       # deauthorize+reauthorize: re-enumerate, no VBUS cut
+sudo usb_recover.sh rebind     <busport>       # usb driver unbind+bind: re-probe
+sudo usb_recover.sh pci-rebind <pciaddr>       # whole HCD controller unbind+bind, e.g. 0000:02:00.0
+sudo usb_recover.sh pci-reset  <pciaddr>       # PCI function-level reset: kills URBs at HW level, no device lock
+sudo usb_recover.sh pci-bind   <pciaddr> [drv] # re-bind a DRIVERLESS controller (auto-tries xHCI drivers)
+```
+
+## Decide first: is anything stuck in D state?
+
+```bash
+ps -eo pid,stat,wchan:30,cmd | awk '$2 ~ /D/'
+```
+
+**If yes** (uninterruptible sleep, typically a usbfs ioctl — e.g. testusb inside
+`usb_sg_wait`): run `pci-reset` and NOTHING ELSE first:
+
+```bash
+sudo usb_recover.sh pci-reset <pciaddr>
+```
+
+FLR kills the URBs at the hardware level without taking the per-device lock;
+the ioctl then returns and the convoy unwinds on its own.
+
+**Not every controller supports FLR.** The Renesas uPD720201 (`0000:01:00.0`)
+has no reset method — `pci-reset` fails with `Inappropriate ioctl for device`
+(ENOTTY). On those, there is no clean D-state cure short of a **reboot**; do NOT
+fall through to `pci-rebind` (see next).
+
+**`pci-rebind` can strand the controller driverless.** Its unbind succeeds but,
+with a D-state process still holding a URB, the *re-bind* hangs — leaving the
+PCI device with **no driver** (`/sys/bus/pci/devices/<addr>/driver` gone) and the
+whole controller's fixtures offline. A second `pci-rebind` then dies with "no
+driver bound". Recover with `pci-bind <addr>` (re-attaches the xHCI driver);
+if that also hangs because the D-state URB is unkillable, **reboot** is the only
+cure. The Renesas binds via `xhci-pci-renesas` (firmware loader), others via
+`xhci_hcd` — `pci-bind` auto-tries both, or pass the driver explicitly.
+
+**Ordering is critical.** `authorized`/`rebind`/`pci-rebind` all take the
+per-device lock the stuck ioctl holds — they block and join the convoy, and
+soon every libusb tool (uhubctl, JLinkExe) hangs too. Worse, a blocked
+`pci-rebind` grabs the PCI device lock on its way in, which `pci-reset` also
+needs: once a rebind has been attempted and is stuck, even FLR deadlocks and
+**only a rig reboot recovers**. pci-reset first (if supported), and never
+`pci-rebind` a D-state wedge.
+
+**If no** (device merely dead or silent), escalate gently:
+
+1. `authorized <busport>` — re-enumerates just that device
+2. `rebind <busport>` — re-probe; also worth trying on the parent hub's busport
+3. `pci-rebind <pciaddr>` — last resort: bounces every fixture on that controller
+
+## Finding targets
+
+```bash
+grep -l <SERIAL> /sys/bus/usb/devices/*/serial          # serial -> busport (dir name)
+readlink -f /sys/bus/usb/devices/usb<N>                  # bus N -> its PCI addr in the path
+```
+
+Rig layout: buses 3+4 = `0000:02:00.0` (main fixture tree: J-Links, ST-Links,
+WCH-Links, DUTs); buses 9+12 = `0000:01:00.0`, the only ones with uhubctl port
+power (ganged VBUS: `sudo uhubctl -l 9 -a cycle`). Hubs on buses 1-4 have no
+port power switching — uhubctl reports "No compatible devices" there.
+
+## Common mistakes
+
+- `resolve` takes a **/dev node**, not a busport or serial ("no such device node").
+- `authorized`/`rebind` take a **busport** (`3-4.7`); `pci-rebind`/`pci-reset`
+  take a **PCI addr**.
+- Command produces no output and doesn't return → it is blocked on the device
+  lock: a D-state holder exists; see above.
+- Trying `pci-rebind` on a D-state hang — its re-bind hangs and strands the
+  controller **driverless**; recover with `pci-bind <addr>`, or reboot if the
+  D-state URB is unkillable. Use `pci-reset` (if supported) for D-state, never
+  `pci-rebind`.
+- Running `pci-reset` on a controller without FLR support (Renesas) → ENOTTY;
+  no recovery but reboot.
+- A J-Link reset (`r; go`) does not disconnect a wedged DUT from the host: the
+  DWC2 soft-connect pullup stays up through a core halt, so stuck URBs stay stuck.
diff --git a/.claude/skills/usb-recover/scripts/usb_recover.sh b/.claude/skills/usb-recover/scripts/usb_recover.sh
new file mode 100755
index 000000000..35bd4c784
--- /dev/null
+++ b/.claude/skills/usb-recover/scripts/usb_recover.sh
@@ -0,0 +1,111 @@
+#!/usr/bin/env bash
+# usb_recover.sh — USB recovery helper for the HIL rig; run with sudo. Writes only
+# to the specific sysfs control files below; arg regexes block path traversal.
+#
+# Usage:
+#   sudo usb_recover.sh authorized <busport>   # e.g. 3-2  -> deauthorize+reauthorize (re-enumerate, NO VBUS cut)
+#   sudo usb_recover.sh rebind     <busport>   # e.g. 3-2  -> usb driver unbind+bind (re-probe)
+#   sudo usb_recover.sh pci-rebind <pciaddr>   # e.g. 0000:01:00.0 -> HCD unbind+bind (WHOLE controller)
+#   sudo usb_recover.sh pci-reset  <pciaddr>   # e.g. 0000:01:00.0 -> PCI function-level reset: kills URBs at
+#                                              # HW level WITHOUT the device lock; the only cure when a process
+#                                              # is stuck in D state (usbfs ioctl) and unbind paths would convoy
+#   sudo usb_recover.sh pci-bind   <pciaddr> [driver]  # bind a DRIVERLESS controller (e.g. after a pci-rebind
+#                                              # whose re-bind hung and left it unbound). Auto-tries the xHCI
+#                                              # drivers (xhci-pci-renesas, xhci_hcd) unless one is named.
+#   sudo usb_recover.sh resolve    <devnode>   # e.g. /dev/ttyACM3 -> print its <busport> (no privilege needed)
+set -euo pipefail
+
+USBPATH_RE='^[0-9]+-[0-9]+(\.[0-9]+)*$'
+PCI_RE='^[0-9a-fA-F]{4}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}\.[0-9]$'
+DRIVER_RE='^[A-Za-z0-9_-]+$'
+
+die() { echo "usb_recover: $*" >&2; exit 1; }
+usage() { grep -E '^#   sudo usb_recover' "$0" >&2; exit 2; }
+
+# Refuse to touch a PCI function that is not a USB controller (class 0x0c03xx), so a stray or
+# mistyped BDF can't unbind/reset an unrelated device (storage, NIC) on a shared HIL host.
+require_usb_controller() {
+  local addr=$1 cls
+  cls=$(cat "/sys/bus/pci/devices/$addr/class" 2>/dev/null) || die "no such pci device: $addr"
+  [[ "$cls" =~ ^0x0c03 ]] || die "$addr is not a USB controller (class $cls); refusing"
+}
+
+# Resolve a /dev node (ttyACMx, ttyUSBx, sgN, ...) up to its USB device busport.
+resolve() {
+  local node=$1 syspath dev
+  [ -e "$node" ] || die "no such device node: $node"
+  syspath=$(udevadm info -q path -n "$node" 2>/dev/null) || die "udevadm failed for $node"
+  dev="/sys$syspath"
+  while [ "$dev" != "/sys" ] && [ -n "$dev" ]; do
+    if [ -e "$dev/busnum" ] && [ -e "$dev/devnum" ] && [ -e "$dev/authorized" ]; then
+      basename "$dev"; return 0
+    fi
+    dev=$(dirname "$dev")
+  done
+  die "could not find parent USB device for $node"
+}
+
+action=${1:-}; target=${2:-}
+[ -n "$action" ] && [ -n "$target" ] || usage
+
+case "$action" in
+  resolve)
+    resolve "$target"
+    ;;
+  authorized)
+    [[ "$target" =~ $USBPATH_RE ]] || die "bad usb path: $target"
+    d="/sys/bus/usb/devices/$target"
+    [ -e "$d/authorized" ] || die "no such usb device: $target"
+    echo 0 > "$d/authorized"; sleep 1; echo 1 > "$d/authorized"
+    echo "re-authorized $target"
+    ;;
+  rebind)
+    [[ "$target" =~ $USBPATH_RE ]] || die "bad usb path: $target"
+    [ -e "/sys/bus/usb/devices/$target" ] || die "no such usb device: $target"
+    echo "$target" > /sys/bus/usb/drivers/usb/unbind; sleep 1
+    echo "$target" > /sys/bus/usb/drivers/usb/bind
+    echo "rebound $target"
+    ;;
+  pci-rebind)
+    [[ "$target" =~ $PCI_RE ]] || die "bad pci addr: $target"
+    require_usb_controller "$target"
+    [ -e "/sys/bus/pci/devices/$target/driver" ] || die "no driver bound to $target"
+    drv=$(basename "$(readlink -f "/sys/bus/pci/devices/$target/driver")")
+    echo "$target" > "/sys/bus/pci/drivers/$drv/unbind"; sleep 1
+    echo "$target" > "/sys/bus/pci/drivers/$drv/bind"
+    echo "rebound pci $target ($drv)"
+    ;;
+  pci-bind)
+    # Re-attach a driver to a controller left DRIVERLESS (e.g. a pci-rebind whose re-bind hung).
+    [[ "$target" =~ $PCI_RE ]] || die "bad pci addr: $target"
+    require_usb_controller "$target"
+    [ -e "/sys/bus/pci/devices/$target" ] || die "no such pci device: $target"
+    [ -e "/sys/bus/pci/devices/$target/driver" ] && die "$target already has a driver bound"
+    drv=${3:-}
+    if [ -n "$drv" ]; then
+      [[ "$drv" =~ $DRIVER_RE ]] || die "bad driver name: $drv"
+      [ -e "/sys/bus/pci/drivers/$drv/bind" ] || die "no such pci driver: $drv"
+      echo "$target" > "/sys/bus/pci/drivers/$drv/bind"
+      echo "bound pci $target ($drv)"
+    else
+      # Auto-try the xHCI drivers (Renesas uPD720201 uses xhci-pci-renesas; others xhci_hcd).
+      for cand in xhci-pci-renesas xhci_hcd; do
+        [ -e "/sys/bus/pci/drivers/$cand/bind" ] || continue
+        if echo "$target" > "/sys/bus/pci/drivers/$cand/bind" 2>/dev/null; then
+          echo "bound pci $target ($cand)"; exit 0
+        fi
+      done
+      die "could not bind $target with a known xHCI driver; pass the driver explicitly"
+    fi
+    ;;
+  pci-reset)
+    [[ "$target" =~ $PCI_RE ]] || die "bad pci addr: $target"
+    require_usb_controller "$target"
+    [ -e "/sys/bus/pci/devices/$target/reset" ] || die "no reset support on $target"
+    echo 1 > "/sys/bus/pci/devices/$target/reset"
+    echo "flr-reset pci $target"
+    ;;
+  *)
+    usage
+    ;;
+esac
diff --git a/.claude/skills/usbtest/SKILL.md b/.claude/skills/usbtest/SKILL.md
new file mode 100644
index 000000000..8146d94e4
--- /dev/null
+++ b/.claude/skills/usbtest/SKILL.md
@@ -0,0 +1,123 @@
+---
+name: usbtest
+description: Use when running, debugging, or porting the Linux usbtest/testusb battery (examples/device/usbtest, cafe:4010) — device "did not bind", SET_CONFIGURATION fails, a case fails with errno 110/32/5/71, toggle-clear/halt/unlink/iso failures, iso packets dropped, or a new MCU/DCD needs the full 30/30 sign-off.
+---
+
+# usbtest — porting & debugging the Linux kernel USB battery
+
+## Overview
+
+`examples/device/usbtest` is the device-side peer of the Linux kernel's `usbtest.ko`/`testusb`
+(gadget-zero source/sink protocol): 30 cases over bulk, EP0, interrupt, and isochronous, including
+halt, data-toggle, and unlink storms. It is the most adversarial exerciser a DCD gets — every port
+so far surfaced at least one real driver bug. Host runner: `test/hil/usbtest.py`; HIL integration
+runs it per board and reports `✅ 30/30` cells.
+
+**Core principle: the battery is a DCD test, not a firmware test.** When a case fails, suspect the
+DCD path it exercises (table below), reproduce that one case, and root-cause on hardware before
+changing anything (`superpowers:systematic-debugging`). One variable at a time; a fix is proven by
+the failing case passing *and* the full battery still at 30/30 across reflash cycles.
+
+## Run
+
+```bash
+# build (cmake); descriptor sizes auto-adapt per MCU via src/usb_descriptors.h + src/tusb_config.h
+cd examples/device/usbtest && cmake -B build -DBOARD=<board> -G Ninja -DCMAKE_BUILD_TYPE=MinSizeRel && cmake --build build
+# flash, wait ~3-5 s for enumeration to settle, then:
+python3 test/hil/usbtest.py --serial <uid> --keep-binding            # full battery for the advertised tier
+python3 test/hil/usbtest.py --serial <uid> --keep-binding --tests 29 # one case
+```
+
+- **Always `--keep-binding`**: the cleanup unbind path has wedged host xHCIs (`usb_hcd_alloc_bandwidth`).
+- Always settle a few seconds after flashing — enumeration can bounce once; testusb into the gap sees
+  the device drop mid-case.
+- On a CI rig: stop the actions runner before touching hardware; restart after. Never run two
+  batteries concurrently (hil_test.py serializes them; concurrent batteries have hard-frozen a rig
+  via a fatal PCIe error on a VFIO-passed xHCI).
+
+## Porting ladder — new MCU/DCD to 30/30
+
+1. **Tier 1 (bulk)**: set `USBTEST_TIER 1`, get enumeration + cases 0,9,10 (EP0) + 1–8,17–20,27,28
+   solid. EP0 correctness first — everything else reports through it.
+2. **Tier 2 (ctrl_out 14/21)**, **tier 3 (interrupt 25/26)**, **tier 4 (iso 15/16/22/23)** — raise
+   the tier only when the layer below is clean; run the *full* battery after each layer.
+3. **Fit the endpoints**: tier 4 needs 6 endpoints + EP0. Small parts need per-MCU mps/epbuf
+   overrides in `src/usb_descriptors.h` (`USBTEST_INT/ISO_EP_MPS_FS`) and `src/tusb_config.h`
+   (`CFG_TUD_VENDOR_TX_EPSIZE`) — follow the existing CH32/LPC11 patterns. Parts that can't fit go
+   in `skip.txt`.
+4. **Sign-off = reliability, not one pass**: 3–10 full flash→battery cycles. One 30/30 proves
+   nothing on a flaky bring-up; deterministic partial counts (e.g. exactly 1-in-8 lost) are a
+   signature, not noise — chase them.
+5. Register the board in `test/hil/tinyusb.json` so the HIL suite runs it.
+
+## Case → DCD subsystem map
+
+| Failing case(s) | Exercises | First suspect |
+|---|---|---|
+| 9, 10 | EP0 control storms | EP0 state machine, ZLP/status stage, control starvation under load |
+| 1–8, 17–20, 27, 28 | bulk source/sink, sg, perf | FIFO handling, multi-packet, ZLP tolerance |
+| 11, 12, 24 | URB unlink mid-transfer | abort/close paths leaving state half-armed |
+| 13 | set/clear halt | stall must kill the transfer; halt on armed IN must flush the TX FIFO |
+| **29** | clear-halt on an **armed, un-halted** ep | **the classic**: `dcd_edpt_clear_stall` resets toggle but disarms the queued receive → NAKs forever, errno 110. Fix: reset toggle to DATA0 *and* re-arm/preserve the pending transfer. Found independently on rp2040, fsdev, ch32_usbhs, rusb2 |
+| 14, 21 | vendor EP0 write/readback | multi-packet control-OUT chunking, DCP flow control |
+| 25, 26 | interrupt src/sink | usually free once bulk works |
+| 15, 16, 22, 23 | isochronous | see iso rules below |
+
+## Iso rules (most-violated contract)
+
+- **DATA0-only in BOTH directions** at FS — never run bulk-style toggle logic on an iso endpoint
+  (manual-toggle parts: skip the ISR toggle flip for iso IN *and* the toggle-mismatch drop for iso
+  OUT). Symptom of violating it: exactly every-other packet lost.
+- **No handshake** — iso never NAKs/STALLs; parts with response fields use their "no response"
+  encoding (e.g. NYET on WCH).
+- `dcd_edpt_iso_alloc`/`iso_activate` **must not be stubs returning false** — usbd fails the
+  interface open and the kernel logs "did not bind"/SET_CONFIG times out. If a DCD refuses iso
+  "because the hardware can't", **verify against the datasheet — the manual outranks the code
+  comment** (two "no iso support" claims in this tree were false, incl. a per-endpoint exception
+  the RM documents for one endpoint number only).
+- A multi-packet iso IN submit is legal: the DCD streams it one packet per frame, refilling in the
+  ISR. Slow cores may need double-buffered iso to make the frame deadline.
+
+## Debug ladder (escalate in order)
+
+| errno | Meaning |
+|---|---|
+| 110 | timeout — endpoint NAKing forever / device wedged |
+| 32 | EPIPE — unexpected STALL |
+| 5 | EIO — iso packet errors (check `dmesg`: "N errors out of M") |
+| 71 | EPROTO — device answered wrong / too slow (after HC retries) |
+
+1. `usbtest.py` per-case output + its captured `dmesg` (`TEST n` markers bracket each case).
+2. **usbmon** (`usbmon` skill): URB-level ground truth. **It cannot show data toggles or NAKs** —
+   a toggle desync and a dead endpoint look identical (Submits without Completes); distinguish
+   device-side with GDB.
+3. **On-device gdb/openocd**: read the EP control registers and DCD structs at the hang.
+4. Heisenbugs (vanish under logging): RAM ring-buffer trace dumped over openocd; for silent lockups
+   JLink PC-sampling (`halt`+`regs` repeatedly — a pinned PC names the spin).
+5. **Cross-check the reference manual** (calibre library) before changing any register-level code —
+   per CLAUDE.md, and because comments/assumptions in DCDs have been wrong about hardware caps.
+6. Check the vendor's **silicon errata** early for timing/DMA hangs (an unimplemented erratum
+   workaround caused a case-10 hang on one port).
+
+## Traps that pass gcc/desk review but fail elsewhere
+
+- `TUD_OPT_HIGH_SPEED` is a **compile-time capability, not the live speed**: the FS config
+  descriptor (and OTHER_SPEED) must use FS-legal sizes (int ≤ 64, iso IN+OUT ≤ 1023 B/frame) even
+  on HS builds — use separate `_FS`/`_HS` descriptor macros.
+- Unused `static inline` helpers: clang `-Wunused-function` and IAR `Pe177` error where gcc stays
+  quiet → `TU_ATTR_UNUSED`.
+- A symbol referenced only inside naked asm is invisible to LTO and gets dropped in `-flto` make
+  builds → keep a `TU_ATTR_USED` C reference to it.
+- Nested USB IRQs on cores with hardware context stacks (QingKe HWSTK): plain
+  `__attribute__((interrupt))` corrupts the return — use naked handlers relying on the HW stack.
+- Dedicated USB RAM budgets (PMA/USB-RAM) differ per part *and* per build system section placement:
+  check the link map, not just that it builds.
+
+## Red flags — stop and re-examine
+
+- "One pass = done" → run reflash cycles.
+- "The DCD comment says the hardware can't" → open the datasheet.
+- "usbmon shows no toggle problem" → usbmon can't see toggles.
+- "It works on gcc" → clang/IAR/LTO/make still pending.
+- "Fixed iso IN" → apply the same exemption to iso OUT (toggle logic is symmetric).
+- A clean single-board run does not validate concurrent/fleet behavior — batteries serialize.