diff options
| author | hathach <[email protected]> | 2026-07-24 14:55:59 +0700 |
|---|---|---|
| committer | hathach <[email protected]> | 2026-07-24 14:55:59 +0700 |
| commit | 21bbcb5bbff18454979a1dc9aaa210e938a004b5 (patch) | |
| tree | d5953da04f7038e80ef8734eac8facbfbbbbec1f /.claude | |
| parent | b1becd8f5fcaf4a8a07aaa76fa68d3e9f7bb18a3 (diff) | |
docs(target-debug): vector catch, SWO trace, verifybin, FreeRTOS threads; table integration
- Vector catch + Cortex-M fault autopsy, verified with a deliberate bad-load
on stm32f407disco: CFSR=0x8200 (BFARVALID|PRECISERR), BFAR = exact bad
address, stacked pc addr2lined to the faulting line; gotchas recorded
(stale FPB comparators fire phantom SIGTRAPs — scrub first; arm DEMCR
after reset; loads precise / stores imprecise; ARMv6-M has no CFSR/BFAR)
- SWO exception trace + hw PC sampling gate PASSED on F407: 680 KB of
packets in 3 s (0x17 PC samples in flash range, 0x0E SysTick enter/exit);
JLinkSWOViewerCL decodes stimulus only — raw SWORead is the recipe;
SWOStart needs an explicit speed headless
- verifybin 'Verify successful.'; FreeRTOS -rtos plugin lists all 6
cdc_msc_freertos tasks after a run->stop cycle (plain attach = 0xDEAD
placeholder); semihosting anti-note; monitor-mode pointer (untested)
- Intrusiveness table gains the new rows; agent playbook bullet updated;
retrieval gate 5/5 with a fresh reader; executed plan committed
Diffstat (limited to '.claude')
| -rw-r--r-- | .claude/agents/target-debugger.md | 3 | ||||
| -rw-r--r-- | .claude/skills/target-debug/SKILL.md | 176 |
2 files changed, 117 insertions, 62 deletions
diff --git a/.claude/agents/target-debugger.md b/.claude/agents/target-debugger.md index 861600428..4d874e38c 100644 --- a/.claude/agents/target-debugger.md +++ b/.claude/agents/target-debugger.md @@ -13,7 +13,8 @@ truth; read the relevant SKILL.md BEFORE acting: - `.claude/skills/target-debug/SKILL.md` — your primary playbook: technique choice by intrusiveness, channel choice by link topology, capture recipes, - GDB autopsy, all rig warnings. + breakpoint/watchpoint budget and cost model, vector catch + fault autopsy, + SWO trace, GDB autopsy, all rig warnings. - `.claude/skills/hil/SKILL.md` — host/config selection, board lock protocol, `hil_test.py` invocation. - `.claude/skills/usbmon/SKILL.md` — Linux-host URB capture; exists only when a diff --git a/.claude/skills/target-debug/SKILL.md b/.claude/skills/target-debug/SKILL.md index 21dea38db..d0ceef2a9 100644 --- a/.claude/skills/target-debug/SKILL.md +++ b/.claude/skills/target-debug/SKILL.md @@ -18,11 +18,10 @@ Raspberry Pi). Pick capture channels by which end runs Linux, not by habit: | `usb-sniffer` | what crossed the wire (PIDs, handshakes, resets) | hardware tap cabled in — role-agnostic | For enumeration/transfer bugs the default posture is **dual-side capture** — -both ends simultaneously, not one-side-first-then-escalate: usbmon plus a -target channel when a Linux PC is the host. When TinyUSB is the host there is -no usbmon on either end — pair the target channel with the wire -(`usb-sniffer`) and, if the peer is a Linux gadget, `usb-kernel-debug` on the -peer. +both ends simultaneously: usbmon + a target +channel when a Linux PC is the host; TinyUSB-as-host has no usbmon on either +end — target channel + the wire (`usb-sniffer`), plus `usb-kernel-debug` on a +Linux gadget peer. ## Rig discipline — lock first, always @@ -52,9 +51,12 @@ that IS a finding (timing-sensitive): move down in intrusiveness, not up. | Technique | Intrusiveness | Reach for it when | |---|---|---| | PC-sampling | none — no halt, no code change | core wedged/spinning somewhere unknown (rusb2 FRDY) | +| SWO exception trace / hw PC-sample | none — needs SWO pin wired | ISR ordering/timing with zero code change | +| Vector catch | none until a fault fires | crash-shaped wedges — autopsy AT the faulting pc | | RAM ring-buffer | ~tens of cycles per event | ISR ordering/timing bugs (musb babble) | -| TU_LOG (RTT) | µs per line | logic bugs that survive logging | -| TU_LOG (UART) | ms per line — blocking write | same, when no J-Link on the board | +| TU_LOG (RTT) | µs per line | logic bugs that survive logging (J-Link or OpenOCD rtt) | +| TU_LOG (UART) | ms per line — blocking write | same, when no debug-probe RTT path | +| dprintf / conditional breakpoint | halt+resume per hit (~ms) | low-rate probes post-wedge; never ISR-rate events | | GDB halt / breakpoints | stops USB service entirely | post-mortem state autopsy once wedged | ## TU_LOG capture @@ -73,15 +75,14 @@ stty -F /dev/ttyACM<N> 115200 raw && timeout 20s cat /dev/ttyACM<N> | tee /tmp/u ```bash # OpenOCD RTT (any probe OpenOCD drives) — in telnet :4444 (or -c equivalents): -rtt setup 0x20000000 0x8000 "SEGGER RTT" # search range = RAM ORIGIN + LENGTH (from the .ld / map file) +rtt setup 0x20000000 0x8000 "SEGGER RTT" # RAM ORIGIN + LENGTH (from the .ld/map) rtt start # after firmware booted; rerun after each reflash rtt server start 19021 0 # then: timeout 20s nc localhost 19021 > /tmp/rtt.log ``` -OpenOCD polls the buffer: bursty logs can drop lines a J-Link would keep — -prefer J-Link where both exist; the drain-model warning below applies -unchanged. +OpenOCD polls — bursty logs can drop lines; prefer J-Link where both +exist. The drain-model warning below applies unchanged. An RTT-built firmware that has since wedged still holds a log tail in RAM — but ONLY what fits the drain model: the default SEGGER mode (NO_BLOCK_SKIP) @@ -99,10 +100,8 @@ reads don't halt the target. ## GDB — state autopsy and watchpoints Connect/load recipes per probe family (J-Link, OpenOCD for ST-Link / -CMSIS-DAP / WCH-Link) are in CLAUDE.md "GDB Debugging". For scripted/batch -sessions add `-singlerun` to JLinkGDBServer — the server exits with the -connection; back-to-back server relaunches race the probe handle and hang at -startup. Release builds keep DWARF (`MinSizeRel`), so `p`/struct access works +CMSIS-DAP / WCH-Link) are in CLAUDE.md "GDB Debugging". Scripted sessions: JLinkGDBServer `-singlerun` (exits with the +connection) — back-to-back relaunches race the probe handle and hang. Release builds keep DWARF (`MinSizeRel`), so `p`/struct access works on HIL firmware. **Autopsy of a wedged board: attach and halt ONLY** — skip CLAUDE.md's @@ -121,55 +120,87 @@ watch xfer_status[2][1].total_len # HW watchpoint (Cortex-M: ~4); dwc2 names break dcd_int_handler # works, but see warning below ``` -**Hardware budget — read it off the chip, not from memory** (verified: F407/M4 -= 6 bp + 4 wp, rp2040/M0+ = 4 + 2; M7 typically 8/4): +**Hardware budget — read it off the chip** (verified: F407/M4 = 6 bp + 4 wp, +rp2040/M0+ = 4 + 2; M7 typically 8/4): ```gdb p ((*(unsigned*)0xE0002000)>>4) & 0xF # FPB NUM_CODE = hw breakpoints (M7 adds bits[14:12]) p (*(unsigned*)0xE0001000)>>28 # DWT_CTRL NUMCOMP = watchpoint comparators ``` -- `hbreak`/`thbreak` force a hardware breakpoint (code in flash can't take a - software break unless the probe does flash breakpoints — J-Link does, - OpenOCD needs `bp <addr> 2 hw`); `tbreak` = one-shot. -- `watch -l <expr>` watches the *address* the expression evaluates to once — - cheap and what you almost always want; `rwatch`/`awatch` trap reads/any - access (hardware-only — they error rather than fall back). OpenOCD (telnet - :4444) adds a data-VALUE match GDB cannot express: `wp <addr> 4 w <value> - [mask]` — fires only when the written value matches (e.g. catch who writes - 0 into a busy flag, ignoring writes of 1). +- `hbreak`/`thbreak` force a hardware breakpoint (software breaks in flash + need flash-breakpoint support — J-Link has it; OpenOCD: `bp <addr> 2 hw`); + `tbreak` = one-shot. +- `watch -l <expr>` watches the address expr evaluates to once — almost + always what you want; `rwatch`/`awatch` trap reads/any access (hardware- + only — they error, never fall back). OpenOCD adds a data-VALUE match GDB + can't express: `wp <addr> 4 w <value> [mask]` — catch who writes 0 into a + busy flag, ignoring writes of 1. - **Demand the word "Hardware" in the confirmation.** `watch` silently falls - back to a SOFTWARE watchpoint when no DWT comparator fits (expression too - wide/complex, budget exhausted): GDB then single-steps the whole program — - hundreds of times slower, certain USB death. `Watchpoint 2:` without - "Hardware" = delete it; narrowing the expression (`watch -l`, cast to a - 4-byte int) is the real fix. -- Conditional breaks/watches (`break dcd_edpt_xfer if ep_addr==0x81`) are - evaluated by GDB on the HOST with our stubs — neither JLinkGDBServer nor - OpenOCD supports target-side agent expressions on Cortex-M — so every hit - is a halt+resume (~ms) whether the condition matches or not: fine - post-wedge or on cold paths, wrong under live USB traffic. -- `commands <bpnum> ... end` auto-runs GDB commands at each hit (start with - `silent`, end with `continue` for hands-free evidence collection) — same - halt-per-hit cost. -- `dprintf <loc>,"fmt",args` = printf without recompiling. Stay on the - default `dprintf-style gdb` (host prints): the `call` style runs the - target's own printf mid-halt and `agent` needs stub support — neither is - viable on these probes. Same cost model as conditional breaks; for - ISR-rate events use the RAM ring buffer instead. -- Stepping while the USB ISR fires between every step is chaos: OpenOCD - `cortex_m maskisr steponly` masks interrupts during single-steps only. - The bus keeps running either way — the host may still reset a device that - stops responding mid-step. -- While halted you can poke state to test a hypothesis (`set var - _usbd_dev.ep_status[2][1].busy = 0`) — but that invalidates the snapshot - as post-mortem evidence; dump first, poke after. + back to a software watchpoint when no DWT comparator fits — GDB then + single-steps the whole program, hundreds of times slower: certain USB + death. Plain `Watchpoint 2:` = delete it and narrow the expression + (`watch -l`, cast to a 4-byte int). +- Conditional breaks (`break ... if ep_addr==0x81`) and `dprintf + <loc>,"fmt",args` (printf without recompiling; keep `dprintf-style gdb`) + are host-evaluated — no Cortex-M agent expressions in our stubs: every hit + halts+resumes (~ms) even when the condition is false. Post-wedge/cold + paths only; ISR-rate events belong in the RAM ring. +- `commands <bpnum> ... end` (start `silent`, end `continue`) auto-collects + evidence per hit — same halt-per-hit cost. +- Stepping with the USB ISR firing between steps is chaos: OpenOCD + `cortex_m maskisr steponly`. The bus runs either way — the host may still + reset a halted-looking device. +- Poking state while halted (`set var _usbd_dev.ep_status[2][1].busy = 0`) + tests a hypothesis but invalidates the post-mortem — dump first, poke after. +- FreeRTOS examples: `-rtos GDBServer/RTOSPlugin_FreeRTOS` (OpenOCD: `-rtos + FreeRTOS`) → `info threads` lists every task with state/prio/frame + (verified: 6 tasks). It populates only after a + run→stop cycle — plain attach shows one 0xDEAD placeholder. Semihosting is + never the answer (traps + halts per call — RTT instead). **Monitor-mode + debugging** (J-Link, M3+) keeps chosen IRQs serviced at a breakpoint — + needs SEGGER's JLINK_MONITOR files + `SetMonModeDebug=1`; not set up here: + <https://kb.segger.com/Monitor_Mode_Debugging> (untested). While halted the device answers **nothing**: host control transfers time out in ~5 s and the OS may reset/re-enumerate — after `continue`, the bus traffic shows recovery, not the original bug. Prefer one halt for a post-mortem dump over stepping through live USB traffic. +## Vector catch + fault autopsy — catch the crash, not the wedge + +A wedge that is really a fault (HardFault loop, lockup) autopsies best AT +the faulting instruction. Two hardware-proven gotchas: **FPB/DWT comparators +survive reflash and dead sessions** — a stale one fires as a phantom SIGTRAP +at an unrelated line of NEW firmware — and J-Link's reset strategy manages +vector-catch bits: scrub first, arm AFTER reset: + +```gdb +# scrub: FP_COMP0..5 = 0xE0002008..201C, DWT_FUNCTIONn = 0xE0001028 + n*0x10 +set *(unsigned*)0xE0002008 = 0 +# ... (repeat per comparator; count from the budget reads above) +# arm (after monitor reset; tool-agnostic — works via JLinkExe w4 too): +set *(unsigned*)0xE000EDFC |= (1<<10)|(1<<9)|(1<<8)|(1<<7)|(1<<6)|(1<<5)|(1<<4) +# = VC_HARDERR|INTERR|BUSERR|STATERR|CHKERR|NOCPERR|MMERR; bit0 VC_CORERESET halts at reset +``` + +OpenOCD native: `cortex_m vector_catch hard_err bus_err state_err chk_err mm_err`. +It halts at exception ENTRY (pc = handler, LR = EXC_RETURN 0xFFFFFFFx); decode: + +```gdb +p/x *(unsigned*)0xE000ED28 # CFSR — low byte MemManage, byte1 BusFault, top half UsageFault +p/x *(unsigned*)0xE000ED2C # HFSR — bit30 FORCED = an escalated lower-priority fault +p/x *(unsigned*)0xE000ED38 # BFAR — faulting address (valid if CFSR bit15 BFARVALID) +x/8wx $msp # stacked frame: r0 r1 r2 r3 r12 lr pc xpsr — pc = culprit +``` + +`addr2line -e <elf> <stacked pc>` names the line (verified: CFSR 0x8200, +BFAR = the bad address, stacked pc = the faulting ldr). Loads fault +precisely; stores usually IMPRECISERR (BFAR invalid, pc late). ARMv6-M has no +CFSR/BFAR, only VC_HARDERR|VC_CORERESET — stacked frame alone. Still a halt +(host URB timeouts apply); clear DEMCR (`&= ~0x7F0`) before handing back; +RISC-V: breakpoint the trap handler; mcause/mepc/mtval are the CFSR/BFAR analogs. + ## RAM ring-buffer trace The zero-print instrument (cracked the musb babble): a small event ring in the @@ -218,11 +249,31 @@ arm-none-eabi-addr2line -e <firmware.elf> -f -a 0x<hot-pc> ... # PCs → funct OpenOCD variant: repeat `mdw 0xE000101C` over telnet :4444. The histogram's top entries are the spin site; a flat histogram = core is servicing normally. +### SWO — hardware-timed trace on one pin (J-Link; verified on F407) + +If SWO (TRACESWO) is wired, DWT emits packets with ZERO code change: +**exception trace** (DWT_CTRL bit16 — every IRQ enter/exit, timestamped) and +**hardware PC sampling** (bit12), better histograms than DWT_PCSR polling. +SWOViewer tools decode only ITM *stimulus* (TinyUSB emits none) — capture +raw: + +```bash +# JLinkExe -CommandFile: +w4 E0001000, 0x00011401 # EXCTRCENA|PCSAMPLENA|SYNCTAP|CYCCNTENA +SWOStart 4000000 # explicit speed — autodetect fails headless +Sleep 3000 +SWORead # hex: 0x17+4B LE = PC sample, 0x0E+2B = IRQ enter/exit +``` + +Verified: 680 KB in 3 s (flash-range PC samples + SysTick enter/exit). +SWORead stuck at 0 = SWO pin not wired (many boards route only SWDIO/SWCLK). +Restore DWT_CTRL when done. + ## Dual-side capture — the default for enumeration/transfer bugs -Start both channels, then trigger the failing test (Linux-PC-host link shown; -TinyUSB-as-host: swap the usbmon line for a `usb-sniffer` capture, plus -`usb-kernel-debug` on the peer if it is a Linux gadget): +Start both channels, then trigger the failing test (Linux-PC-host shown; +TinyUSB-as-host: swap usbmon for `usb-sniffer`, + `usb-kernel-debug` on a +Linux gadget peer): ```bash .claude/skills/usbmon/scripts/usbcap.sh cafe: 30 /tmp/host.pcapng & # host URBs (usbmon skill) @@ -239,14 +290,13 @@ the wire itself: `usb-sniffer` skill (hardware tap, PID-level). ## Manuals -- J-Link / J-Trace User Guide (UM08001): <https://kb.segger.com/UM08001_J-Link_/_J-Trace_User_Guide> — flash breakpoints, RTT, SWO, monitor mode, Commander commands. -- OpenOCD User's Guide: <https://openocd.org/doc/html/index.html> — `rtt`, `bp`/`wp`, `cortex_m vector_catch` / `maskisr`, `itm`/`tpiu`. -- "Debugging with GDB" (the official manual; §5.1 covers break/watch/dprintf): - calibre library first (`read-doc` skill) — use the **Tenth Edition (GDB 18)** - copy, not the 2002 Ninth-Edition txt also present; fallback +- J-Link (UM08001): <https://kb.segger.com/UM08001_J-Link_/_J-Trace_User_Guide> — flash breakpoints, RTT, SWO, monitor mode, Commander. +- OpenOCD: <https://openocd.org/doc/html/index.html> — `rtt`, `bp`/`wp`, `cortex_m vector_catch`/`maskisr`, `itm`/`tpiu`. +- "Debugging with GDB" (§5.1 = break/watch/dprintf): Tenth Edition (GDB 18) + via calibre/`read-doc` — NOT the 2002 Ninth-Edition txt also there — or `curl -sL -o /tmp/gdb.pdf https://sourceware.org/gdb/current/onlinedocs/gdb.pdf` - (the HTML mirror blocks fetchers; the PDF works). The installed - `arm-none-eabi-gdb`'s `help <cmd>` is authoritative for what this rig runs. + (the HTML mirror blocks fetchers). Installed `arm-none-eabi-gdb` + `help <cmd>` is authoritative here. ## Warnings @@ -268,6 +318,10 @@ the wire itself: `usb-sniffer` skill (hardware tap, PID-level). freshly-reset chip (e.g. NVIC ISER = 0) reads like a smoking gun. Read DHCSR (0xE000EDF0: bit 17 S_HALT, bit 25 S_RESET_ST) with every snapshot, and cross-check against something the device demonstrably still does. +- **"Flash OK" can lie** (silent no-op — old firmware keeps running). When + behavior contradicts the flashed code: `objcopy -O binary fw.elf + /tmp/fw.bin`, then `verifybin /tmp/fw.bin,<flash-base>` (J-Link, verified) + or `verify_image` (OpenOCD); on mismatch reflash before debugging further. - **A marginal link can fake a deterministic firmware bug** — down to failing the same test at the same iteration twice. "USB disconnect" in dmesg on a freshly re-cabled port (high devnum = churn) means the plug, not the code: |
