diff options
| -rw-r--r-- | .claude/skills/usb-kernel-debug/SKILL.md | 6 | ||||
| -rw-r--r-- | .claude/skills/usb-kernel-recover/SKILL.md | 73 | ||||
| -rwxr-xr-x | .claude/skills/usb-kernel-recover/scripts/usb_recover.sh | 89 | ||||
| -rwxr-xr-x | test/hil/usbtest.py | 124 |
4 files changed, 236 insertions, 56 deletions
diff --git a/.claude/skills/usb-kernel-debug/SKILL.md b/.claude/skills/usb-kernel-debug/SKILL.md index e4169b049..fd291c0ac 100644 --- a/.claude/skills/usb-kernel-debug/SKILL.md +++ b/.claude/skills/usb-kernel-debug/SKILL.md @@ -12,11 +12,11 @@ sits in the link — the rig PC when it is the host, or a Linux gadget peer (dwc2/UDC + gadget modules) when TinyUSB is the host. It cannot see inside the TinyUSB MCU — that is the `target-debug` skill. -Run this skill's `scripts/usb_dyndbg.sh` with `sudo` (abbreviated to -`usb_dyndbg.sh` in the examples below). It flips the dynamic-debug print flag -for an allowlisted set of USB modules only: +Run this skill's `scripts/usb_dyndbg.sh` with `sudo`. It flips the dynamic-debug +print flag for an allowlisted set of USB modules only: ```bash +# all examples below abbreviate: sudo .claude/skills/usb-kernel-debug/scripts/usb_dyndbg.sh sudo usb_dyndbg.sh on usbcore xhci_hcd # enable +p; pick modules from `lsusb -t` Driver= sudo usb_dyndbg.sh status [module] # list enabled print sites sudo usb_dyndbg.sh off usbcore xhci_hcd # ALWAYS turn off when done — very noisy diff --git a/.claude/skills/usb-kernel-recover/SKILL.md b/.claude/skills/usb-kernel-recover/SKILL.md index 3f03722fe..9e456417f 100644 --- a/.claude/skills/usb-kernel-recover/SKILL.md +++ b/.claude/skills/usb-kernel-recover/SKILL.md @@ -5,18 +5,20 @@ description: Use when a USB device or fixture attached to the ci HIL rig's Linux # USB Recovery on the HIL Rig (Linux kernel side) -Run this skill's `scripts/usb_recover.sh` with `sudo` (abbreviated to -`usb_recover.sh` in the examples below). It wraps the sysfs reset actions, a -uhubctl power-cycle escalator, and a resolver: +Run this skill's `scripts/usb_recover.sh` with `sudo`. It wraps the sysfs reset +actions, a uhubctl power-cycle escalator, and a resolver: ```bash +# all examples below abbreviate: sudo .claude/skills/usb-kernel-recover/scripts/usb_recover.sh sudo usb_recover.sh resolve /dev/ttyACM3 # /dev node -> busport (e.g. 3-4.7); also ttyUSB*, sg* sudo usb_recover.sh authorized <busport> # deauthorize+reauthorize: re-enumerate, no VBUS cut sudo usb_recover.sh rebind <busport> # usb driver unbind+bind: re-probe sudo usb_recover.sh hub-cycle <busport> # uhubctl VBUS cycle of the feeding port, walking parent hub # -> root port until the device re-enumerates +sudo usb_recover.sh root-cycle <busport> [serial] # uhubctl VBUS cut straight at the ROOT port (real ppps), no + # leaf walk, no device-lock touch: the D-state cure. + # [serial] is checked and a mismatch refused. sudo usb_recover.sh pci-rebind <pciaddr> # whole HCD controller unbind+bind, e.g. 0000:02:00.0 -sudo usb_recover.sh pci-reset <pciaddr> # PCI function-level reset: kills URBs at HW level, no device lock sudo usb_recover.sh pci-bind <pciaddr> [drv] # re-bind a DRIVERLESS controller (auto-tries xHCI drivers) ``` @@ -34,21 +36,45 @@ ps -eo pid,stat,wchan:30,cmd | awk '$2 ~ /D/' ``` **If yes** (uninterruptible sleep, typically a usbfs ioctl — e.g. testusb inside -`usb_sg_wait`): run `pci-reset` and NOTHING ELSE first: +`usb_sg_wait`): cut VBUS at the root port, and nothing else. ```bash -sudo usb_recover.sh pci-reset <pciaddr> +sudo usb_recover.sh root-cycle <busport> # e.g. 11-3.7 -> cycles bus 11 root port 3 ``` -FLR kills the URBs at the hardware level without taking the per-device lock; -the ioctl then returns and the convoy unwinds on its own. +This drops power to the wedged device, so its in-flight URB fails and the ioctl +returns. It targets the *root hub* — a different USB device from the wedged one — +and never *writes* the wedged device's sysfs. It reads a few attributes from it — +`idVendor`/`idProduct`/`serial`/`product` to report and check the target, and the +directory inode plus `devnum` afterwards — none of which take the device lock, so +it does not join the convoy the way `authorized`/`rebind`/`pci-rebind` do. +Recovery is proven by that inode changing — a real disconnect destroys the +kobject and reconnecting creates a new one, whereas a disconnect blocked on the +device lock leaves it untouched. It exits non-zero if the device does not come +back; a **zero exit only means it re-enumerated**, so still confirm the D-state +process actually let go. Pass the expected serial as a third argument and it +refuses a busport that now names a different device. -**Not every controller supports FLR.** The Renesas uPD720201 (`0000:01:00.0`) -has no reset method — `pci-reset` fails with `Inappropriate ioctl for device` -(ENOTTY). On those, there is no clean software D-state cure — a VM reboot is NOT +It bounces **every fixture under that root port** — on ci that is up to 25 +devices. Hold the affected boards' locks first if you can, but note +`board_lock.py` uses `LOCK_EX | LOCK_NB` and so fails immediately when CI already +holds them; there is no wait-for-lock. When CI is mid-run you are choosing +between bouncing its fixtures and leaving the bus wedged for everything. The +automated path in `usbtest.py` takes no locks at all and accepts that collateral +deliberately: by the time a D-state wedge exists the convoy will take the bus +down anyway. + +(The VBUS mechanism is verified on the ci rig — the leaf hubs report +`bmAttributes=e0`, "self-powered", but are physically bus-powered with no adapter, +so a root-port cut really does kill downstream power. Do not re-derive this from +the descriptor; it lies. Not yet confirmed against a live D-state wedge. If +`uhubctl` itself hangs, the convoy has already spread — escalate.) + +If `root-cycle` does not free the D-state process, there is no software cure +left: ask the operator for a full PVE **host** power cycle. A VM reboot is NOT reliable (downstream hubs can latch up across the PCIe reset and need a physical -replug); ask the operator for a full PVE host power cycle instead. Do NOT -fall through to `pci-rebind` (see next). +replug), and a graceful reboot stalls on the D-state process anyway. Do NOT fall +through to `pci-rebind` (see next). **`pci-rebind` can strand the controller driverless.** Its unbind succeeds but, with a D-state process still holding a URB, the *re-bind* hangs — leaving the @@ -62,10 +88,9 @@ power cycle (operator action) recovers. The Renesas binds via `xhci-pci-renesas` **Ordering is critical.** `authorized`/`rebind`/`pci-rebind` all take the per-device lock the stuck ioctl holds — they block and join the convoy, and soon every libusb tool (uhubctl, JLinkExe) hangs too. Worse, a blocked -`pci-rebind` grabs the PCI device lock on its way in, which `pci-reset` also -needs: once a rebind has been attempted and is stuck, even FLR deadlocks and -**only a full PVE host power cycle recovers**. pci-reset first (if supported), and never -`pci-rebind` a D-state wedge. +`pci-rebind` grabs the PCI device lock on its way in and can wedge the whole +function, after which **only a full PVE host power cycle recovers**. `root-cycle` +first, and never `pci-rebind` a D-state wedge. **If no** (device merely dead or silent), escalate gently: @@ -93,15 +118,19 @@ hubs themselves claim "ganged" switching but do not actually cut power. ## Common mistakes - `resolve` takes a **/dev node**, not a busport or serial ("no such device node"). -- `authorized`/`rebind` take a **busport** (`3-4.7`); `pci-rebind`/`pci-reset` - take a **PCI addr**. +- `authorized`/`rebind`/`hub-cycle`/`root-cycle` take a **busport** (`3-4.7`); + `pci-rebind`/`pci-bind` take a **PCI addr**. - Command produces no output and doesn't return → it is blocked on the device lock: a D-state holder exists; see above. - Trying `pci-rebind` on a D-state hang — its re-bind hangs and strands the controller **driverless**; recover with `pci-bind <addr>`, or a PVE host power - cycle if the D-state URB is unkillable. Use `pci-reset` (if supported) for D-state, never + cycle if the D-state URB is unkillable. Use `root-cycle` for D-state, never `pci-rebind`. -- Running `pci-reset` on a controller without FLR support (Renesas) → ENOTTY; - no software recovery — needs a PVE host power cycle. +- Writing `/sys/bus/pci/devices/<addr>/reset` because the attribute is there. No + rig controller has FLR, so it becomes a PCIe bus reset that resets the xHCI + behind its live driver — the write succeeds, the card is halted for good, and + only a PVE host power cycle brings it back. Use `root-cycle`. +- `root-cycle` bounces **every** fixture under that root port, not just the target + — hold the sibling boards' locks first. - A J-Link reset (`r; go`) does not disconnect a wedged DUT from the host: the DWC2 soft-connect pullup stays up through a core halt, so stuck URBs stay stuck. diff --git a/.claude/skills/usb-kernel-recover/scripts/usb_recover.sh b/.claude/skills/usb-kernel-recover/scripts/usb_recover.sh index 7652253fa..2230602b9 100755 --- a/.claude/skills/usb-kernel-recover/scripts/usb_recover.sh +++ b/.claude/skills/usb-kernel-recover/scripts/usb_recover.sh @@ -6,9 +6,6 @@ # sudo usb_recover.sh authorized <busport> # e.g. 3-2 -> deauthorize+reauthorize (re-enumerate, NO VBUS cut) # sudo usb_recover.sh rebind <busport> # e.g. 3-2 -> usb driver unbind+bind (re-probe) # sudo usb_recover.sh pci-rebind <pciaddr> # e.g. 0000:01:00.0 -> HCD unbind+bind (WHOLE controller) -# sudo usb_recover.sh pci-reset <pciaddr> # e.g. 0000:01:00.0 -> PCI function-level reset: kills URBs at -# # HW level WITHOUT the device lock; the only cure when a process -# # is stuck in D state (usbfs ioctl) and unbind paths would convoy # sudo usb_recover.sh pci-bind <pciaddr> [driver] # bind a DRIVERLESS controller (e.g. after a pci-rebind # # whose re-bind hung and left it unbound). Auto-tries the xHCI # # drivers (xhci-pci-renesas, xhci_hcd) unless one is named. @@ -17,6 +14,11 @@ # # re-enumerates. Ganged/fake-switching hubs may bounce ALL # # siblings; self-powered hubs only reset their uplink, which # # is why the walk ends at the root port (real xHCI ppps). +# sudo usb_recover.sh root-cycle <busport> [serial] # e.g. 13-1.6 -> uhubctl VBUS cut at the ROOT port feeding +# # it; [serial] is verified against the device and refused on mismatch, +# # skipping the leaf hubs (which fake ganged switching and do not +# # actually cut power). Bounces every sibling under that root port. +# # The D-state escape: no device lock, so it cannot convoy. # sudo usb_recover.sh resolve <devnode> # e.g. /dev/ttyACM3 -> print its <busport> (no privilege needed) set -euo pipefail @@ -25,6 +27,29 @@ PCI_RE='^[0-9a-fA-F]{4}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}\.[0-9]$' DRIVER_RE='^[A-Za-z0-9_-]+$' die() { echo "usb_recover: $*" >&2; exit 1; } + +# Generation marker for "did this device actually re-enumerate". A real disconnect destroys the +# usb_device and its sysfs kobject; reconnecting creates a new one, and kernfs hands out inode +# numbers monotonically, so the directory inode changes. Verified on the rig: ports re-enumerated +# minutes ago carry inodes in the millions while ports untouched since boot are still in the tens +# of thousands, ranking identically to their mtimes. +# +# This beats comparing devnum, which Linux reuses once the per-bus map wraps (observed live: a +# single cycle moved one device 123 -> 113). It also beats watching for the node to vanish, since +# `uhubctl -a cycle` holds the whole power-off window inside itself and a poll afterwards can +# never witness the gap. The inode survives the gap, so no observation window is needed. +# +# Crucially, if the disconnect is blocked on the wedged device's lock the kobject is never +# recreated -- same inode -- which is exactly the case that must be reported as a failure. Verified +# against kernfs: __kernfs_new_node() allocates via idr_alloc_cyclic() but kernfs_id_ino() exposes +# the full 64-bit (id_highbits<<32 | lowbits) as st_ino on 64-bit ino_t, so a repeat needs ~2^64 +# node creations. authorized-toggle, set_configuration and suspend/resume all leave the parent +# device kobject alone, so none of them can move the marker and fake a success. +# +# The trailing slash is load-bearing: /sys/bus/usb/devices/<busport> is a SYMLINK with its own +# separate inode, so without it stat reports the link rather than the device it points at, and the +# value would never change. Do not "tidy" it away. +sysfs_gen() { stat -c %i "/sys/bus/usb/devices/$1/" 2>/dev/null || echo none; } usage() { grep -E '^# sudo usb_recover' "$0" >&2; exit 2; } # Refuse to touch a PCI function that is not a USB controller (class 0x0c03xx), so a stray or @@ -107,6 +132,11 @@ case "$action" in [[ "$target" =~ $USBPATH_RE ]] || die "bad usb path: $target" UHUBCTL=$(command -v uhubctl || echo /sbin/uhubctl) [ -x "$UHUBCTL" ] || die "uhubctl not installed" + # sysfs generation, not node existence: a disconnect blocked on the device lock leaves the + # old node (and its idVendor) in place, so an existence check reports success without anything + # having happened -- and the walk to the root port, which is the part that actually cuts power + # on these fake-ganged leaf hubs, would never run. + gen=$(sysfs_gen "$target") dev="$target" while :; do if [[ "$dev" =~ ^([0-9]+)-([0-9]+)$ ]]; then # parent is the root hub @@ -118,8 +148,9 @@ case "$action" in "$UHUBCTL" -l "$loc" -p "$port" -a cycle -d 5 -f || echo " (uhubctl failed at $loc; walking up)" for _ in $(seq 1 10); do sleep 1 - if [ -e "/sys/bus/usb/devices/$target/idVendor" ]; then - echo "recovered: $target re-enumerated"; exit 0 + now=$(sysfs_gen "$target") + if [ "$now" != none ] && [ "$now" != "$gen" ]; then + echo "recovered: $target re-enumerated (gen $gen -> $now)"; exit 0 fi done [ -n "$up" ] || break @@ -127,12 +158,48 @@ case "$action" in done die "hub-cycle: $target still not enumerated after cycling up to the root port" ;; - pci-reset) - [[ "$target" =~ $PCI_RE ]] || die "bad pci addr: $target" - require_usb_controller "$target" - [ -e "/sys/bus/pci/devices/$target/reset" ] || die "no reset support on $target" - echo 1 > "/sys/bus/pci/devices/$target/reset" - echo "flr-reset pci $target" + root-cycle) + # VBUS cut at the ROOT port, where xHCI ppps is real. Unlike hub-cycle this does not walk up + # from the leaf (the 1a40:0201 hubs claim ganged switching but never cut power) and never + # writes the wedged device's sysfs or takes its lock, so it cannot join a D-state convoy. + # uhubctl exits 0 even when it does nothing ("No compatible devices detected" still returns + # 0), so its status proves nothing -- the sysfs_gen check below is the only real verdict. + [[ "$target" =~ $USBPATH_RE ]] || die "bad usb path: $target" + UHUBCTL=$(command -v uhubctl || echo /sbin/uhubctl) + [ -x "$UHUBCTL" ] || die "uhubctl not installed" + # Existence alone only proves *something* occupies that path -- bus numbers renumber every + # boot, so a stale busport can name a different device entirely and we would cut power to its + # whole subtree (up to 25 fixtures on this rig). Callers that know what they expect pass the + # serial as a third argument and we refuse on mismatch; otherwise print the identity so a + # wrong target is at least visible. + [ -e "/sys/bus/usb/devices/$target" ] || die "no such usb device: $target" + idf="/sys/bus/usb/devices/$target" + serial=$(cat "$idf/serial" 2>/dev/null || echo -) + expect=${3:-} + [ -z "$expect" ] || [ "$expect" = "$serial" ] || \ + die "root-cycle: $target has serial '$serial', expected '$expect' — stale busport, refusing" + echo "root-cycle: target $target is $(cat "$idf/idVendor" 2>/dev/null || echo -):$(cat "$idf/idProduct" 2>/dev/null || echo -)" \ + "serial=$serial product=$(cat "$idf/product" 2>/dev/null || echo -)" + bus=${target%%-*}; rest=${target#*-}; rootport=${rest%%.*} + gen=$(sysfs_gen "$target") + echo "root-cycle: cutting VBUS on bus $bus root port $rootport (feeds $target, bounces its siblings)" + # -S is load-bearing. By default uhubctl writes /sys/.../usb<bus>-port<n>/disable (verified: + # two O_WRONLY opens per cycle), and the kernel's disable_store() takes the ROOT HUB's lock and + # synchronously usb_disconnect()s the child BEFORE cutting power -- against a wedged device that + # blocks on the lock we are trying to free, so power would never drop and uhubctl would D-state + # holding the root hub's lock, poisoning the whole bus. -S forces the libusb path, which sends + # the power-off control transfer straight to the root hub with no child-disconnect in front. + "$UHUBCTL" -S -l "$bus" -p "$rootport" -a cycle -d 5 \ + || die "uhubctl failed to cycle bus $bus port $rootport" + for _ in $(seq 1 10); do + sleep 1 + now=$(sysfs_gen "$target") + if [ "$now" != none ] && [ "$now" != "$gen" ]; then + echo "root-cycled $bus port $rootport: $target re-enumerated"\ + "(devnum $(cat "/sys/bus/usb/devices/$target/devnum" 2>/dev/null || echo ?), gen $gen -> $now)"; exit 0 + fi + done + die "root-cycle: $target did not re-enumerate after cycling bus $bus port $rootport (sysfs generation still $gen: no disconnect happened)" ;; *) usage diff --git a/test/hil/usbtest.py b/test/hil/usbtest.py index e17705a48..83ea3e24c 100755 --- a/test/hil/usbtest.py +++ b/test/hil/usbtest.py @@ -254,11 +254,46 @@ def dmesg_tail(): return '\n'.join(lines[-8:]) -def pci_addr_of_bus(busnum): - """Return the PCI B:D.F backing a USB bus, or None for a non-PCI (SoC/platform) controller.""" - m = re.search(r'([0-9a-f]{4}:[0-9a-f]{2}:[0-9a-f]{2}\.[0-9])/usb\d+$', - os.path.realpath(f'/sys/bus/usb/devices/usb{int(busnum)}')) - return m.group(1) if m else None +def wedged_pids(devnode): + """Return (pids, complete): PIDs in uninterruptible sleep whose cmdline names devnode, i.e. + still holding its usbfs device lock, and whether every /proc entry could actually be read. + + Matched by device node rather than by our child's pid because run_case() may wrap testusb in + sudo, in which case the Popen pid is the wrapper and the blocked process is its child -- + killing the wrapper would make a pid-based check look clean while the real holder is stuck. + + complete is False when a PermissionError hid an entry (a hidepid/ProtectProc mount, or the + root-owned child of that same sudo). An entry we could not read might be the holder, so the + caller must treat that as unrecovered rather than as an all-clear.""" + stuck, complete = [], True + # hidepid=2 and systemd's ProtectProc=invisible omit other users' processes from iterdir() + # entirely -- no entry at all, so no PermissionError to catch -- and testusb runs under sudo + # whenever the device node is not writable. The scan would then look clean while hiding the + # very holder it exists to find. pid 1 is always root-owned, so being unable to read it means + # enumeration is restricted and no result from this scan can be trusted as complete. + if os.geteuid() != 0 and not os.access('/proc/1/cmdline', os.R_OK): + complete = False + for entry in Path('/proc').iterdir(): + if not entry.name.isdigit(): + continue + try: + cmdline = (entry / 'cmdline').read_bytes() + except PermissionError: + complete = False # cannot rule this pid out + continue + except OSError: + continue # raced with process exit: genuinely gone, not hidden + if devnode.encode() not in cmdline: + continue + try: + stat = (entry / 'stat').read_text() + if stat[stat.rindex(')') + 2] == 'D': # comm may contain ')', so scan from the right + stuck.append(int(entry.name)) + except PermissionError: + complete = False + except (OSError, ValueError, IndexError): + continue + return stuck, complete def run_case(num, dev, testusb, quick, timeout): @@ -387,20 +422,68 @@ def main(): extra += f" {r['mbps']} MB/s" if 'mbps' in r else '' print(f"test {num:2d} {r['name']:22s} {r['status']:6s}{extra}") if r['status'] == 'HUNG': - pci = pci_addr_of_bus(dev['node'].split('/')[-2]) - if pci: - print(f'aborting battery: kernel-side hang, device wedged mid-transfer.\n' - f'auto-recovering: sudo {USB_RECOVER} pci-reset {pci} ' - f'(see .claude/skills/usb-kernel-recover)', file=sys.stderr) - # FLR frees the D-state ioctl without the device lock; must run BEFORE - # any unbind/remove_id, which would deadlock the bus otherwise - if sudo([str(USB_RECOVER), 'pci-reset', pci]).returncode != 0: - unrecovered_hang = True - time.sleep(5) # let the bus re-enumerate before cleanup touches sysfs - else: - unrecovered_hang = True - print('aborting battery: kernel-side hang, and the controller has no PCI address ' - 'for FLR recovery — manual intervention (reboot) required', file=sys.stderr) + print(f'aborting battery: kernel-side hang, device wedged mid-transfer.\n' + f'auto-recovering: {USB_RECOVER.name} root-cycle {dev["sysname"]} ' + f'(see .claude/skills/usb-kernel-recover)', file=sys.stderr) + # Cutting VBUS at the root port fails the in-flight URB so the usbfs ioctl returns. + # Must run BEFORE any unbind/remove_id, which would take the device lock the stuck + # ioctl holds and deadlock the bus. + # + # Assume unrecovered until proven otherwise, so that any early exit from this block + # -- an OSError spawning the helper, a KeyboardInterrupt, a sudo prompt killing the + # run -- still reaches the finally cleanup with the flag set, instead of running + # the remove_id/unbind the comments there forbid while a device lock is held. + unrecovered_hang = True + # Pass the serial so the helper refuses a stale busport rather than cutting power + # to whatever else now occupies that path. Popen rather than sudo()/subprocess.run: + # run() would kill() then wait() unbounded on timeout, which never returns if + # uhubctl is itself in D state -- the case the timeout exists for. Merge stderr + # into stdout so the helper's target-identity and action lines are not lost. + # Only pass the serial when we actually have one: an empty third argument reads as + # "no expectation" and would silently disable the helper's stale-busport guard. + cmd = [str(USB_RECOVER), 'root-cycle', dev['sysname']] + if dev['serial']: + cmd.append(dev['serial']) + if os.geteuid() != 0: + cmd = ['sudo', '-n'] + cmd + try: + p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, + text=True) + except OSError as e: + # helper missing or not executable, or sudo unavailable. unrecovered_hang is + # already True so the finally block still skips the unsafe cleanup -- this only + # replaces a traceback with a message that says what to fix. + print(f'cannot run {USB_RECOVER}: {e}', file=sys.stderr) + break + rc = None + try: + out, _ = p.communicate(timeout=60) # normal run is ~8s + rc = p.returncode + except subprocess.TimeoutExpired: + p.kill() + try: + out, _ = p.communicate(timeout=5) + rc = p.returncode + except subprocess.TimeoutExpired: + out = ('root-cycle abandoned after 60s: uhubctl did not die to SIGKILL, so ' + 'it is wedged too and the convoy has spread beyond this device') + if out: + print(out.strip(), file=sys.stderr) + if rc is not None: + time.sleep(5) # let the bus settle and the freed ioctl unwind + # Authoritative either way. A non-zero exit only means the device did not come + # back within the poll (a slow bootloader will do that) -- if nothing still + # holds the lock, the bus is usable and cleanup is safe. Conversely a zero exit + # only proves re-enumeration, not that the D-state holder let go. + stuck, complete = wedged_pids(dev['node']) + if stuck: + print(f'{dev["sysname"]}: pid(s) {stuck} still in D state on ' + f'{dev["node"]} — the device lock was never released', file=sys.stderr) + elif not complete: + print('cannot confirm recovery: /proc is only partly readable, so a ' + 'hidden D-state holder cannot be ruled out', file=sys.stderr) + else: + unrecovered_hang = False break # re-resolve: after a mid-battery re-enumeration the devnum (and thus the node # path) changes; keep testing the live node instead of the stale one. Match on the @@ -419,7 +502,8 @@ def main(): if unrecovered_hang: # testusb is still stuck in a usbfs ioctl holding the device lock; remove_id/unbind # would join the convoy and deadlock the bus (see usb-kernel-recover skill) — leave it be - print('skipping cleanup after unrecovered hang: reboot required to release the bus', + print('skipping cleanup after unrecovered hang: ask the operator for a full PVE host ' + 'power cycle (a VM reboot is not reliable — hubs latch up across the PCIe reset)', file=sys.stderr) elif not args.keep_binding: sysfs_write(DRIVER / 'remove_id', f'{VID} {PID}', check=False) |