hil: add usb_recover hub-cycle action; drop MosChip skips, gate it as incompatible

The MosChip MCS9990 card is physically removed from the rig: delete its
cases-11/25 SKIP workaround (and the now-orphaned SKIP accounting) from
usbtest.py and refuse to run outright if a DUT ever sits behind one again.

usb_recover.sh gains `hub-cycle <busport>`: uhubctl VBUS cycle of the port
feeding the device, walking upstream (parent hub -> root port) until it
re-enumerates. Verified on the rig: leaf-level recovery (13-4.4 usbtest
device) and full walk to the root port on a dead branch. SKILL.md updated
for the action and the two-Renesas topology (root-port ppps is real; leaf
1a40:0201 hubs fake their "ganged" switching).

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WxUeX4Yn26KibfjvDg2pN9

4 files changed, 74 insertions, 52 deletions

diff --git a/.claude/skills/usb-recover/SKILL.md b/.claude/skills/usb-recover/SKILL.md
index 3f72b7e21..beb6fd862 100644
--- a/.claude/skills/usb-recover/SKILL.md
+++ b/.claude/skills/usb-recover/SKILL.md
@@ -6,18 +6,27 @@ description: Use when a USB device or fixture on the ci HIL rig is stuck, hung,
 # USB Recovery on the HIL Rig
 
 Run this skill's `scripts/usb_recover.sh` with `sudo` (abbreviated to
-`usb_recover.sh` in the examples below). It wraps four sysfs reset actions plus
-a resolver:
+`usb_recover.sh` in the examples below). It wraps the sysfs reset actions, a
+uhubctl power-cycle escalator, and a resolver:
 
 ```bash
 sudo usb_recover.sh resolve    /dev/ttyACM3    # /dev node -> busport (e.g. 3-4.7); also ttyUSB*, sg*
 sudo usb_recover.sh authorized <busport>       # deauthorize+reauthorize: re-enumerate, no VBUS cut
 sudo usb_recover.sh rebind     <busport>       # usb driver unbind+bind: re-probe
+sudo usb_recover.sh hub-cycle  <busport>       # uhubctl VBUS cycle of the feeding port, walking parent hub
+                                               # -> root port until the device re-enumerates
 sudo usb_recover.sh pci-rebind <pciaddr>       # whole HCD controller unbind+bind, e.g. 0000:02:00.0
 sudo usb_recover.sh pci-reset  <pciaddr>       # PCI function-level reset: kills URBs at HW level, no device lock
 sudo usb_recover.sh pci-bind   <pciaddr> [drv] # re-bind a DRIVERLESS controller (auto-tries xHCI drivers)
 ```
 
+`hub-cycle` caveats: leaf hubs that gang (or fake) port power switching bounce
+**all siblings** on that hub when cycled; a **self-powered** leaf hub keeps
+downstream VBUS up, so cycling it only resets its uplink — that's why the walk
+escalates to the root port, where the Renesas cards' per-port power (ppps) is
+real. A device that is wedged but bus-powered from a switching hub gets a true
+power cycle; one on a self-powered hub may only get a re-enumeration.
+
 ## Decide first: is anything stuck in D state?
 
 ```bash
@@ -37,8 +46,8 @@ the ioctl then returns and the convoy unwinds on its own.
 **Not every controller supports FLR.** The Renesas uPD720201 (`0000:01:00.0`)
 has no reset method — `pci-reset` fails with `Inappropriate ioctl for device`
 (ENOTTY). On those, there is no clean software D-state cure — a VM reboot is NOT
-reliable (the MosChip downstream hubs latch up across the PCIe reset and need a
-physical replug); ask the operator for a full PVE host power cycle instead. Do NOT
+reliable (downstream hubs can latch up across the PCIe reset and need a physical
+replug); ask the operator for a full PVE host power cycle instead. Do NOT
 fall through to `pci-rebind` (see next).
 
 **`pci-rebind` can strand the controller driverless.** Its unbind succeeds but,
@@ -62,7 +71,9 @@ needs: once a rebind has been attempted and is stuck, even FLR deadlocks and
 
 1. `authorized <busport>` — re-enumerates just that device
 2. `rebind <busport>` — re-probe; also worth trying on the parent hub's busport
-3. `pci-rebind <pciaddr>` — last resort: bounces every fixture on that controller
+3. `hub-cycle <busport>` — VBUS cycle of the feeding port, walking up to the
+   root port; may bounce sibling fixtures on ganged hubs
+4. `pci-rebind <pciaddr>` — last resort: bounces every fixture on that controller
 
 ## Finding targets
 
@@ -71,10 +82,13 @@ grep -l <SERIAL> /sys/bus/usb/devices/*/serial          # serial -> busport (dir
 readlink -f /sys/bus/usb/devices/usb<N>                  # bus N -> its PCI addr in the path
 ```
 
-Rig layout: buses 3+4 = `0000:02:00.0` (main fixture tree: J-Links, ST-Links,
-WCH-Links, DUTs); buses 9+12 = `0000:01:00.0`, the only ones with uhubctl port
-power (ganged VBUS: `sudo uhubctl -l 9 -a cycle`). Hubs on buses 1-4 have no
-port power switching — uhubctl reports "No compatible devices" there.
+Rig layout (2026-07-15, two Renesas uPD720201 cards; bus numbers renumber every
+boot — re-derive with `readlink`): AMD `0000:02:00.0` = the debug-probe tree
+(J-Links, ST-Links, WCH-Links), no port power switching; Renesas `0000:01:00.0`
+and `0000:03:00.0` = DUT device hubs + serial fixtures, and ALL their root-hub
+ports have real per-port power (`ppps`, 4+4 each) — `sudo uhubctl -l <bus> -p
+<port> -a cycle` cuts VBUS to the leaf hub on that port. The 1a40:0201 leaf
+hubs themselves claim "ganged" switching but do not actually cut power.
 
 ## Common mistakes
 
diff --git a/.claude/skills/usb-recover/scripts/usb_recover.sh b/.claude/skills/usb-recover/scripts/usb_recover.sh
index 35bd4c784..7652253fa 100755
--- a/.claude/skills/usb-recover/scripts/usb_recover.sh
+++ b/.claude/skills/usb-recover/scripts/usb_recover.sh
@@ -12,6 +12,11 @@
 #   sudo usb_recover.sh pci-bind   <pciaddr> [driver]  # bind a DRIVERLESS controller (e.g. after a pci-rebind
 #                                              # whose re-bind hung and left it unbound). Auto-tries the xHCI
 #                                              # drivers (xhci-pci-renesas, xhci_hcd) unless one is named.
+#   sudo usb_recover.sh hub-cycle  <busport>   # e.g. 13-1.6 -> uhubctl power-cycle of the port feeding it,
+#                                              # walking upstream (parent hub -> root port) until the device
+#                                              # re-enumerates. Ganged/fake-switching hubs may bounce ALL
+#                                              # siblings; self-powered hubs only reset their uplink, which
+#                                              # is why the walk ends at the root port (real xHCI ppps).
 #   sudo usb_recover.sh resolve    <devnode>   # e.g. /dev/ttyACM3 -> print its <busport> (no privilege needed)
 set -euo pipefail
 
@@ -98,6 +103,30 @@ case "$action" in
       die "could not bind $target with a known xHCI driver; pass the driver explicitly"
     fi
     ;;
+  hub-cycle)
+    [[ "$target" =~ $USBPATH_RE ]] || die "bad usb path: $target"
+    UHUBCTL=$(command -v uhubctl || echo /sbin/uhubctl)
+    [ -x "$UHUBCTL" ] || die "uhubctl not installed"
+    dev="$target"
+    while :; do
+      if [[ "$dev" =~ ^([0-9]+)-([0-9]+)$ ]]; then    # parent is the root hub
+        loc="${BASH_REMATCH[1]}"; port="${BASH_REMATCH[2]}"; up=""
+      else                                            # parent is a downstream hub
+        loc="${dev%.*}"; port="${dev##*.}"; up="$loc"
+      fi
+      echo "hub-cycle: power-cycling hub $loc port $port (feeds $dev)"
+      "$UHUBCTL" -l "$loc" -p "$port" -a cycle -d 5 -f || echo "  (uhubctl failed at $loc; walking up)"
+      for _ in $(seq 1 10); do
+        sleep 1
+        if [ -e "/sys/bus/usb/devices/$target/idVendor" ]; then
+          echo "recovered: $target re-enumerated"; exit 0
+        fi
+      done
+      [ -n "$up" ] || break
+      dev="$up"
+    done
+    die "hub-cycle: $target still not enumerated after cycling up to the root port"
+    ;;
   pci-reset)
     [[ "$target" =~ $PCI_RE ]] || die "bad pci addr: $target"
     require_usb_controller "$target"
diff --git a/test/hil/hil_test.py b/test/hil/hil_test.py
index bfb73234f..40ae80572 100755
--- a/test/hil/hil_test.py
+++ b/test/hil/hil_test.py
@@ -1734,13 +1734,10 @@ def test_device_usbtest(board):
     except (ValueError, KeyError, json.JSONDecodeError):
         raise AssertionError(f'usbtest did not run: {compact_output(out) or cmd_stdout_text(r.stderr)}')
 
-    skipped = int(data.get('skipped', 0))  # host-controller limitation (see usbtest.py host_broken_cases)
     total = passed + failed
-    if total == 0 and skipped > 0:
-        return 'skipped'  # every case host-skipped: a skip, not a 0/0 failure
     if failed == 0 and total > 0:
-        return f'{REPORT_CELL["pass"]} {passed}/{total}' + (f' +{skipped}skip' if skipped else '')
-    bad = [c.get('num') for c in data.get('cases', []) if c.get('status') not in ('PASS', 'SKIP')]
+        return f'{REPORT_CELL["pass"]} {passed}/{total}'
+    bad = [c.get('num') for c in data.get('cases', []) if c.get('status') != 'PASS']
     raise TestFail(f'usbtest {passed}/{total} (cases failed: {bad})',
                    metric=f'{REPORT_CELL["fail"]} {passed}/{total}')
 
diff --git a/test/hil/usbtest.py b/test/hil/usbtest.py
index 9aec8ac0a..a2841f0b6 100755
--- a/test/hil/usbtest.py
+++ b/test/hil/usbtest.py
@@ -152,17 +152,11 @@ def find_device(serial, first=False):
     return matches[0]
 
 
-def host_broken_cases(dev):
-    """Cases the DUT's upstream host controller cannot run: {case: reason}. Exits the
-    whole run instead if the host is a uPD720201 on pre-2.0.2.6 firmware (see below).
-    The MosChip MCS9990 (9710:9990) EHCI cannot run interrupt-OUT: its FRINDEX
-    register is buggy silicon (the kernel probes it with "applying MosChip
-    frame-index workaround") and ehci-hcd never keeps the int-OUT QH in the
-    hardware periodic schedule, so every int-OUT URB times out regardless of
-    bInterval/mps/size while the device sits armed. Verified A/B 2026-07-09,
-    same board+hub: EHCI FAIL (QH absent from the debugfs periodic schedule the
-    whole hang), OHCI companion PASS, xHCI fine; int-IN unaffected. Skip with a
-    visible SKIP so the battery self-heals once the DUT tree is back on an xHCI."""
+def check_host_compat(dev):
+    """Refuse to run when the DUT's upstream host controller is known-incompatible:
+    the MosChip MCS9990 (9710:9990) outright (buggy FRINDEX silicon: EHCI never
+    schedules int-OUT URBs and mangles unlinked reads - verified A/B 2026-07-09),
+    and the Renesas uPD720201/02 unless it runs firmware >= 2.0.2.6 (see below)."""
     for attempt in range(3):
         try:
             root = Path(f"/sys/bus/usb/devices/usb{int(dev['node'].split('/')[-2])}")
@@ -172,20 +166,17 @@ def host_broken_cases(dev):
             break
         except (OSError, ValueError):
             # transient sysfs error (e.g. racing a re-enumeration): retry so a blip doesn't
-            # silently run known-broken cases; if the probe truly fails, fail open but say so
+            # silently pass an incompatible host; if the probe truly fails, fail open but say so
             if attempt == 2:
                 print('warning: cannot probe the upstream host controller; '
-                      'known-broken-host cases will run instead of being skipped', file=sys.stderr)
-                return {}
+                      'skipping the host compatibility check', file=sys.stderr)
+                return
             time.sleep(1)
-    if drv.startswith('ehci') and vid_did == ('0x9710', '0x9990'):
-        return {
-            25: 'host EHCI (MosChip MCS9990) loses interrupt-OUT completions',
-            # Unlinking an in-progress read intermittently completes it as a short transfer
-            # (EREMOTEIO) instead of -ECONNRESET; device-side exonerated by TX counters (only
-            # full-mps loads, no ZLP). Passes on xHCI. Some boards dodge it by timing.
-            11: 'host EHCI (MosChip MCS9990) completes unlinked reads as short (EREMOTEIO)',
-        }
+    if vid_did == ('0x9710', '0x9990'):
+        sys.exit(f'REFUSING to run: DUT is behind a MosChip MCS9990 ({pci.name}), which is '
+                 'incompatible with usbtest: broken FRINDEX silicon - int-OUT URBs are never '
+                 'placed in the EHCI periodic schedule and unlinked reads complete as short '
+                 'transfers (EREMOTEIO). Move the DUT to an xHCI port.')
     if drv.startswith('xhci') and vid_did in (('0x1912', '0x0014'), ('0x1912', '0x0015')):
         # The Renesas uPD720201/uPD720202 must run its latest firmware (>= 2.0.2.6,
         # K2026090.mem; RAM-uploaded, so it reverts to ROM on every power cycle unless
@@ -215,7 +206,6 @@ def host_broken_cases(dev):
                      '< 0x00202609 (2.0.2.6) - its command ring dies under usbtest unlink '
                      'stress. Load the latest firmware (K2026090.mem; it is RAM-uploaded and '
                      'reverts to ROM on every power cycle).')
-    return {}
 
 
 def bind_usbtest(dev):
@@ -379,9 +369,9 @@ def main():
     if not args.json:
         print(info)
 
-    # probe the upstream controller before touching the device: an unsupported host
-    # (uPD720201 on pre-2.0.2.6 firmware) exits here, before any bind
-    broken = host_broken_cases(dev)
+    # probe the upstream controller before touching the device: an incompatible host
+    # (MosChip MCS9990, or uPD720201 on pre-2.0.2.6 firmware) exits here, before any bind
+    check_host_compat(dev)
 
     results = []
     unrecovered_hang = False
@@ -390,12 +380,6 @@ def main():
         set_pattern(0)  # tier 1 firmware sources zeros; also required by perf cases 27/28
 
         for num in cases:
-            if num in broken:
-                results.append({'num': num, 'name': CASE_NAMES[num], 'status': 'SKIP',
-                                'detail': broken[num]})
-                if not args.json:
-                    print(f"test {num:2d} {CASE_NAMES[num]:22s} SKIP   {broken[num]}")
-                continue
             results.append(run_case(num, dev, testusb, args.quick, args.timeout))
             r = results[-1]
             if not args.json:
@@ -447,16 +431,14 @@ def main():
         except SystemExit:
             pass
 
-    failed = [r for r in results if r['status'] not in ('PASS', 'SKIP')]
-    skipped = [r for r in results if r['status'] == 'SKIP']
-    ran = len(results) - len(skipped)
+    failed = [r for r in results if r['status'] != 'PASS']
+    ran = len(results)
     if args.json:
         print(json.dumps({'serial': dev['serial'], 'speed': dev['speed'], 'tier': tier,
                           'passed': ran - len(failed), 'failed': len(failed),
-                          'skipped': len(skipped), 'cases': results}, indent=2))
+                          'cases': results}, indent=2))
     else:
-        note = f", {len(skipped)} skipped (host limitation)" if skipped else ''
-        print(f"{ran - len(failed)}/{ran} passed{note}")
+        print(f"{ran - len(failed)}/{ran} passed")
         for r in failed:
             print(f"  FAILED test {r['num']}: {r.get('detail', '')}")
             if r.get('dmesg'):