test/hil: usbtest fleet enablement, shuffled scheduling, unique PIDs

Pool/config:
- record real uids (ra8m1_ek), enable usbtest for espressif s3/p4, then
  park ra6m5_ek and ra8m1_ek in boards-skip (ra6m5's usbtest/MSC traffic
  can kill the uPD720201 host on its ROM firmware; ra8m1 USBHS bring-up
  pending); max32666/nrf54lm20 stay enabled - their MosChip flakiness
  never wedges
- re-enable device/usbtest on HS boards (mimxrt1064, ch32v307) now that
  uPD720201 firmware 2.0.2.6 fixes the command-ring death; mimxrt1015
  stays skipped - its HS battery killed the controller on both ROM and
  2.0.2.6 firmware (board-specific); match the moved host-test bundles
  (f723 <-> rt1064); skip never-passing tests on the new
  nrf5340dk/nrf54lm20dk boards and the detached pico host bundle, each
  documented with a comment

Host-controller quirk gating in usbtest.py (auto-skip, self-heals on a
healthy xHCI):
- MosChip MCS9990 EHCI: case 25 (int-OUT never scheduled, FRINDEX bug)
  and case 11 (unlinked reads complete short/EREMOTEIO)
- Renesas uPD720201 xHCI: firmware-gated. The card must run firmware
  >= 2.0.2.6 (RAM-uploaded - it reverts to ROM on every power cycle):
  on older firmware the command ring dies under unlink stress (a
  Configure Endpoint command stops completing; the hub worker deadlocks
  holding the device lock; only a host power cycle recovers; three
  boards reproduced it). usbtest.py reads the FW version register (PCI
  config 0x6c) and refuses to run at all on older firmware - hil_test
  surfaces that as a failed test with the reason. On current firmware
  the full 30-case battery runs (validated FS+HS: metro_m4, f723,
  f723-DMA all 30/30).

Scheduling (hil_test.py):
- Shuffle each (board, variant)'s test order with a seeded RNG
  (HIL_SHUFFLE_SEED to replay) so usbtest batteries and flash churn spread
  across the timeline instead of convoying on one controller.
- Per-controller usbtest + flash semaphores: HIL_USBTEST_PARALLEL
  (default 4) concurrent usbtest batteries and HIL_FLASH_PARALLEL
  (default 8) concurrent flashes per host controller. Profiled on
  uPD720201 firmware 2.0.2.6 across 8/1..12/8: wall time falls
  22.2/14.3/12.5/10.8 min at usbtest width 1/2/3/4 and plateaus there;
  zero controller errors everywhere; first battery case failures
  (leaf-hub bandwidth stretch) appear at 12/8, and flash width 12 only
  amplifies flasher-hub contention flakes - so 8/4 is the optimum. A
  separate battery-window flash throttle was profiled and dropped.
- Give every example a unique hardcoded USB PID (0x4001-0x4022, usbtest
  keeps 0x4010) instead of the PID_MAP interface bitmap: different
  examples now always re-enumerate back-to-back, even on boards whose
  CPU reset does not drop D+ (WCH CH58x), so the EXAMPLE_PID table and
  same-PID adjacency reordering in hil_test.py are gone; only the
  variant-boundary same-example repeat needs a swap.
- Report matrix: stable columns with the metric-bearing tests pinned
  first (usbtest, cdc_msc_throughput, msc_file_explorer[_freertos]),
  the rest alphabetical.

Fail fast:
- enum wait budget 8 s on the first attempt, 4 s on retries; dfu waits
  are deadline-based so dfu-util's own runtime counts against the
  budget.
  A device-absent failure now costs ~3-5x a passing test (20-30 s)
  instead of 10-30x (47-150 s).
- CI runs hil_test with --retry 1 and no in-run second pass: a broken
  fixture fails the job fast instead of holding the self-hosted runner
  for hours and blocking other PRs' HIL jobs. hil_test still writes the
  .skip sidecar, so a manual re-run attempt only retests what failed.

Review fixes (multi-agent adversarial review of this commit):
- tinyusb_win_usbser.inf: the PID rework moved five CDC examples onto
  even PIDs the INF's odd-only DeviceList never matched (legacy-Windows
  usbser binding) - appended 0x4006/4008/400a/4020/4022 to both lists.
- usbtest example: USBTEST_TIER is now overridable and the descriptors
  and pumps are tier-conditional, so a board whose DCD cannot serve a
  tier lowers it instead of skipping the whole example - RA2A1 (RUSB2
  with no isochronous pipe) builds at tier 3 via its BOARD_ define; the
  host battery follows the tier advertised in bcdDevice. Tier-4 output
  verified byte-identical after the refactor.
- dynamic_configuration's second config derived USB_PID + 11 = 0x4018,
  colliding with net_lwip_webserver - now USB_PID + 0x0100, outside the
  per-example space. tools/check_example_pids.py (pre-commit hook)
  enforces PID uniqueness incl. derived and literal idProduct values.
- usbtest.py firmware gate: matched by device ID (uPD720201/720202,
  both use the 0x6c FW register), and an unreadable version (setpci
  missing/denied) now refuses with its own message instead of
  masquerading as "firmware 0x00000000"; noted the gate is necessary
  but not sufficient (board-specific kills stay per-board skips).
- hil_test: deadline waits use time.monotonic(); multiprocessing
  context pinned to fork (raw semaphores in Pool initargs); flash and
  usbtest permits unified into one fail-closed, exception-safe
  ctrl_permit (unknown controller takes every slot and logs a warning
  instead of silently borrowing slot 0); an all-skipped battery
  reports as skip, not "0/0" failure; slow-body polls (mtp, printer,
  disk read) go through a shared deadline-based wait_until so their
  bodies count against the enum budget; throughput's FS detection
  compares serials case-insensitively like every other walk; a missing
  MSC read-speed line now fails the host msc_file_explorer test
  instead of passing with an empty metric.

Hardening:
- fail fast (15 s) when a driver-registry sysfs write blocks: a wedged
  device otherwise turns every subsequent battery into an unkillable
  D-state writer and silently hangs the whole run
- usb-recover skill: a VM reboot is not a reliable cure (MosChip hubs
  latch up across the PCIe reset); full host power cycle is

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HeF2gZ1M7GWkz6Av4BpKPg

3 files changed, 34 insertions, 9 deletions

diff --git a/examples/device/usbtest/src/main.c b/examples/device/usbtest/src/main.c
index 78575ac9b..e57a90161 100644
--- a/examples/device/usbtest/src/main.c
+++ b/examples/device/usbtest/src/main.c
@@ -61,7 +61,9 @@ static uint32_t blink_interval_ms = BLINK_NOT_MOUNTED;
 // unintended short packet or ZLP.
 static uint8_t const tx_chunk[CFG_TUD_VENDOR_TX_EPSIZE];
 static uint8_t const int_tx_chunk[USBTEST_INT_EP_MPS];
+#if USBTEST_TIER >= 4
 static uint8_t const iso_tx_chunk[USBTEST_ISO_EP_MPS];
+#endif
 
 // Interrupt/iso submit one packet per (micro)frame, sized to the NEGOTIATED speed's mps — a
 // high-speed build enumerated at full speed must submit the FS length, not the HS-capacity buffer
@@ -69,9 +71,11 @@ static uint8_t const iso_tx_chunk[USBTEST_ISO_EP_MPS];
 static inline uint16_t usbtest_int_len(void) {
   return (tud_speed_get() == TUSB_SPEED_HIGH) ? USBTEST_INT_EP_MPS_HS : USBTEST_INT_EP_MPS_FS;
 }
+#if USBTEST_TIER >= 4
 static inline uint16_t usbtest_iso_len(void) {
   return (tud_speed_get() == TUSB_SPEED_HIGH) ? USBTEST_ISO_EP_MPS_HS : USBTEST_ISO_EP_MPS_FS;
 }
+#endif
 
 //------------- prototypes -------------//
 void led_blinking_task(void* param);
@@ -125,10 +129,12 @@ static void usbtest_pump(void) {
       tud_vendor_int_write(int_tx_chunk, usbtest_int_len());
     }
 
+#if USBTEST_TIER >= 4
     tud_vendor_iso_read_xfer(); // isochronous sink
     if (tud_vendor_iso_write_available()) {
       tud_vendor_iso_write(iso_tx_chunk, usbtest_iso_len());
     }
+#endif
   }
 }
 
@@ -175,6 +181,7 @@ void tud_vendor_int_tx_cb(uint8_t idx, uint32_t sent_bytes) {
 
 // Isochronous pair: same discard/refill pumps; a completion may be a missed
 // frame, re-arm regardless
+#if USBTEST_TIER >= 4
 void tud_vendor_iso_rx_cb(uint8_t idx, const uint8_t* buffer, uint32_t bufsize) {
   (void) idx;
   (void) buffer;
@@ -187,6 +194,7 @@ void tud_vendor_iso_tx_cb(uint8_t idx, uint32_t sent_bytes) {
   (void) sent_bytes;
   tud_vendor_iso_write(iso_tx_chunk, usbtest_iso_len());
 }
+#endif
 
 //--------------------------------------------------------------------+
 // Vendor control requests (EP0)
diff --git a/examples/device/usbtest/src/usb_descriptors.c b/examples/device/usbtest/src/usb_descriptors.c
index 24efef453..b4f46adb8 100644
--- a/examples/device/usbtest/src/usb_descriptors.c
+++ b/examples/device/usbtest/src/usb_descriptors.c
@@ -67,21 +67,29 @@ enum {
 
 // Vendor interface, Gadget-Zero style altsettings: alt 0 carries no endpoints (an
 // isochronous endpoint must not claim bandwidth in the default altsetting, USB 2.0
-// 5.6.3), alt 1 carries bulk + interrupt + isochronous IN/OUT. The host usbtest
-// driver skips altsettings without pipes and selects alt 1 itself. No TUD_ macro
-// covers this layout, hand-rolled.
-#define USBTEST_DESC_LEN  (9 + 9 + 6*7)
+// 5.6.3), alt 1 carries bulk + interrupt (+ isochronous IN/OUT at tier 4). The host
+// usbtest driver skips altsettings without pipes and selects alt 1 itself. No TUD_
+// macro covers this layout, hand-rolled.
+#if USBTEST_TIER >= 4
+  #define USBTEST_EP_COUNT  6
+  #define USBTEST_ISO_EPS(_isoout, _isoin, _iso_mps, _iso_interval) \
+    ,7, TUSB_DESC_ENDPOINT, _isoout, (uint8_t)(TUSB_XFER_ISOCHRONOUS | (uint8_t)(TUSB_ISO_EP_ATT_ASYNCHRONOUS)), U16_TO_U8S_LE(_iso_mps), _iso_interval,\
+    7, TUSB_DESC_ENDPOINT, _isoin, (uint8_t)(TUSB_XFER_ISOCHRONOUS | (uint8_t)(TUSB_ISO_EP_ATT_ASYNCHRONOUS)), U16_TO_U8S_LE(_iso_mps), _iso_interval
+#else
+  #define USBTEST_EP_COUNT  4
+  #define USBTEST_ISO_EPS(_isoout, _isoin, _iso_mps, _iso_interval)
+#endif
+#define USBTEST_DESC_LEN  (9 + 9 + USBTEST_EP_COUNT*7)
 #define USBTEST_DESCRIPTOR(_itfnum, _stridx, _epout, _epin, _bulk_mps, _intout, _intin, _int_mps, _int_interval, _isoout, _isoin, _iso_mps, _iso_interval) \
   /* alt 0: zero bandwidth, no endpoints */\
   9, TUSB_DESC_INTERFACE, _itfnum, 0, 0, TUSB_CLASS_VENDOR_SPECIFIC, 0x00, 0x00, _stridx,\
   /* alt 1: full source/sink set */\
-  9, TUSB_DESC_INTERFACE, _itfnum, 1, 6, TUSB_CLASS_VENDOR_SPECIFIC, 0x00, 0x00, _stridx,\
+  9, TUSB_DESC_INTERFACE, _itfnum, 1, USBTEST_EP_COUNT, TUSB_CLASS_VENDOR_SPECIFIC, 0x00, 0x00, _stridx,\
   7, TUSB_DESC_ENDPOINT, _epout, TUSB_XFER_BULK, U16_TO_U8S_LE(_bulk_mps), 0,\
   7, TUSB_DESC_ENDPOINT, _epin, TUSB_XFER_BULK, U16_TO_U8S_LE(_bulk_mps), 0,\
   7, TUSB_DESC_ENDPOINT, _intout, TUSB_XFER_INTERRUPT, U16_TO_U8S_LE(_int_mps), _int_interval,\
-  7, TUSB_DESC_ENDPOINT, _intin, TUSB_XFER_INTERRUPT, U16_TO_U8S_LE(_int_mps), _int_interval,\
-  7, TUSB_DESC_ENDPOINT, _isoout, (uint8_t)(TUSB_XFER_ISOCHRONOUS | (uint8_t)(TUSB_ISO_EP_ATT_ASYNCHRONOUS)), U16_TO_U8S_LE(_iso_mps), _iso_interval,\
-  7, TUSB_DESC_ENDPOINT, _isoin, (uint8_t)(TUSB_XFER_ISOCHRONOUS | (uint8_t)(TUSB_ISO_EP_ATT_ASYNCHRONOUS)), U16_TO_U8S_LE(_iso_mps), _iso_interval
+  7, TUSB_DESC_ENDPOINT, _intin, TUSB_XFER_INTERRUPT, U16_TO_U8S_LE(_int_mps), _int_interval\
+  USBTEST_ISO_EPS(_isoout, _isoin, _iso_mps, _iso_interval)
 
 #define CONFIG_TOTAL_LEN    (TUD_CONFIG_DESC_LEN + USBTEST_DESC_LEN)
 
diff --git a/examples/device/usbtest/src/usb_descriptors.h b/examples/device/usbtest/src/usb_descriptors.h
index 61b931bd0..bcf8b5ec4 100644
--- a/examples/device/usbtest/src/usb_descriptors.h
+++ b/examples/device/usbtest/src/usb_descriptors.h
@@ -32,7 +32,16 @@
 //   2: + vendor control 0x5b/0x5c (ctrl_out)
 //   3: + interrupt source/sink
 //   4: + isochronous source/sink
-#define USBTEST_TIER  4
+// Default is the full tier 4; a board whose DCD cannot serve a tier lowers it here
+// (BOARD_<NAME> is defined by both build systems) and the host battery follows.
+#ifndef USBTEST_TIER
+  #if defined(BOARD_RA2A1_EK)
+    // RA2A1's RUSB2 instance has no isochronous pipe (other RA parts have pipes 1-2)
+    #define USBTEST_TIER  3
+  #else
+    #define USBTEST_TIER  4
+  #endif
+#endif
 
 // Interrupt/isochronous endpoint max packet sizes, must match the configuration descriptor.
 // TUD_OPT_HIGH_SPEED is a compile-time capability flag, NOT the live bus speed, so the full-speed