diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/assets/hil/cable-xh254.jpg | bin | 0 -> 13306 bytes | |||
| -rw-r--r-- | docs/assets/hil/leaf-hub.jpg | bin | 0 -> 94557 bytes | |||
| -rw-r--r-- | docs/assets/hil/pcie-card.jpg | bin | 0 -> 80079 bytes | |||
| -rw-r--r-- | docs/assets/hil/storage-box.jpg | bin | 0 -> 177136 bytes | |||
| -rwxr-xr-x | docs/conf.py | 4 | ||||
| -rw-r--r-- | docs/reference/hardware-in-the-loop.md | 351 | ||||
| -rw-r--r-- | docs/reference/hil_boards.md | 45 | ||||
| -rw-r--r-- | docs/reference/index.rst | 1 |
8 files changed, 400 insertions, 1 deletions
diff --git a/docs/assets/hil/cable-xh254.jpg b/docs/assets/hil/cable-xh254.jpg Binary files differnew file mode 100644 index 000000000..1f8ec8a07 --- /dev/null +++ b/docs/assets/hil/cable-xh254.jpg diff --git a/docs/assets/hil/leaf-hub.jpg b/docs/assets/hil/leaf-hub.jpg Binary files differnew file mode 100644 index 000000000..3557047e6 --- /dev/null +++ b/docs/assets/hil/leaf-hub.jpg diff --git a/docs/assets/hil/pcie-card.jpg b/docs/assets/hil/pcie-card.jpg Binary files differnew file mode 100644 index 000000000..4563bb9f7 --- /dev/null +++ b/docs/assets/hil/pcie-card.jpg diff --git a/docs/assets/hil/storage-box.jpg b/docs/assets/hil/storage-box.jpg Binary files differnew file mode 100644 index 000000000..03f389ab0 --- /dev/null +++ b/docs/assets/hil/storage-box.jpg diff --git a/docs/conf.py b/docs/conf.py index c6d04bff5..ea3de0c44 100755 --- a/docs/conf.py +++ b/docs/conf.py @@ -31,7 +31,9 @@ extensions = [ templates_path = ['_templates'] -exclude_patterns = ['_build'] +# 'superpowers' holds internal plans/specs/handoffs (see CLAUDE.md), not published docs. +# 'reference/hil_boards.md' is a generated partial that hardware-in-the-loop.md includes. +exclude_patterns = ['_build', 'superpowers', 'reference/hil_boards.md'] # -- Options for HTML output ------------------------------------------------- diff --git a/docs/reference/hardware-in-the-loop.md b/docs/reference/hardware-in-the-loop.md new file mode 100644 index 000000000..48c362f4f --- /dev/null +++ b/docs/reference/hardware-in-the-loop.md @@ -0,0 +1,351 @@ +# Hardware in the Loop (HIL) + +Every pull request that touches code builds the examples and runs them on real silicon +before it can merge. This page documents the rigs that do it, in enough detail to +reproduce one. + +Two rigs run the CI matrix: + +| Rig | Config | Runner labels | +|-------|-------------------------|---------------------------------------------------------| +| `ci` | `test/hil/tinyusb.json` | `self-hosted`, `X64`, `hathach`, `hardware-in-the-loop` | +| `hfp` | `test/hil/hfp.json` | `self-hosted`, `Linux`, `X64`, `hifiphile` | + +`ci` is hathach's rig and is what the rest of this page describes. `hfp` is a similar VM +with a uPD720201 card, hosted by hifiphile. + +## Bill of materials + +| Part | Used on `ci` | Notes | +|-----------------|----------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------| +| Host PC | Ryzen 9 3900X, MSI MAG B550M MORTAR WIFI, 32 GB | Any x86 with a working IOMMU | +| USB controllers | 4 × Renesas uPD720201 (`1912:0014` rev 03) on one PCIe card | [SSU SU-U3244-12U][aio-card]: four controllers behind an on-board PCIe switch, 12 ports | +| Leaf hubs | [MCS-92M 7-port USB 2.0 hub board][hub-board] | XH2.54 headers instead of Type-A: sturdier under handling and far tidier to route | +| Cables | XH2.54 → [Type-C][cable-c] / [micro-B][cable-micro] pigtails | Hub-end pin order: `+`, `D−`, `D+`, `−` | +| Debug probes | J-Link, ST-Link, RP2040 debug probe (CMSIS-DAP), WCH-Link, TI ICDI, ESP USB-JTAG | One per board — see Attached boards below | +| USB fixtures | Per host-capable board: one USB-serial adapter and one USB flash drive | Only for boards that run host/dual tests — see below | + +[aio-card]: https://item.taobao.com/item.htm?id=990655153501 +[hub-board]: https://item.taobao.com/item.htm?id=556123792554 +[cable-c]: https://item.taobao.com/item.htm?id=826743445229 +[cable-micro]: https://item.taobao.com/item.htm?id=591895354552 + +```{figure} ../assets/hil/pcie-card.jpg +:alt: Four-controller USB PCIe card +:width: 360px + +One card, four uPD720201 controllers behind a PCIe switch. +``` + +```{figure} ../assets/hil/leaf-hub.jpg +:alt: MCS-92M leaf hub board +:width: 360px + +One leaf hub: power in, upstream to a root port, seven XH2.54 ports out. +``` + +```{figure} ../assets/hil/cable-xh254.jpg +:alt: XH2.54 to USB-C pigtail +:width: 240px + +Hub-end XH2.54, board-end USB — Type-C shown, micro-B is the same cable. +``` + +## Proxmox host + +### 1. BIOS + +Enable SVM (or VT-x/VT-d), IOMMU, and *Above 4G decoding*. + +### 2. Kernel command line + +In `/etc/default/grub`, then `update-grub`: + +``` +GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt pcie_acs_override=downstream,multifunction" +``` + +`pcie_acs_override` is required because the card's four controllers sit behind its own +PCIe switch, and that switch does not advertise ACS. Without the override all four land +in one IOMMU group and none can be passed through individually. It relaxes DMA isolation +between them — fine on a dedicated test rig, not on a shared host. Note it is a +Proxmox-kernel patch, not mainline: a stock kernel ignores it silently. + +### 3. Bind the controllers to vfio-pci at boot + +`/etc/modules`: + +``` +vfio +vfio_iommu_type1 +vfio_pci +``` + +`/etc/modprobe.d/vfio.conf`: + +``` +options vfio-pci ids=1912:0014 +softdep xhci_pci pre: vfio-pci +softdep xhci_pci_renesas pre: vfio-pci +``` + +Bind at boot, ahead of the host's xhci driver — do not rely on Proxmox's late binding. +If the host ever owns these ports, the constant failed enumerations from the boards keep +udev busy past 120 s, `udevadm settle` times out inside `ifupdown2-pre`, +`networking.service` is cancelled, and the host comes up with no network. + +Then `update-initramfs -u -k all`, reboot, and check: + +```bash +lspci -nnk -d 1912:0014 | grep -i 'kernel driver' # vfio-pci +``` + +### 4. Pass the controllers to the VM + +One `hostpci` entry per controller, not per card — take the BDFs from +`lspci -nn -d 1912:0014`: + +```bash +qm set <vmid> --machine q35 --cpu host \ + --hostpci0 0000:07:00,pcie=1 --hostpci1 0000:08:00,pcie=1 \ + --hostpci2 0000:09:00,pcie=1 --hostpci3 0000:0a:00,pcie=1 +``` + +`qm config <vmid>` should then list all four. + +## Guest + +Debian 13, 16 vCPU, 18 GB RAM. + +### Renesas firmware + +The controllers' ROM firmware is not reliable under HIL churn: Address Device fails with +`unexpected setup address command completion code 0x11`, and the controller eventually +dies outright (`xHCI host controller not responding, assume dead`). Install Renesas +firmware 2.0.2.6, which the kernel loads into the controller at boot. + +Do this on the kernel that *binds* the controllers — with passthrough that is the guest, +not the Proxmox host. + +1. Download 2.0.2.6 from [station-drivers][fw-dl]. It arrives as `k2026fwup1.exe`, a + Windows self-extracting installer of 1,895,424 bytes. Verify the firmware it contains, + not the installer — the md5 in the next step is the one that matters. +2. Unpack it — despite the name, the firmware inside is called `UPDATE.mem`: + + ```bash + 7z x k2026fwup1.exe -oupd # or: cabextract -d upd k2026fwup1.exe + md5sum upd/UPDATE.mem # 11b49c68a400564b704c6ef17a0e6c0a, 13012 bytes + ``` + +3. Install it under the name the kernel looks for, and rebuild the initramfs + (`xhci-pci-renesas` lives there): + + ```bash + sudo install -m 644 upd/UPDATE.mem /lib/firmware/renesas_usb_fw.mem + sudo update-initramfs -u -k all + sudo reboot + ``` + +4. Confirm the controller is running it. The first check is the one + `test/hil/usbtest.py` gates its own battery on — anything lower and it refuses to + run, failing that board's `usbtest` cell: + + ```bash + sudo setpci -s <bdf> 0x6c.l # whole dword, must be >= 00202609 + dmesg | grep 'hcc params' # 0x014051cf = firmware loaded, 0x014050cf = ROM fallback + ``` + +The kernel reloads the firmware on every power cycle, so the file must stay installed — +that is what the initramfs step is for. The uPD720202 (`1912:0015`) takes the same +firmware and the same check. + +A one-off `soft lockup` warning in `renesas_fw_download_image` while the firmware is +written is expected — it busy-waits over PCI config space for ~30 s. + +[fw-dl]: https://www.station-drivers.com/index.php?option=com_remository&Itemid=353&func=fileinfo&id=1348&lang=en + +### Software + +| Purpose | What `ci` uses | +|-----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------| +| Build | `cmake`, `ninja-build`, and a toolchain per family: `gcc-arm-none-eabi`, a RISC-V GCC, ESP-IDF | +| Flashing | Five tools, one per `Flasher` value — see below | +| Test harness | `pip install -r test/hil/requirements.txt` — hidapi, pyserial, esptool | +| Host-side test tools | `dfu-util`, `mtools`, `libmtp9`, `libmtp-runtime`, `alsa-utils` (apt) — the DFU, MSC, MTP and audio tests shell out to these | +| USB inspection and recovery | `pciutils` (the `usbtest` firmware gate), `uhubctl` (apt), `tshark` for usbmon capture, `testusb` from the kernel's `tools/usb/testusb.c` | + +The `Flasher` column in Attached boards names one of five values; only the ones your own +boards use have to be installed. The mapping is not always guessable: + +| `Flasher` | Binary | +|------------|-------------------------------------------------------------------------------------------------------------------------------------------------------| +| `jlink` | `JLinkExe`, from the SEGGER J-Link software | +| `stlink` | `STM32_Programmer_CLI`, from STM32CubeProgrammer — **not** `st-flash` | +| `openocd` | [`hathach/openocd`][openocd-fork] branch `tinyusb` — one build merging the Raspberry Pi (RP2350), WCH and Analog Devices (MAX32) forks, none upstream | +| `esptool` | `esptool` (pip) | +| `lm4flash` | `lm4flash` (apt) | + +[openocd-fork]: https://github.com/hathach/openocd/tree/tinyusb + +### Permissions and tools + +```bash +sudo cp tools/88-tinyusb.rules /etc/udev/rules.d/ +sudo udevadm control --reload-rules && sudo udevadm trigger +# the groups 88-tinyusb.rules assigns; skip any the distro does not have +# (`wireshark` only exists once wireshark-common is installed) +for g in adm dialout plugdev users wireshark; do + getent group "$g" >/dev/null && sudo usermod -aG "$g" "$USER" +done +``` + +Add the vendor rules for the probes you use (J-Link, picotool). `uhubctl` needs one too +and no package ships it — without it every port toggle wants root: + +``` +# /etc/udev/rules.d/52-uhubctl.rules - root hubs, plus each hub vendor in the rig +SUBSYSTEM=="usb", ATTR{idVendor}=="1d6b", MODE="0664", GROUP="plugdev" +SUBSYSTEM=="usb", ATTR{idVendor}=="1a40", MODE="0664", GROUP="plugdev" +SUBSYSTEM=="usb", ATTR{idVendor}=="045b", MODE="0664", GROUP="plugdev" +``` + +Flasher CLIs and toolchains must be reachable from *non-interactive* shells — neither the +Actions runner nor `hil_ci.sh` sources a login profile. Keep them in `~/.local/bin` and +`~/bin` (symlinks are fine) and add both to the runner's `.path`. + +`pciutils` and passwordless sudo are hard requirements, not conveniences: +`test/hil/usbtest.py` shells out as `sudo -n` for `setpci`, `modprobe`, `dmesg` and +`testusb`, and exits outright if it cannot read the host controller's firmware version. +`helper/hil_pool_check.py` gates recovery on the same `sudo -n` plus +`.claude/skills/usb-kernel-recover/scripts/usb_recover.sh` being present; without both it +cannot re-authorize a wedged probe's port and files the board `flash-failed` instead. + +The `usbtest` battery additionally needs `testusb` built from the kernel tools and +`CONFIG_USB_TEST=m` available. + +## USB topology + +**One 7-port hub per uPD720201 root port. Never chain hubs.** + +Each controller presents four root ports (on both its USB 2 and USB 3 root hubs); the +card brings 12 of those 16 out to connectors. Hang exactly one leaf hub on a root port. + +Boards are grouped into storage boxes, each holding **two** leaf hubs: one carries only +debug probes, the other only the boards under test. Keeping them apart is what makes +recovery tractable — a DUT re-enumerates constantly and can wedge its hub, while the +probes stay on a bus that never moves, so the probe you need to reset a hung board is +still there when you reach for it. + +```{figure} ../assets/hil/storage-box.jpg +:alt: A storage box of boards, probes and two leaf hubs +:width: 800px + +One box: boards, their probes, and the two leaf hubs serving them. +``` + +Boards that run **host** or **dual** tests additionally need a USB peripheral plugged +into the board's *own* USB port — a USB-serial adapter and/or a flash drive for the host +stack to enumerate. Ten `ci` boards have these, recorded as `dev_attached` in the rig +config and matched by exact VID:PID and serial, so a substitute part means updating the +config. The two Espressif +boards also use a TS3USB30 mux to drive device and host tests through one connector. + +Why the rule matters: + +- **Bandwidth.** Every leaf hub gets its own 480 Mbit uplink to the controller. Chaining + puts a second hub's whole subtree behind one of those uplinks, and the `usbtest` + battery saturates whatever it is given. +- **Blast radius.** A board that wedges its hub costs seven ports, not the rig. +- **Scheduling.** `hil_test.py` budgets flashing and `usbtest` concurrency per host + controller (`test/hil/helper/hil_lock.py`: `FLASH_PARALLEL`, `USBTEST_PARALLEL`), which + only means anything when a controller's set of devices is fixed. + +Bus numbers are *not* stable across reboots or recabling, so nothing in the harness +addresses a board by bus path. Boards are identified by the MCU's unique ID and probes by +their serial, both recorded in the rig config — which is why every HIL board must +implement `board_get_unique_id()`. + +## Attached boards + +Roles come from each board's `tests` entry; `Flasher` is the tool that programs it. +Both files are the source of truth — this table is generated from them. + +```{include} hil_boards.md +``` + +## How CI runs the tests + +1. `hil-build` and `hil-build-esp` build the examples on GitHub-hosted runners and upload + the binaries as artifacts. +2. `hil-tinyusb` runs on the self-hosted rigs, downloads those artifacts and calls + `test/hil/hil_test.py`, which flashes each board and runs its tests. Espressif boards + run in `hil-tinyusb-esp`, gated on the slower ESP-IDF build, and `hil-hfp-iar` builds + with IAR inside the job. +3. On pull requests, `test/hil/helper/hil_select.py` narrows the run to the boards a diff + can affect, falling open to the full matrix when it cannot tell. +4. Each board is arbitrated by a kernel flock in `/tmp/tinyusb-hil-locks/`, so interactive + work and CI can share the rig without colliding. +5. Each rig job uploads its report as an artifact; `pr_comment.yml` downloads them and + posts the combined tables onto the pull request. + +From a development PC, the same run can be driven remotely. `REMOTE` and `CONFIG` +default to `ci`, so point them at your own: + +```bash +REMOTE=myrig.lan CONFIG=$PWD/test/hil/local.json bash test/hil/hil_ci.sh -b <board> +``` + +## Gotchas + +- **The Renesas firmware is not optional.** On ROM firmware these controllers fail Address + Device and eventually die under test churn. +- **Port power is logical only.** `uhubctl` "off" on these controllers drops D+/D− but leaves + VBUS hot — boards stay powered and running. Real per-port power switching needs the + controller's PPON pins wired to load switches, which the card omits. +- **Use `uhubctl -S` on root ports.** Without it, uhubctl writes sysfs `disable`, which + takes the root hub's lock — and if anything in that subtree is in D state it blocks + there, leaving the whole bus untouchable. `-S` forces the libusb path instead, which is + why `usb_recover.sh root-cycle` uses it. Resetting the board through its debug probe is + the surer cure, but a wedged *probe* has none, so the port-side drop is the only lever + left there. +- **Park firmware must busy-spin, never `wfe`/`wfi`.** A parked core in a low-power state + can make SWD unreachable and leave the board needing recovery. +- **Most "7-port" hubs are two 4-port hubs in series.** Commodity 7-port hubs commonly + cascade two controllers internally — three ports on the first, four behind a second. + `lsusb -t` tells you which you bought: a single-tier hub appears as one device with + seven ports, a cascaded one shows a hub inside a hub. Every hub on `ci` sits directly + under a root port and reports `maxchild=7`. +- **Size the hub supplies.** Boards take VBUS from the leaf hub, so a seven-board hub on + an undersized supply browns out under load. + +## A minimal rig + +None of the above is a prerequisite. The VM, the uPD720201 cards and the leaf hubs are +what let one machine hold 27 boards and recover them unattended — the harness itself runs +fine against boards plugged straight into a development PC's own USB ports, on whatever +xHCI that PC already has. All it takes is the boards, their debug probes, and a +`test/hil/local.json` describing them in the same shape as `tinyusb.json`. + +Host-side prerequisites, beyond a cross toolchain: + +```bash +python3 tools/get_deps.py <family> # MCU SDKs for your boards +pip install -r test/hil/requirements.txt # hidapi, pyserial, esptool +sudo apt install cmake ninja-build uhubctl \ + dfu-util mtools libmtp9 libmtp-runtime alsa-utils +``` + +`cmake` and `ninja-build` are needed by any run and `uhubctl` by recovery; the rest only +by the tests that shell out to them, so dropping one just fails the DFU, MSC, MTP or audio +cells on an otherwise healthy rig. `test/hil/requirements.txt` names those at the top, +along with `iperf` for the `device/net_lwip_*` tests, which are off in the default matrix. + +Only two of this page's host-controller concerns carry over. `test/hil/usbtest.py` refuses +a DUT behind a MosChip MCS9990 (`9710:9990`) outright, and it applies the Renesas firmware +check only when the DUT really is behind a uPD720201/02 — on a stock Intel or AMD xHCI +there is nothing to install, and `pciutils` is only needed for that check. + +```bash +cd examples && cmake --preset <board> && cmake --build --preset <board> +cd .. && python3 test/hil/hil_test.py -B examples test/hil/local.json +``` diff --git a/docs/reference/hil_boards.md b/docs/reference/hil_boards.md new file mode 100644 index 000000000..e8f364646 --- /dev/null +++ b/docs/reference/hil_boards.md @@ -0,0 +1,45 @@ +<!-- Generated by tools/gen_doc.py - do not edit. --> + +### ci rig + +27 boards, from `test/hil/tinyusb.json`. + +| Board | Roles | Flasher | Variants | Note | +|--------------------------|--------------------|-----------|--------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| frdm_k64f | host | jlink | | | +| ek_tm4c123gxl | device | lm4flash | | | +| espressif_p4_function_ev | device, host | esptool | espressif_p4_function_ev, espressif_p4_function_ev-DMA | Use TS3USB30 mux to test both device and host | +| espressif_s3_devkitm | device, host | esptool | espressif_s3_devkitm, espressif_s3_devkitm-DMA | Use TS3USB30 mux to test both device and host | +| feather_nrf52840_express | device | jlink | | | +| max32666fthr | device | openocd | | | +| metro_m4_express | device, dual | jlink | | pl23x; audio_test_freertos skipped: samd51 iso-IN capture fails (arecord EIO) | +| lpcxpresso11u37 | device | jlink | | | +| lpcxpresso55s28 | device | jlink | | | +| ra4m1_ek | device | jlink | | | +| raspberry_pi_pico | device, host, dual | openocd | raspberry_pi_pico | | +| raspberry_pi_pico_w | host | openocd | | Test native host | +| raspberry_pi_pico2 | host | openocd | | | +| adafruit_fruit_jam | device, host, dual | openocd | | | +| stm32f072disco | device | jlink | | 2x16 access scheme with 1KB USB SRAM | +| stm32f407disco | device | jlink | | | +| stm32f723disco | device, host | jlink | stm32f723disco, stm32f723disco-DMA | Device port0 FS (slave only), Host port1 HS with DMA | +| stm32h743nucleo | device | stlink | stm32h743nucleo, stm32h743nucleo-DMA | | +| stm32g0b1nucleo | device | stlink | | 32-bit scheme, 2KB USB SRAM | +| stm32l476disco | device | jlink | | | +| stm32u083nucleo | device | stlink | | | +| nanoch32v203 | device | openocd | nanoch32v203-fsdev, nanoch32v203-usbfs | | +| ch32v103r_r1_1v0 | device | openocd | | | +| ch32v307v_r1_1v0 | device | openocd | ch32v307v_r1_1v0-usbhs, ch32v307v_r1_1v0-usbfs | | +| ch582m_evt | device | openocd | | | +| mimxrt1064_evk | device, host, dual | jlink | | | +| nrf54lm20dk | device | jlink | | board new to HIL: audio_test_freertos never reaches dcd_init (FreeRTOS itself runs; cdc_msc_freertos and usbtest pass) - example-level issue on nRF54L, fix separately | + +### hfp rig + +3 boards, from `test/hil/hfp.json`. + +| Board | Roles | Flasher | Variants | Note | +|-----------------|---------|-----------|------------------------------------|--------| +| stm32l412nucleo | device | stlink | | | +| stm32f746disco | device | stlink | stm32f746disco, stm32f746disco-DMA | | +| lpcxpresso43s67 | device | jlink | | | diff --git a/docs/reference/index.rst b/docs/reference/index.rst index c66ce618f..85fd767ea 100644 --- a/docs/reference/index.rst +++ b/docs/reference/index.rst @@ -14,4 +14,5 @@ Complete reference documentation for TinyUSB APIs, configuration, and supported dependencies concurrency device_issues + hardware-in-the-loop glossary |
