test/hil, ci: contain a wedged USB stack instead of stranding the runner

A wedged USB device used to take the whole HIL run with it. Every worker that
touched the poisoned node blocked uninterruptibly, the pool could not be joined,
map_async discarded every board's result, and the job ran to the GitHub ceiling
with no report at all -- while the self-hosted runner's single job slot stayed
occupied and every queued job waited behind it.

Bound the calls a worker makes itself. read_sysfs, bounded_open and run_cmd all
answer within a wall clock; read_sysfs distinguishes "absent" from "unknown",
because a blocked read is not evidence of absence, and caps stranded readers at
four (each costs a thread and an fd for the life of the process) after which the
worker declares itself blind. mtype, the gio unmount, the libmtp session and the
arecord/iperf reaps go through those bounds; the MTP session runs in a disposable
subprocess, since libmtp's ctypes calls block unkillably in D state.

Bound the run. A pool guard (HIL_POOL_TIMEOUT, 60 min) fires before any job
ceiling and still writes a report. When the pool will not shut down, the sweep
kills what the workers spawned -- descendants, not just direct children, since
flashers run in their own session -- confirms each kill actually landed, and
exits early so the runner is freed. Whatever survived is named in the report.

Deliberately shallow past that point. We do not re-scan process groups, prove
pid ownership, or escalate through sudo: a root-owned survivor is reported, not
force-killed, because signalling a pid we cannot prove is ours is the worse
failure, and the job ceiling backstops whatever this misses. A D-state holder
was never killable anyway.

Recover instead of reporting a wedge. A HUNG usbtest case reflashes its own DUT
through its roster flasher, but only where the flasher can reach its probe past
a poisoned node -- openocd pinned to a validated vid_pid, or esptool. Where it
cannot, the run says so rather than reserving budget for a path that cannot fire.

Raise the CI ceilings above the pool guard so the guard fires first and still
writes its report, and pin --retry 1 on every HIL leg: the guard is a flat
constant and does not scale with max_retry, so argparse's default of 3 would
triple the serialized usbtest tail against an unchanged guard.

Split the module: execution in hil_test/hil_flash/usbtest, infrastructure in
helper/ (locking, health, selection, shared bounded IO), and the two matrix
generators into .github/scripts/ -- ci_set_matrix.py sat in workflows/, where
GitHub treats every file as a workflow definition. 193 tests cover the bounded
paths, the kill ladder, the guard and the selector against synthetic /proc trees
and PATH-injected fakes; a real wedge cannot be manufactured on demand.

2 files changed, 203 insertions, 0 deletions

diff --git a/.github/scripts/ci_set_matrix.py b/.github/scripts/ci_set_matrix.py
new file mode 100755
index 000000000..50ada5964
--- /dev/null
+++ b/.github/scripts/ci_set_matrix.py
@@ -0,0 +1,111 @@
+#!/usr/bin/env python3
+import json
+
+# toolchain, url
+toolchain_list = [
+    "aarch64-gcc",
+    "arm-clang",
+    "arm-iar",
+    "arm-gcc",
+    "esp-idf",
+    "ft9xx-gcc",
+    "msp430-gcc",
+    "riscv-gcc",
+    "rx-gcc"
+]
+
+# family: [supported toolchain]
+family_list = {
+    "apm32f0xx": ["arm-gcc"],
+    "at32f402_405": ["arm-gcc"],
+    "at32f403a_407": ["arm-gcc"],
+    "at32f413": ["arm-gcc"],
+    "at32f415": ["arm-gcc"],
+    "at32f423": ["arm-gcc"],
+    "at32f425": ["arm-gcc"],
+    "at32f435_437": ["arm-gcc"],
+    "at32f45x": ["arm-gcc"],
+    "broadcom_32bit": ["arm-gcc"],
+    "broadcom_64bit": ["aarch64-gcc"],
+    "ch32f20x": ["arm-gcc"],
+    "ch32v10x": ["riscv-gcc"],
+    "ch32v20x": ["riscv-gcc"],
+    "ch32v30x": ["riscv-gcc"],
+    "ch583": ["riscv-gcc"],
+    "da1469x": ["arm-gcc"],
+    "fomu": ["riscv-gcc"],
+    "ft9xx": ["ft9xx-gcc"],
+    "gd32vf103": ["riscv-gcc"],
+    "hpmicro": ["riscv-gcc"],
+    "imxrt": ["arm-gcc", "arm-clang"],
+    "kinetis_k": ["arm-gcc"],
+    "kinetis_k32l": ["arm-gcc"],
+    "kinetis_kl": ["arm-gcc"],
+    "lpc11": ["arm-gcc", "arm-clang"],
+    "lpc13": ["arm-gcc", "arm-clang"],
+    "lpc15": ["arm-gcc", "arm-clang"],
+    "lpc17": ["arm-gcc", "arm-clang"],
+    "lpc18": ["arm-gcc", "arm-clang"],
+    "lpc40": ["arm-gcc", "arm-clang"],
+    "lpc43": ["arm-gcc", "arm-clang"],
+    "lpc51": ["arm-gcc", "arm-clang"],
+    "lpc54": ["arm-gcc", "arm-clang"],
+    "lpc55": ["arm-gcc", "arm-clang"],
+    "maxim": ["arm-gcc"],
+    "mcx": ["arm-gcc"],
+    "mm32": ["arm-gcc"],
+    "msp430": ["msp430-gcc"],
+    "msp432e4": ["arm-gcc"],
+    "nrf": ["arm-gcc", "arm-clang"],
+    "nuc100_120": ["arm-gcc"],
+    "nuc121_125": ["arm-gcc"],
+    "nuc126": ["arm-gcc"],
+    "nuc505": ["arm-gcc"],
+    "ra": ["arm-gcc"],
+    "rp2040": ["arm-gcc"],
+    "rw61x": ["arm-gcc"],
+    "rx": ["rx-gcc"],
+    "samd11": ["arm-gcc", "arm-clang"],
+    "samd2x_l2x": ["arm-gcc", "arm-clang"],
+    "samd5x_e5x": ["arm-gcc", "arm-clang"],
+    "samg": ["arm-gcc", "arm-clang"],
+    "stm32c0": ["arm-gcc", "arm-clang", "arm-iar"],
+    "stm32c5": ["arm-gcc", "arm-clang", "arm-iar"],
+    "stm32f0": ["arm-gcc", "arm-clang", "arm-iar"],
+    "stm32f1": ["arm-gcc", "arm-clang", "arm-iar"],
+    "stm32f2": ["arm-gcc", "arm-clang", "arm-iar"],
+    "stm32f3": ["arm-gcc", "arm-clang", "arm-iar"],
+    "stm32f4": ["arm-gcc", "arm-clang", "arm-iar"],
+    "stm32f7": ["arm-gcc", "arm-clang", "arm-iar"],
+    "stm32g0": ["arm-gcc", "arm-clang", "arm-iar"],
+    "stm32g4": ["arm-gcc", "arm-clang", "arm-iar"],
+    "stm32h5": ["arm-gcc", "arm-clang", "arm-iar"],
+    "stm32h7": ["arm-gcc", "arm-clang", "arm-iar"],
+    "stm32h7rs": ["arm-gcc", "arm-clang", "arm-iar"],
+    "stm32l0": ["arm-gcc", "arm-clang", "arm-iar"],
+    "stm32l4": ["arm-gcc", "arm-clang", "arm-iar"],
+    "stm32n6": ["arm-gcc"],
+    "stm32u0": ["arm-gcc", "arm-clang", "arm-iar"],
+    "stm32u5": ["arm-gcc", "arm-clang", "arm-iar"],
+    "stm32wb": ["arm-gcc", "arm-clang", "arm-iar"],
+    "stm32wba": ["arm-gcc", "arm-clang", "arm-iar"],
+    "tm4c": ["arm-gcc"],
+    "xmc4000": ["arm-gcc"],
+    # S3, P4 will be built by hil test
+    # "-bespressif_s3_devkitm": ["esp-idf"],
+    # "-bespressif_p4_function_ev": ["esp-idf"],
+}
+
+
+def set_matrix_json():
+    matrix = {}
+    for toolchain in toolchain_list:
+        filtered_families = [family for family, supported_toolchain in family_list.items() if
+                             toolchain in supported_toolchain]
+        matrix[toolchain] = filtered_families
+
+    print(json.dumps(matrix))
+
+
+if __name__ == '__main__':
+    set_matrix_json()
diff --git a/.github/scripts/hil_ci_set_matrix.py b/.github/scripts/hil_ci_set_matrix.py
new file mode 100644
index 000000000..65f50788e
--- /dev/null
+++ b/.github/scripts/hil_ci_set_matrix.py
@@ -0,0 +1,92 @@
+import argparse
+import json
+import os
+
+
+def _resolve_config_path(config_file):
+    if os.path.exists(config_file):
+        return config_file
+
+    # bare roster names resolve against the repo's test/hil (this script lives in
+    # .github/scripts); build.yml passes explicit paths, this is for hand-runs
+    repo_relative = os.path.join(os.path.dirname(__file__), '..', '..', 'test', 'hil', config_file)
+    if os.path.exists(repo_relative):
+        return repo_relative
+
+    raise FileNotFoundError(f'Config file not found: {config_file}')
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('config_files', nargs='+', help='Configuration JSON file(s)')
+    parser.add_argument('--select', help='hil_select.py JSON; scopes boards when full=false')
+    args = parser.parse_args()
+
+    selected = None
+    sel = json.loads(args.select) if args.select else None
+    if sel and not sel.get('full'):
+        selected = set(sel.get('boards', {}))
+
+    # Toolchain buckets must match the toolchains instantiated by the hil-build
+    # job in .github/workflows/build.yml. Keep all keys present (even if empty)
+    # so `fromJSON(hil_json)[toolchain]` always resolves to a list.
+    matrix = {
+        'arm-gcc': [],
+        'riscv-gcc': [],
+        'esp-idf': []
+    }
+
+    seen = {toolchain: set() for toolchain in matrix}
+
+    def append_build_arg(toolchain, build_arg):
+        if build_arg not in seen[toolchain]:
+            seen[toolchain].add(build_arg)
+            matrix[toolchain].append(build_arg)
+
+    for config_file in args.config_files:
+        with open(_resolve_config_path(config_file)) as f:
+            config = json.load(f)
+
+        for board in config['boards']:
+            if selected is not None and board['name'] not in selected:
+                continue
+            name = board['name']
+            flasher = board['flasher']
+            # esptool boards must build under esp-idf; others default to arm-gcc
+            # but may opt into another bucket via an explicit "toolchain" field
+            # (e.g. RISC-V boards like ch32v20x need "riscv-gcc").
+            if flasher['name'] == 'esptool':
+                toolchain = 'esp-idf'
+            else:
+                toolchain = board.get('toolchain', 'arm-gcc')
+            if toolchain not in matrix:
+                # a board in no bucket would never be built, and the bare KeyError
+                # below would only say so as a traceback from the set-matrix job
+                raise SystemExit(
+                    f'{name}: toolchain {toolchain!r} is not a build bucket '
+                    f'({", ".join(matrix)}); add it here and to the hil-build / '
+                    f'hil-build-esp jobs in .github/workflows/build.yml')
+
+            build_board = f'-b {name}'
+            if 'build' in board and 'args' in board['build']:
+                build_board += ' ' + ' '.join(f'-D{a}' for a in board['build']['args'])
+
+            # Each variant builds into cmake-build-<variant.name> with its own cmake
+            # -D defines and raw CFLAGS. No 'variant' -> a single build named after
+            # the board.
+            variants = board.get('variant') or [{'name': name, 'flags': ''}]
+            for v in variants:
+                arg = build_board
+                if v['name'] != name:
+                    arg += f' --build-name {v["name"]}'
+                for d in v.get('defines', []):
+                    arg += f' -D{d}'
+                for tok in v.get('flags', '').split():
+                    arg += f' --cflag={tok}'
+                append_build_arg(toolchain, arg)
+
+    print(json.dumps(matrix))
+
+
+if __name__ == '__main__':
+    main()