hil: controller-aware scheduling of flash and usbtest concurrency

Full-fleet profiling (HIL_PROFILE=1 instrumentation, included) showed each
uPD720201 controller's serialized usbtest battery chain dominates wall time,
and a board whose marginal device port bounces during concurrent batteries
can wedge or kill the controller ("xHCI host not responding to stop endpoint
command"). Every such death traced to mimxrt1015's port (its old "kills the
uPD720201" reputation) - it is removed from the config until recabled;
mimxrt1064's enum-retry stalls were a loose device cable (re-seated).
nrf54lm20dk moves to boards-skip until its failing J-Link probe is replugged.
With the hardware fixed both cards run width-4 batteries plus full flash
churn clean, so scheduling stays simple: two symmetric knobs, flashes and
batteries budgeted per controller.

- schedule_boards(): dispatch boards round-robin across host controllers from
  a persisted hint cache (~/.cache/tinyusb-hil/ctrl_cache.json), learned and
  merge-on-write refreshed each run (concurrent HIL jobs keep each other's
  entries). Only the cached PCI address is consumed - dispatch order and
  first-flash budgeting, never battery serialization (batteries resolve live
  or fail closed to an all-slot permit).
- HIL_FLASH_PARALLEL (8) and HIL_USBTEST_PARALLEL (4) are budgeted per
  controller via lock slots assigned on first sight.
- re-runs: a failed run writes <report dir>/<config>.failed with the exact
  re-run spec (--accumulate -b <failed board> -bt <board>:<its failed
  tests>) instead of the inverted --skip-board list of everything that
  passed; --skip-board is gone, --flasher/--exclude-flasher scope a config
  across CI jobs by flasher type (no board names hardcoded in workflows),
  and -a/--accumulate merges a re-run into the existing report. The spec is
  stamped with GITHUB_RUN_ID and cleared on fresh runs, so a retry can never
  consume a spec left behind by a different run's dead or skipped attempt.
- CI: esp-idf firmware builds move out of hil-build into hil-build-esp, and
  the esptool-flashed boards run in their own hil-tinyusb-esp job, so the
  main hil-tinyusb run starts as soon as the fast toolchains finish instead
  of waiting on the slow esp-idf build (an esp toolchain flake previously
  skipped the whole rig run). Artifacts are namespaced per toolchain so the
  esp job downloads only esp-idf binaries.
- HIL_PROFILE=1: timestamped log lines, per-flash durations, permit-wait
  logging, uid->controller map dump for analysis.
- hil_report: per-variant test duration as a dedicated trailing column,
  recorded only by full runs.

Validated on the ci rig (fixed seeds 20260716/777, full fleet at 8/4):
738s/780s walls with only known-flake failures and no controller deaths,
vs 1134-1211s serialized-battery baseline.

2 files changed, 88 insertions, 11 deletions

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index f24f3ae1f..818e7ba81 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -274,13 +274,25 @@ jobs:
         toolchain:
           - 'arm-gcc'
           - 'riscv-gcc'
-          - 'esp-idf'
     with:
       build-system: 'cmake'
       toolchain: ${{ matrix.toolchain }}
       build-args: ${{ toJSON(fromJSON(needs.set-matrix.outputs.hil_json)[matrix.toolchain]) }}
       upload-artifacts: true
 
+  # esp-idf builds are by far the slowest; keep them out of hil-build so the main
+  # hil-tinyusb run starts as soon as the fast toolchains finish (esp boards get
+  # their own hil-tinyusb-esp run gated only on this job)
+  hil-build-esp:
+    needs: [ check-paths, set-matrix ]
+    if: needs.check-paths.outputs.code_changed == 'true' && github.repository_owner == 'hathach'
+    uses: ./.github/workflows/build_util.yml
+    with:
+      build-system: 'cmake'
+      toolchain: 'esp-idf'
+      build-args: ${{ toJSON(fromJSON(needs.set-matrix.outputs.hil_json)['esp-idf']) }}
+      upload-artifacts: true
+
   # ---------------------------------------
   # Hardware in the loop (HIL)
   # self-hosted on local VM, for attached hardware checkout HIL_JSON
@@ -295,9 +307,13 @@ jobs:
           - display: tinyusb.json
             runner: [ self-hosted, X64, hathach, hardware-in-the-loop ]
             hil_json: test/hil/tinyusb.json
+            # esptool-flashed (espressif) boards run in hil-tinyusb-esp,
+            # gated on the slow esp-idf build
+            test_args: '--exclude-flasher esptool'
           - display: hfp.json
             runner: [ self-hosted, Linux, X64, hifiphile ]
             hil_json: test/hil/hfp.json
+            test_args: ''
     runs-on: ${{ matrix.runner }}
     env:
       HIL_JSON: ${{ matrix.hil_json }}
@@ -305,16 +321,19 @@ jobs:
       - name: Set HIL report dir (sibling of workspace; persists across run attempts)
         run: echo "HIL_REPORT_DIR=$(dirname "$GITHUB_WORKSPACE")/hil-report" >> "$GITHUB_ENV"
 
-      - name: Get Skip Boards from previous run
+      - name: Get re-run spec from previous attempt
         if: github.run_attempt != '1'
         run: |
-          if [ -f "${{ env.HIL_JSON }}.skip" ]; then
-            SKIP_BOARDS=$(cat "${{ env.HIL_JSON }}.skip")
+          # only honor a spec stamped by THIS run: a spec left by another run (attempt 1
+          # died or was skipped before hil_test.py could clear it) must not be consumed
+          SPEC="$HIL_REPORT_DIR/$(basename "${{ env.HIL_JSON }}").failed"
+          if [ -f "$SPEC" ] && [ "$(cat "$SPEC.run" 2>/dev/null)" = "$GITHUB_RUN_ID" ]; then
+            RERUN_ARGS=$(cat "$SPEC")
           else
-            SKIP_BOARDS=""
+            RERUN_ARGS=""
           fi
-          echo "SKIP_BOARDS=$SKIP_BOARDS"
-          echo "SKIP_BOARDS=$SKIP_BOARDS" >> $GITHUB_ENV
+          echo "RERUN_ARGS=$RERUN_ARGS"
+          echo "RERUN_ARGS=$RERUN_ARGS" >> $GITHUB_ENV
 
       - name: Clean workspace
         run: |
@@ -335,9 +354,7 @@ jobs:
       - name: Test on actual hardware
         # Single attempt per test (--retry 1), no in-run second pass: a broken fixture
         # fails fast instead of holding the runner (and other PRs' HIL jobs) for hours.
-        # hil_test.py still writes ${HIL_JSON}.skip, so a manual re-run attempt only
-        # retests what failed (see "Get Skip Boards from previous run").
-        run: python3 test/hil/hil_test.py --retry 1 ${{ env.HIL_JSON }} $SKIP_BOARDS
+        run: python3 test/hil/hil_test.py --retry 1 ${{ matrix.test_args }} ${{ env.HIL_JSON }} $RERUN_ARGS
 
       - name: Upload HIL report
         if: always() && github.event_name == 'pull_request'
@@ -349,6 +366,66 @@ jobs:
           overwrite: true
 
   # ---------------------------------------
+  # Hardware in the loop (HIL) - espressif boards only
+  # Same rig as hil-tinyusb (tinyusb.json) but gated only on the slow esp-idf build,
+  # so the main run does not wait for it. Per-board flocks arbitrate the shared rig;
+  # the runner has a single job slot, so the two HIL jobs never overlap - adding a
+  # second slot would double the per-controller flash/usbtest budgets.
+  # ---------------------------------------
+  hil-tinyusb-esp:
+    needs: hil-build-esp
+    name: hil-tinyusb (tinyusb-esp.json)
+    runs-on: [ self-hosted, X64, hathach, hardware-in-the-loop ]
+    env:
+      HIL_JSON: test/hil/tinyusb.json
+      TEST_ARGS: '--flasher esptool'
+    steps:
+      - name: Set HIL report dir (sibling of workspace; persists across run attempts)
+        run: echo "HIL_REPORT_DIR=$(dirname "$GITHUB_WORKSPACE")/hil-report-esp" >> "$GITHUB_ENV"
+
+      - name: Get re-run spec from previous attempt
+        if: github.run_attempt != '1'
+        run: |
+          # only honor a spec stamped by THIS run: a spec left by another run (attempt 1
+          # died or was skipped before hil_test.py could clear it) must not be consumed
+          SPEC="$HIL_REPORT_DIR/$(basename "${{ env.HIL_JSON }}").failed"
+          if [ -f "$SPEC" ] && [ "$(cat "$SPEC.run" 2>/dev/null)" = "$GITHUB_RUN_ID" ]; then
+            RERUN_ARGS=$(cat "$SPEC")
+          else
+            RERUN_ARGS=""
+          fi
+          echo "RERUN_ARGS=$RERUN_ARGS"
+          echo "RERUN_ARGS=$RERUN_ARGS" >> $GITHUB_ENV
+
+      - name: Clean workspace
+        run: |
+          echo "Cleaning up for the first run"
+          rm -rf "${{ github.workspace }}"
+          mkdir -p "${{ github.workspace }}"
+
+      - name: Checkout TinyUSB
+        uses: actions/checkout@v6
+
+      - name: Download Artifacts
+        uses: actions/download-artifact@v5
+        with:
+          pattern: binaries-esp-idf-*
+          path: cmake-build
+          merge-multiple: true
+
+      - name: Test on actual hardware
+        run: python3 test/hil/hil_test.py --retry 1 $TEST_ARGS ${{ env.HIL_JSON }} $RERUN_ARGS
+
+      - name: Upload HIL report
+        if: always() && github.event_name == 'pull_request'
+        uses: actions/upload-artifact@v7
+        with:
+          name: hil-report-tinyusb-esp.json
+          path: ${{ env.HIL_REPORT_DIR }}/hil_report.md
+          if-no-files-found: ignore
+          overwrite: true
+
+  # ---------------------------------------
   # Hardware in the loop (HIL)
   # self-hosted by HFP, build with IAR toolchain, for attached hardware checkout test/hil/hfp.json
   # Since IAR Token secret is not passed to forked PR, only build non-forked PR
diff --git a/.github/workflows/build_util.yml b/.github/workflows/build_util.yml
index 2532caebe..90115862b 100644
--- a/.github/workflows/build_util.yml
+++ b/.github/workflows/build_util.yml
@@ -99,7 +99,7 @@ jobs:
         if: inputs.upload-artifacts == true && inputs.code-changed == true
         uses: actions/upload-artifact@v7
         with:
-          name: binaries-${{ matrix.arg }}
+          name: binaries-${{ inputs.toolchain }}-${{ matrix.arg }}
           path: |
             cmake-build/cmake-build-*/*/*/*.elf
             cmake-build/cmake-build-*/*/*/*.bin