From 3963a1b70a572132aced1c1a0033e1c8249a0c7e Mon Sep 17 00:00:00 2001 From: hathach Date: Fri, 14 Aug 2026 01:08:40 +0700 Subject: test/hil, ci: contain a wedged USB stack instead of stranding the runner A wedged USB device used to take the whole HIL run with it. Every worker that touched the poisoned node blocked uninterruptibly, the pool could not be joined, map_async discarded every board's result, and the job ran to the GitHub ceiling with no report at all -- while the self-hosted runner's single job slot stayed occupied and every queued job waited behind it. Bound the calls a worker makes itself. read_sysfs, bounded_open and run_cmd all answer within a wall clock; read_sysfs distinguishes "absent" from "unknown", because a blocked read is not evidence of absence, and caps stranded readers at four (each costs a thread and an fd for the life of the process) after which the worker declares itself blind. mtype, the gio unmount, the libmtp session and the arecord/iperf reaps go through those bounds; the MTP session runs in a disposable subprocess, since libmtp's ctypes calls block unkillably in D state. Bound the run. A pool guard (HIL_POOL_TIMEOUT, 60 min) fires before any job ceiling and still writes a report. When the pool will not shut down, the sweep kills what the workers spawned -- descendants, not just direct children, since flashers run in their own session -- confirms each kill actually landed, and exits early so the runner is freed. Whatever survived is named in the report. Deliberately shallow past that point. We do not re-scan process groups, prove pid ownership, or escalate through sudo: a root-owned survivor is reported, not force-killed, because signalling a pid we cannot prove is ours is the worse failure, and the job ceiling backstops whatever this misses. A D-state holder was never killable anyway. Recover instead of reporting a wedge. A HUNG usbtest case reflashes its own DUT through its roster flasher, but only where the flasher can reach its probe past a poisoned node -- openocd pinned to a validated vid_pid, or esptool. Where it cannot, the run says so rather than reserving budget for a path that cannot fire. Raise the CI ceilings above the pool guard so the guard fires first and still writes its report, and pin --retry 1 on every HIL leg: the guard is a flat constant and does not scale with max_retry, so argparse's default of 3 would triple the serialized usbtest tail against an unchanged guard. Split the module: execution in hil_test/hil_flash/usbtest, infrastructure in helper/ (locking, health, selection, shared bounded IO), and the two matrix generators into .github/scripts/ -- ci_set_matrix.py sat in workflows/, where GitHub treats every file as a workflow definition. 193 tests cover the bounded paths, the kill ladder, the guard and the selector against synthetic /proc trees and PATH-injected fakes; a real wedge cannot be manufactured on demand. --- .github/workflows/build.yml | 71 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 58 insertions(+), 13 deletions(-) (limited to '.github/workflows/build.yml') diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index bdef81553..8f6014f48 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -41,7 +41,7 @@ jobs: - '.github/actions/**' - '.github/workflows/build.yml' - '.github/workflows/build_util.yml' - - '.github/workflows/ci_set_matrix.py' + - '.github/scripts/**' set-matrix: runs-on: ubuntu-latest @@ -72,10 +72,16 @@ jobs: # so a missing origin/, a shallow-clone hiccup or a selector traceback # must fall back to the FULL matrix (no --select, run=true, no args) instead # of failing the job. Same fail-open shape as pr_comment.yml's `|| true`. + # + # The selector's own unit suite gates it (stdlib-only, seconds): a selector + # whose tests fail can still exit 0 with valid-but-WRONG JSON -- fail-open alone + # never catches that class, and the pre-commit hil-test hook is a separate, + # advisory workflow that nothing here can `needs:`. Test-failing selector => + # full matrix, same as a crashing one. SELECT_JSON='' - if ! python3 test/hil/test_hil_select.py; then - echo "::error::hil_select unit tests failed - falling back to the full HIL matrix" - elif ! SELECT_JSON=$(python3 test/hil/hil_select.py --base "origin/$BASE_REF" test/hil/tinyusb.json test/hil/hfp.json); then + if ! python3 test/hil/test/test_hil_select.py; then + echo "::warning::hil_select unit suite failed - falling back to the full HIL matrix" + elif ! SELECT_JSON=$(python3 test/hil/helper/hil_select.py --base "origin/$BASE_REF" test/hil/tinyusb.json test/hil/hfp.json); then echo "::warning::hil_select failed - falling back to the full HIL matrix" SELECT_JSON='' fi @@ -113,7 +119,7 @@ jobs: SELECT: ${{ steps.hil-select.outputs.select }} run: | # build matrix - MATRIX_JSON=$(python .github/workflows/ci_set_matrix.py) + MATRIX_JSON=$(python .github/scripts/ci_set_matrix.py) echo "matrix=$MATRIX_JSON" echo "matrix=$MATRIX_JSON" >> $GITHUB_OUTPUT @@ -121,13 +127,13 @@ jobs: # Scoping is best-effort too: fall back to the unscoped (full) matrix. HIL_MATRIX_JSON='' if [ -n "$SELECT" ]; then - HIL_MATRIX_JSON=$(python test/hil/hil_ci_set_matrix.py --select "$SELECT" test/hil/tinyusb.json test/hil/hfp.json) || HIL_MATRIX_JSON='' + HIL_MATRIX_JSON=$(python .github/scripts/hil_ci_set_matrix.py --select "$SELECT" test/hil/tinyusb.json test/hil/hfp.json) || HIL_MATRIX_JSON='' if [ -z "$HIL_MATRIX_JSON" ]; then echo "::warning::scoped HIL matrix failed - falling back to the full HIL matrix" fi fi if [ -z "$HIL_MATRIX_JSON" ]; then - HIL_MATRIX_JSON=$(python test/hil/hil_ci_set_matrix.py test/hil/tinyusb.json test/hil/hfp.json) + HIL_MATRIX_JSON=$(python .github/scripts/hil_ci_set_matrix.py test/hil/tinyusb.json test/hil/hfp.json) fi echo "hil_matrix=$HIL_MATRIX_JSON" echo "hil_matrix=$HIL_MATRIX_JSON" >> $GITHUB_OUTPUT @@ -338,7 +344,7 @@ jobs: strategy: fail-fast: false matrix: - # These names are the bucket keys of test/hil/hil_ci_set_matrix.py: every + # These names are the bucket keys of .github/scripts/hil_ci_set_matrix.py: every # non-esptool roster board must land in one of them (esptool boards go to # 'esp-idf', built by hil-build-esp below). hil_ci_set_matrix.py rejects a # board whose "toolchain" is not a bucket, so a new bucket must be added in @@ -379,6 +385,14 @@ jobs: hil-tinyusb: needs: [ hil-build, set-matrix ] name: hil-tinyusb (${{ matrix.display }}) + # Above hil_test.py's pool guard (HIL_POOL_TIMEOUT, 60 min) so the guard fires first + # and still gets to write its report. The 30 min on top is what the job pays OUTSIDE + # the guard clock: workspace cleanup, checkout, the multi-board artifact merge and + # the D-state note before it; kill_worker_children, shutdown_pool's 30 s grace, the + # report write and the upload after it. On a multi-stray convoy that tail alone is + # minutes, and a ceiling below guard+tail cancels the job before hil_report.md exists + # -- the inversion this branch removes. Both legs share the script and the guard. + timeout-minutes: 90 strategy: fail-fast: false matrix: @@ -395,6 +409,9 @@ jobs: test_args: '' runs-on: ${{ matrix.runner }} env: + # HIL_POOL_TIMEOUT deliberately unset: hil_test.py's 60 min default is below every + # ceiling here, so ceiling > guard holds by construction. Pin it to SHORTEN a run + # only -- pinning it above a ceiling re-inverts the two. HIL_JSON: ${{ matrix.hil_json }} steps: - name: Set HIL report dir (per run+job; persists across run attempts) @@ -482,6 +499,11 @@ jobs: needs: [ hil-build-esp, set-matrix ] name: hil-tinyusb (tinyusb-esp.json) runs-on: [ self-hosted, X64, hathach, hardware-in-the-loop ] + # above hil_test.py's pool guard (60 min) with room for the pre-pool checkout + # and the post-guard sweep + report upload, so its own guard still writes a report; + # only a job wedged past that (unkillable D-state worker) hits this ceiling, which + # must exist because the runner has one job slot and holds every queued job hostage + timeout-minutes: 90 env: HIL_JSON: test/hil/tinyusb.json TEST_ARGS: '--flasher esptool' @@ -564,7 +586,13 @@ jobs: github.repository_owner == 'hathach' && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == true) runs-on: [ self-hosted, Linux, X64, hifiphile ] - timeout-minutes: 30 + # Unlike the hil-tinyusb jobs, this one BUILDS with IAR in the same job before running + # hil_test.py -- hfp.json's 3 boards, 4 variant entries, "up to 30 minutes" (see the + # comment above the selection step). The ceiling has to cover build + the 60 min pool + # guard + overhead, or GitHub cancels before the guard can write its report -- the + # inversion this branch removes. 30 + 60 = 90; the remaining 30 is the full-history + # checkout, get_deps, the post-guard sweep and the report upload. + timeout-minutes: 120 env: IAR_LMS_BEARER_TOKEN: ${{ secrets.IAR_LMS_BEARER_TOKEN }} PYTHONUNBUFFERED: '1' @@ -601,7 +629,12 @@ jobs: # failures cannot kill hfp coverage - a selector failure here must likewise # fall back to the full hfp matrix (no hil_select.json, no SEL_* vars), never # fail the job. - if ! python3 test/hil/hil_select.py --base "origin/$BASE_REF" test/hil/hfp.json > hil_select.json; then + if ! python3 test/hil/test/test_hil_select.py; then + echo "::warning::hil_select unit suite failed - running the full hfp matrix" + rm -f hil_select.json + exit 0 + fi + if ! python3 test/hil/helper/hil_select.py --base "origin/$BASE_REF" test/hil/hfp.json > hil_select.json; then echo "::warning::hil_select failed - running the full hfp matrix" rm -f hil_select.json exit 0 @@ -629,9 +662,9 @@ jobs: if: env.SEL_RUN != 'false' run: | if [ -f hil_select.json ]; then - MATRIX_JSON=$(python test/hil/hil_ci_set_matrix.py --select "$(cat hil_select.json)" test/hil/hfp.json) + MATRIX_JSON=$(python .github/scripts/hil_ci_set_matrix.py --select "$(cat hil_select.json)" test/hil/hfp.json) else - MATRIX_JSON=$(python test/hil/hil_ci_set_matrix.py test/hil/hfp.json) + MATRIX_JSON=$(python .github/scripts/hil_ci_set_matrix.py test/hil/hfp.json) fi # Each variant carries its own --build-name/--cflag, which are global to a # single build.py invocation — so keep one matrix entry per line and build @@ -648,6 +681,13 @@ jobs: - name: Build if: env.SEL_RUN != 'false' + # Bounded SEPARATELY from the job. This is the only HIL job that builds inline + # (hil-tinyusb downloads artifacts), and the job ceiling went 30 -> 120 to give the + # HIL step room -- which would hand a stalled IAR build the whole two hours on the + # shared self-hosted runner, never reaching hil_test.py or the report upload. That + # is the stranded-runner-with-no-report failure this branch exists to prevent. + # Typical full build here is a few minutes; 30 leaves generous headroom. + timeout-minutes: 30 run: | readarray -t ENTRIES < hil_build_entries.txt for entry in "${ENTRIES[@]}"; do @@ -666,7 +706,12 @@ jobs: fi # empty/absent on a non-PR event or a selector fallback -> full hfp matrix SEL_ARGS=$(cat hil_sel_args.txt 2>/dev/null || true) - python3 test/hil/hil_test.py $SEL_ARGS hfp.json + # --retry 1, like the other two HIL legs. The pool guard is a FLAT 3600s and + # does NOT scale with max_retry, so argparse's default of 3 would multiply the + # serialized usbtest tail (hfp.json runs four batteries) by three against an + # unchanged guard -- on a runner with a single job slot that queues every other + # job behind it. + python3 test/hil/hil_test.py --retry 1 $SEL_ARGS hfp.json - name: Upload HIL report if: always() && github.event_name == 'pull_request' -- cgit v1.3.1