Unnamed repository; edit this file 'description' to name the repository. http://cgit.235523.xyz/tinyusb.git/atom/docs?h=hil-report 2026-08-21T15:25:13Z 2026-08-21T15:25:13Z hathach thach@tinyusb.org 2026-08-21T15:16:25Z urn:sha1:aa4a07820dd2dd54fa5fef0375c709d67b1c78d0 Records why the report code was spread across three modules and what the consolidation buys, so the next reader does not have to re-derive it from the diff. Names the new functions explicitly rather than presenting the change as pure code motion -- that framing points reviewers away from the code that carried the defects. Three findings this PR deliberately does not close get one handoff each, per CLAUDE.md: the worker-result tuple hil_report still unpacks positionally, SKILL.md's no-boards rule drifting from the code, and write_report's two non-atomic writes. Drops the pr3836 handoff, which this branch implements. 2026-08-21T15:25:13Z hathach thach@tinyusb.org 2026-08-21T15:16:06Z urn:sha1:018fd75693ba9dc99173312909c3c055c8010574 The markdown IS a rendering of the sidecar now: every writer goes through render_report(), so a table can never contain something the JSON does not. Four writers previously composed it independently and three wrote no JSON at all -- and those three are exactly the paths where the run died, so a JSON consumer saw nothing on the runs that mattered most. The per-board verdicts an agent hands back reported the whole fleet as "no report row" while a human reading the markdown saw the real story. The document gains the two fields the markdown carried but the JSON did not: `scope` (a three-board PR run and a full run that lost 24 boards were indistinguishable) and `caveat` (how the run ENDED -- abandoned, aborted, no boards). `banner` keeps its existing meaning: the rig-health conditions the cells were collected under. The distinction is load-bearing, because banner carries across an --accumulate retry and caveat must not; conflating them made a clean retry publish an abandonment that never happened, and let a stale notice from an earlier attempt silence a genuinely new one. Consolidating into helper/hil_report.py is what makes that complete. The renderer, the writers, the merge and the fold to per-board verdicts live in one module that hil_test.py and hil_health.py both import. That dissolves the import cycle which forced write_timeout_report to compose its own markdown -- the pool-guard fallback renders like everyone else, so all five writers are byte-identical -- and removes the second copy of the cell classifier, which hil_summary.py's docstring described as "the EXACT classifier hil_test.py's own tally uses". Two copies of one rule, kept in sync by hand against re-typed emoji literals: change REPORT_CELL and the human's table and the agent's verdict silently disagree. hil_summary.py is deleted; its CLI moves here and the two harness docs that invoke it by name follow. `caveat` gates the workflow verdict and is required by its schema, because on the abandon path every row can legitimately pass while hil_test.py exits non-zero, and an operator omitting the field would silently disable the gate. NOT purely code motion, and worth reading as new code: measured against master, hil_test.py held only render_matrix and accumulate_report. render_report, write_report, mark_report_abandoned, mark_report_no_boards, _load, cell_state and the scope/caveat plumbing are new, and three rounds of review found their defects there. Each was reproduced before being fixed and is pinned by a test: a stuck board that already had a row got no pool-timeout cell and summarized as a pass; a stale board-locked cell masked it, so a board that wedged the rig was published as LOCKED and re-run; mark_report_abandoned republished the markdown even when it declined to stamp, inspected the wrong field, gave up on a missing or torn sidecar, and called the table "partial" against SKILL.md's contract that it IS this run's; write_report swallowed OSError, making two layers of fallback dead code, and committed the JSON before rendering; a sidecar with a null banner or a non-list rows killed a fully successful run with no artifact at all; an unhashable cell value raised on the normal accumulate path; the no-boards exit republished a previous run's rows, and its guard blocked even a fresh run. _load is the trust boundary for all of it, since hil_ci.sh uploads a sidecar as the --accumulate merge base and it is therefore untrusted input. A corrupt cell drops rather than being coerced to str, which would classify it as a pass. Verified on the rig: a 25-board fleet run, ten randomized passes mixing fresh and --accumulate over different board sets, and every containment path exercised against the module actually staged there. 2026-08-21T05:41:47Z hathach thach@tinyusb.org 2026-08-21T05:41:47Z urn:sha1:e13eff8d4e757ebe7709a58fce44017b8be5a84d Every one of these dropped coverage silently - the worst failure mode here, because the PR still goes green. Found by review, each reproduced first. Selection rules: * class_macros derived the config macro from the class DIRECTORY, so a change to src/class/midi/midi2_device.c selected the midi_test examples (which do not compile it) and never examples/device/midi2_device (the only one that enables CFG_TUD_MIDI2, and the only one that does). The file's own macro is unioned in where it differs - union, never replace: over-selecting costs a build, under-selecting merges a break. * the ${FAMILY_MCUS} fallback added for espressif fired on any family whose _family_mcus came back empty, and _cmake_sets is if()-blind and keeps the FIRST definition - so mcx/frdm_mcxn947 answered MCXA15, a token six examples' skip.txt names, dropping 12 firmware images CMake builds. Limited now to families that never spell set(FAMILY_MCUS ...) at all. * lib_examples read only an example's top-level CMakeLists.txt/Makefile; host/msc_file_explorer_freertos names lib/embedded-cli in src/CMakeLists.txt and survived by luck. The whole example tree is scanned. (SEGGER_RTT and rt-thread still resolve to nothing: all three references sit inside a LOGGER=rtt guard no CI build sets - the documented ruling, not a miss.) * get_family_boards applied ci_skip_boards/ci_preferred_boards only under GITHUB_ACTIONS/CIRCLECI, so the selector answered differently on a laptop than on a runner; _prune_buildable forces CI semantics. Its one-board pick also abandoned the whole preferred list when entry one could not build the -e set, and asked skip_example without the build's -D tokens. * _config_enables and lib_examples still read with the locale encoding - under LC_ALL=C the selector tracebacked on three tracked tusb_config.h files. The whole selector and its suite run clean there now. Workflows: * the Membrowse Upload step omitted $EX_ARGS, but --one-first now picks the board from the -e set, so it configured a different, empty build dir and uploaded --identical for a board never compiled. It takes $EX_ARGS for the BOARD; the target stays the aggregate, which has no DEPENDS and still records every example. * blanking FAM_REGEX reset only build_filtered, leaving the build scoped while code-metrics took the UNSCOPED branch and diffed a 1-family run against the full averaged baseline. All three drop together now, as CircleCI's fall-open does. * CircleCI's EX_ARGS had no character screen and is used unquoted, and its code-metrics job still exit 1'd on an empty metrics set - which a scoped build makes a legitimate outcome. * a `ci-full` PR label now turns the scoping off for one PR. A selector bug under-selects silently, and without a label the only ways back to a full matrix are accidental. Performance, since the selector gates every other job: family.cmake texts are read once rather than per changed directory (a 6,000-file dep bump re-read 84 files 99,892 times) and _scrape_mcu is cached: 2.2s -> 0.29s there, 0.8s -> 0.33s on a class diff. Tests: a drift guard for hw/bsp families absent from ci_set_matrix.family_list (they select zero legs now, where they used to ride the full matrix); the rule-4 port test asserted a SUBSET, which set() satisfies, so it could not fail on the empty selection it exists to catch; the GITHUB_ENV guard test counted a SUM of two guards. Drops metrics.py's --only-examples, which nothing called, and applies the TOTAL scrub to the by-example branch that skipped it. 2026-08-21T04:09:03Z hathach thach@tinyusb.org 2026-08-21T04:09:03Z urn:sha1:f96ddbaa1e11a98f8076df48ba73026191c28399 The binding rule table (17 rows x 3 answer columns), the measured effect per PR shape, and the reasoning behind the parts that look surprising: why empty means empty, why hw/mcu and lib are rules rather than full-matrix paths, why get_deps.py is diffed as data, and which build system is the reference. The plan is the task-by-task record of how it was built, kept as the origin trail. 2026-08-21T04:07:27Z hathach thach@tinyusb.org 2026-08-21T04:07:27Z urn:sha1:04d0f71984117b8c72349f4584bd9e26a37b129c Every PR built all 74 legs (2494 example builds on GHA cmake alone) and flashed all 30 rig boards, whatever it touched. One classifier now walks the PR diff twice and answers three questions: which families to build, which examples per family, and which boards run which tests. Fail-open throughout - anything no rule classifies, any exception, any unusable output falls back to the full matrix, and a master push always builds everything. test/hil/helper/hil_select.py moves to tools/ci_select.py: it is no longer HIL-only, and tools/ is where the build side can import it. test_hil_select.py follows it as test_ci_select.py. Rules (docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md holds the full table): a port selects the families whose family.cmake references it, and its role - a dcd change skips host examples and vice versa; a class selects only the examples whose tusb_config.h enables its CFG_TU[DH]_ macro, following cross-class includes; an example selects itself; hw/bsp selects its family or board; hw/mcu and lib select whoever references them. CMake is the reference for all of it - make follows whatever cmake decides, family.mk is never scanned. Empty means empty (maintainer ruling): a rule that classifies a path to nothing selects nothing. Ports no family references, classes no config enables, libs no example builds and hw/mcu paths that resolve nowhere are all real - nothing compiles them, so nothing can validate them, and the master-push build is the net. Structural tests pin each such case with an explicit allowlist, so the day one stops being empty it fails pre-commit instead of silently narrowing CI. Per-example builds: build.py grows a repeatable -e, resolved against the targets CMake actually registered and batched into one `cmake --build --target a b c`. build_utils mirrors CMake's family_filter (the whole FAMILY_MCUS list, ${...} and string(TOUPPER ...) resolved) for the cmake side, while the make side keeps master's algorithm verbatim - the two build systems answer differently and a shared answer breaks lpc54's make link. hil-build gains this even on a full selection: 1702 example builds become 515. Transport: the selection travels as a file, never an argv or env var - a mass-sweep diff selects 261 KB against a 128 KiB exec limit, and E2BIG would fail the step before its own fallback could run. CircleCI carries the example map inside the generated config (pipeline parameters cap at 512 chars), swapped into the parameter defaults by sentinel match, and drops the scoping wholesale if that rewrite fails. Every PR-derived value written to $GITHUB_ENV/$GITHUB_OUTPUT is character-screened. Code metrics follow the scoping: metrics.py emits per-example totals, and metrics_pair_compare compares the (board, example) pairs present on both sides instead of a scoped run against a full-matrix average. The selector's own suite gates it in both providers: a selector that exits 0 with valid-but-wrong JSON is the one failure fail-open cannot catch, so a red suite means the full matrix. 2026-08-21T04:06:55Z hathach thach@tinyusb.org 2026-08-21T04:06:55Z urn:sha1:696c7807f543a6c55656d81a8f6d8969584e9614 get_deps matches a family token against a requested family name verbatim (`f in deps_optional[d][2].split()`), so a token naming no hw/bsp directory makes its entry unreachable: hw/mcu/allwinner said 'fc100s'; the family is hw/bsp/f1c100s, and f1c100s/family.cmake sets SDK_DIR to ${TOP}/hw/mcu/allwinner/f1c100s hw/mcu/sony/cxd56/spresense-exported-sdk said 'spresense' (the SDK's name); the family is hw/bsp/cxd56, whose family.cmake points SDK_DIR at it `python3 tools/get_deps.py f1c100s` and `... cxd56` now fetch the SDK each of those families builds against; before, both printed "no additional dependencies found". docs/reference/dependencies.rst is generated from deps_all by tools/gen_doc.py, so it is updated to match - column widths are unchanged (the widest cell is lib/CMSIS_5's, untouched) and every row was cross-checked against deps_all. 2026-08-20T17:47:27Z hathach thach@tinyusb.org 2026-08-20T07:30:59Z urn:sha1:872b4fbdc3c607b522211839a6b3b82e18310a1b Document the ci and hfp HIL rigs in enough detail to reproduce one: bill of materials with photos, BIOS/IOMMU and vfio-pci passthrough on the Proxmox host, the Renesas uPD720201 firmware install, the guest software and permissions, the one-hub-per-root-port USB topology rule and the per-box split of probe and DUT hubs, how CI drives the rigs, and the operational gotchas. The attached-board table is generated from test/hil/tinyusb.json and test/hil/hfp.json by tools/gen_doc.py into docs/reference/hil_boards.md, which the page includes. Sphinx excludes that partial so it is not also built as an orphan document. Also exclude docs/superpowers/ from the Sphinx build: it holds internal plans, specs and handoffs rather than published documentation, and since nothing references them from a toctree each emitted "document isn't included in any toctree" -- 26 warnings in total, so build_doc.py -W could never pass. It now does. 2026-08-20T10:50:46Z hathach thach@tinyusb.org 2026-08-20T10:50:46Z urn:sha1:208ce50e0d5c08eead9c854d73fd397748e1413e The doc carried its own rename instruction for when the branch gained a number; the branch is PR #3836. 2026-08-20T09:47:23Z hathach thach@tinyusb.org 2026-08-20T09:47:23Z urn:sha1:198f5ce623646574129256218e60c1fc68238fc7 Four writers produce hil_report.md and three of them write no JSON - the no-boards exit, the pool-guard fallback and _abandon_exit's text prepend. Those are exactly the runs that failed, so hil_summary.py, which builds an agent's per-board verdicts from the sidecar, sees nothing while a human reading the markdown sees the real story. The scope note is markdown-only too, so a three-board PR run and a full run that lost 24 boards are indistinguishable in JSON. Five tasks: put scope in the sidecar, render the markdown from the document, give the two early-exit paths a document, make _abandon_exit set a caveat field instead of prepending to a file it did not write, then pin the invariant that re-rendering the JSON reproduces the markdown byte for byte. Split out because it is a hil_test.py reporting refactor, and the abandon path runs while the interpreter is being torn down - it deserves its own review. 2026-08-20T09:47:23Z hathach thach@tinyusb.org 2026-08-20T09:47:23Z urn:sha1:c514a6ee91a53927f55ba35e0a2933eddb5cefeb Nothing checked the agents, workflows and skills against the code they describe, and the surface had drifted into stating incompatible rules. This records the protocol that found the defects and what it cost. Method: parallel subagents extract every falsifiable claim into JSONL ledgers; a validator re-reads each cited line and rejects any ledger whose quoted text is not there, so an extractor that paraphrases or hallucinates fails a script rather than reaching the verification queue - 1,387 claims, zero such failures. Verification runs only in the main session, and the same gate pointed at `citation` then checks the verifier's own work. Hard-earned evidence is source of truth: code is authoritative about code, experience about hardware, so claims get a fourth verdict, EARNED, and "no backing found" is never grounds for deletion. All 1,387 claims carry a verdict; the behavior sweep deliberately never emits CONFIRMED from a token match, because finding a claim's vocabulary proves presence, not truth. Every real defect came from cross-document comparison - none from any mechanical pass. A path-existence lint was built, measured (11 flags on the audited tree, all false positives, and the target defect is lexically identical to correct text elsewhere), and rejected; recorded so nobody rebuilds it.