summaryrefslogtreecommitdiff
path: root/docs/superpowers/specs
diff options
context:
space:
mode:
Diffstat (limited to 'docs/superpowers/specs')
-rw-r--r--docs/superpowers/specs/2026-08-18-claude-doc-audit-design.md135
1 files changed, 135 insertions, 0 deletions
diff --git a/docs/superpowers/specs/2026-08-18-claude-doc-audit-design.md b/docs/superpowers/specs/2026-08-18-claude-doc-audit-design.md
new file mode 100644
index 000000000..5b150dc7b
--- /dev/null
+++ b/docs/superpowers/specs/2026-08-18-claude-doc-audit-design.md
@@ -0,0 +1,135 @@
+# Audit of the `.claude/` instruction surface — design
+
+**Date:** 2026-08-18
+**Branch:** `claude/hil-doc-audit`
+
+## Why
+
+`hil-operator.md` told an operator two incompatible things at once: one rule forbade
+pre-holding a board lock because `hil_test.py` self-locks, while a rule added in the same
+revision made the lock the thing that keeps concurrent operators off each other's hardware —
+so an operator following the second would take a hold that made its own run fail fast against
+it. Both statements were fixed before this branch was folded, so neither survives in history;
+what survives is the lesson that nothing checks these files against the code they describe.
+
+That is not an isolated slip. A scan of the 36 repo paths cited across `.claude/` flags 8
+that do not resolve. Five are legitimate — placeholders (`docs/changelog/X.Y.Z.md`,
+`src/portable/x/dcd_x.c`, a `test_*.py` glob), a generated file
+(`examples/cmake-build-pvs/compile_commands.json`), and a per-host gitignored config
+(`test/hil/local.json`, whose absence the skill already handles). Three are drift:
+`usbtest/SKILL.md:24,50` cites `src/usb_descriptors.h` and `src/tusb_config.h`, which are
+example-relative but read as repo paths, and `:101` cites `tools/usb/testusb.c`, a Linux
+kernel path presented like a repo file.
+
+Cross-references are in better shape: every `agentType` in a workflow resolves to an agent
+in `.claude/agents/`, every `.claude/skills/<name>` referenced by an agent or workflow
+exists, and the workflow scripts call only harness functions that exist. The drift is in
+**prose claims about behavior** — the class that made `hil-validate` parallelize at the
+wrong layer, on top of a `hil_test.py` that already schedules boards across host
+controllers under per-controller permits (`hil_lock.py:7,133-134`; `hil_test.py:2249,2419`).
+
+## Scope
+
+**In:** `.claude/agents/*.md` (7), `.claude/workflows/*.js` + `check.sh` (7),
+`.claude/skills/*/SKILL.md` (16) and their 8 helper scripts, and the repo `CLAUDE.md`.
+~4,700 lines (2,874 of prose, the rest helper scripts and `etm-trace/boards.md`).
+
+**Out:** `docs/superpowers/**` (historical records — correcting them rewrites history
+rather than fixing what a future session executes), `.claude/settings*.json` and hooks, the
+memory index, and any behavior change to the scripts themselves.
+
+## Claim taxonomy
+
+Only falsifiable classes get a verdict. Guidance ("bias toward caution") is checked solely
+for contradiction with the classes below.
+
+| Class | Settled by | Example |
+|---|---|---|
+| Path | `ls`/`find`, with the base dir made explicit | `src/tusb_config.h` — example-relative, reads as repo-relative |
+| Interface | argparse/grep in the named source | `-b` is `action='append'` (`hil_test.py:2249`) |
+| Behavior | reading the implementing code, cited `file:line` | "permits are in-process semaphores" (`hil_lock.py:7`) |
+| Number | the constant's definition | `FLASH_PARALLEL=4` (`hil_lock.py:133`) |
+| Rig state | read-only `ssh ci.lan` probe | bus map, probe uids, sudoers entries, installed tools |
+| Cross-doc | diffing the same rule's two statements | `hil-operator.md:18` vs `:37` |
+
+### Verdicts
+
+- **CONFIRMED** — current source says so. Cite `file:line`. Leave alone.
+- **REFUTED** — current source says otherwise. Cite, correct the doc.
+- **EARNED** — no source in scope settles it, and it is hard-earned rig knowledge. Stays in
+ the docs untouched; see the rule below.
+- **UNVERIFIABLE** — no source in scope settles it and it is not earned knowledge either
+ (a placeholder, a generated file, a claim about something outside the repo).
+
+### Hard-earned evidence is source of truth
+
+A claim with no code backing is **not** a cut candidate when it is earned rig knowledge:
+an observed hardware quirk, a failure mode paid for in rig downtime, a workaround whose
+rationale lives only in the incident that produced it. Code is authoritative about code;
+experience is authoritative about hardware, and the hardware does not document itself.
+
+Consequences:
+
+- Only a claim the **current source actively refutes** gets corrected. "I could not find
+ backing" is never grounds for deletion.
+- Rig-state claims that have gone stale (a bus map, a probe uid) are **re-derived and
+ updated**, or converted into a derivation recipe ("buses renumber every boot — re-derive
+ with X"), never dropped.
+- Where earned knowledge and current code disagree, that is a **finding to report**, not an
+ edit to make: one of them is a bug, and deciding which is out of this audit's scope.
+
+## Passes
+
+1. **Extraction (fan-out, 9 agents, no verdicts).** One agent per cluster, each writing a
+ ledger to the scratchpad and returning only a count and the ledger path. Per claim:
+ `file:line`, verbatim claim, class, what source would settle it, and a flag for
+ suspected hard-earned evidence. Agents return no judgments, so nothing arrives as a
+ verdict that would have to be unwound.
+2. **Verification (mine).** Every claim checked against source myself: scripted checks for
+ paths/interfaces/numbers, code reading for behavior, read-only `ssh ci.lan` for rig
+ state (`ls`, `--help`, `which`, `lspci`, `lsusb`, `hil_lock.py status`, `sudo -l`,
+ `uname -r` — no locks, no flashing, no `uhubctl`, no recovery). Nothing acted on is
+ taken on an extractor's word.
+3. **Cross-doc consistency (mine).** Build a rule inventory — board locks, timeouts,
+ output contracts, retry policy, config selection, forcing — and diff every place each
+ rule is stated. No per-file agent can do this pass; it is where the `hil-operator`
+ failure lived.
+4. **Edits.** Delete only what is refuted by source, restates the command it precedes, or
+ duplicates a rule that has a canonical home elsewhere (keep one, reference it). Keep
+ every claim source confirms that changes behavior, every hard-earned observation, and
+ the "why" behind non-obvious rules. Structure stays as is.
+5. **Gate.** Re-run the path and interface scans; `check.sh` on every workflow; `bash -n`
+ and `py_compile` on all 8 helper scripts; the four `test/hil` suites;
+ `pre-commit run --all-files`.
+
+## Extraction clusters
+
+| # | Cluster | Lines |
+|---|---|---|
+| 1 | `.claude/agents/*.md` (7 files) | 313 |
+| 2 | `.claude/workflows/*.js` + `check.sh` | 659 |
+| 3 | `hil`, `hil-pool-check` | 223 |
+| 4 | `usb-kernel-recover`, `usb-kernel-debug` + 2 scripts | 253 + scripts |
+| 5 | `target-debug`, `esp-target-debug` | 496 |
+| 6 | `usbtest`, `usbmon`, `usb-sniffer` + `usbcap.sh` | 382 + script |
+| 7 | `etm-trace` + `boards.md` + 2 scripts | 203 + files |
+| 8 | `build-doc`, `code-size`, `pvs`, `make-release`, `read-doc`, `pre-pr` + 2 scripts | 345 + scripts |
+| 9 | `CLAUDE.md` | 139 |
+
+## Deliverables
+
+Commits split by surface (agents / workflows / skills / CLAUDE.md) so review stays
+tractable, on `claude/claude-doc-audit`. A findings report covering every REFUTED claim
+with its citation, and every earned-knowledge-vs-code disagreement found in pass 2.
+
+A refuted claim whose *code* is the wrong half does not get a silent code edit: it becomes
+a handoff doc under `docs/superpowers/followup/`, per the repo's deferred-work rule.
+
+## Success criteria
+
+- Every falsifiable claim in scope carries a verdict with a citation.
+- No claim that current source refutes survives in the tree.
+- No hard-earned observation is deleted; stale rig state is re-derived or turned into a
+ derivation recipe.
+- No rule is stated in two places with two different meanings.
+- The gate in pass 5 passes.