diff options
| author | hathach <[email protected]> | 2026-08-20 16:47:23 +0700 |
|---|---|---|
| committer | hathach <[email protected]> | 2026-08-20 16:47:23 +0700 |
| commit | c514a6ee91a53927f55ba35e0a2933eddb5cefeb (patch) | |
| tree | 17e32eb998fe1e278d0226cb2d60c585136cd7a8 /docs/superpowers/specs | |
| parent | 126e29f777f07a35b06b327b50e62e0150521b48 (diff) | |
docs: spec, plan and outcome of the .claude instruction-surface audit
Nothing checked the agents, workflows and skills against the code they describe, and
the surface had drifted into stating incompatible rules. This records the protocol
that found the defects and what it cost.
Method: parallel subagents extract every falsifiable claim into JSONL ledgers; a
validator re-reads each cited line and rejects any ledger whose quoted text is not
there, so an extractor that paraphrases or hallucinates fails a script rather than
reaching the verification queue - 1,387 claims, zero such failures. Verification runs
only in the main session, and the same gate pointed at `citation` then checks the
verifier's own work. Hard-earned evidence is source of truth: code is authoritative
about code, experience about hardware, so claims get a fourth verdict, EARNED, and
"no backing found" is never grounds for deletion. All 1,387 claims carry a verdict;
the behavior sweep deliberately never emits CONFIRMED from a token match, because
finding a claim's vocabulary proves presence, not truth.
Every real defect came from cross-document comparison - none from any mechanical
pass. A path-existence lint was built, measured (11 flags on the audited tree, all
false positives, and the target defect is lexically identical to correct text
elsewhere), and rejected; recorded so nobody rebuilds it.
Diffstat (limited to 'docs/superpowers/specs')
| -rw-r--r-- | docs/superpowers/specs/2026-08-18-claude-doc-audit-design.md | 135 |
1 files changed, 135 insertions, 0 deletions
diff --git a/docs/superpowers/specs/2026-08-18-claude-doc-audit-design.md b/docs/superpowers/specs/2026-08-18-claude-doc-audit-design.md new file mode 100644 index 000000000..5b150dc7b --- /dev/null +++ b/docs/superpowers/specs/2026-08-18-claude-doc-audit-design.md @@ -0,0 +1,135 @@ +# Audit of the `.claude/` instruction surface — design + +**Date:** 2026-08-18 +**Branch:** `claude/hil-doc-audit` + +## Why + +`hil-operator.md` told an operator two incompatible things at once: one rule forbade +pre-holding a board lock because `hil_test.py` self-locks, while a rule added in the same +revision made the lock the thing that keeps concurrent operators off each other's hardware — +so an operator following the second would take a hold that made its own run fail fast against +it. Both statements were fixed before this branch was folded, so neither survives in history; +what survives is the lesson that nothing checks these files against the code they describe. + +That is not an isolated slip. A scan of the 36 repo paths cited across `.claude/` flags 8 +that do not resolve. Five are legitimate — placeholders (`docs/changelog/X.Y.Z.md`, +`src/portable/x/dcd_x.c`, a `test_*.py` glob), a generated file +(`examples/cmake-build-pvs/compile_commands.json`), and a per-host gitignored config +(`test/hil/local.json`, whose absence the skill already handles). Three are drift: +`usbtest/SKILL.md:24,50` cites `src/usb_descriptors.h` and `src/tusb_config.h`, which are +example-relative but read as repo paths, and `:101` cites `tools/usb/testusb.c`, a Linux +kernel path presented like a repo file. + +Cross-references are in better shape: every `agentType` in a workflow resolves to an agent +in `.claude/agents/`, every `.claude/skills/<name>` referenced by an agent or workflow +exists, and the workflow scripts call only harness functions that exist. The drift is in +**prose claims about behavior** — the class that made `hil-validate` parallelize at the +wrong layer, on top of a `hil_test.py` that already schedules boards across host +controllers under per-controller permits (`hil_lock.py:7,133-134`; `hil_test.py:2249,2419`). + +## Scope + +**In:** `.claude/agents/*.md` (7), `.claude/workflows/*.js` + `check.sh` (7), +`.claude/skills/*/SKILL.md` (16) and their 8 helper scripts, and the repo `CLAUDE.md`. +~4,700 lines (2,874 of prose, the rest helper scripts and `etm-trace/boards.md`). + +**Out:** `docs/superpowers/**` (historical records — correcting them rewrites history +rather than fixing what a future session executes), `.claude/settings*.json` and hooks, the +memory index, and any behavior change to the scripts themselves. + +## Claim taxonomy + +Only falsifiable classes get a verdict. Guidance ("bias toward caution") is checked solely +for contradiction with the classes below. + +| Class | Settled by | Example | +|---|---|---| +| Path | `ls`/`find`, with the base dir made explicit | `src/tusb_config.h` — example-relative, reads as repo-relative | +| Interface | argparse/grep in the named source | `-b` is `action='append'` (`hil_test.py:2249`) | +| Behavior | reading the implementing code, cited `file:line` | "permits are in-process semaphores" (`hil_lock.py:7`) | +| Number | the constant's definition | `FLASH_PARALLEL=4` (`hil_lock.py:133`) | +| Rig state | read-only `ssh ci.lan` probe | bus map, probe uids, sudoers entries, installed tools | +| Cross-doc | diffing the same rule's two statements | `hil-operator.md:18` vs `:37` | + +### Verdicts + +- **CONFIRMED** — current source says so. Cite `file:line`. Leave alone. +- **REFUTED** — current source says otherwise. Cite, correct the doc. +- **EARNED** — no source in scope settles it, and it is hard-earned rig knowledge. Stays in + the docs untouched; see the rule below. +- **UNVERIFIABLE** — no source in scope settles it and it is not earned knowledge either + (a placeholder, a generated file, a claim about something outside the repo). + +### Hard-earned evidence is source of truth + +A claim with no code backing is **not** a cut candidate when it is earned rig knowledge: +an observed hardware quirk, a failure mode paid for in rig downtime, a workaround whose +rationale lives only in the incident that produced it. Code is authoritative about code; +experience is authoritative about hardware, and the hardware does not document itself. + +Consequences: + +- Only a claim the **current source actively refutes** gets corrected. "I could not find + backing" is never grounds for deletion. +- Rig-state claims that have gone stale (a bus map, a probe uid) are **re-derived and + updated**, or converted into a derivation recipe ("buses renumber every boot — re-derive + with X"), never dropped. +- Where earned knowledge and current code disagree, that is a **finding to report**, not an + edit to make: one of them is a bug, and deciding which is out of this audit's scope. + +## Passes + +1. **Extraction (fan-out, 9 agents, no verdicts).** One agent per cluster, each writing a + ledger to the scratchpad and returning only a count and the ledger path. Per claim: + `file:line`, verbatim claim, class, what source would settle it, and a flag for + suspected hard-earned evidence. Agents return no judgments, so nothing arrives as a + verdict that would have to be unwound. +2. **Verification (mine).** Every claim checked against source myself: scripted checks for + paths/interfaces/numbers, code reading for behavior, read-only `ssh ci.lan` for rig + state (`ls`, `--help`, `which`, `lspci`, `lsusb`, `hil_lock.py status`, `sudo -l`, + `uname -r` — no locks, no flashing, no `uhubctl`, no recovery). Nothing acted on is + taken on an extractor's word. +3. **Cross-doc consistency (mine).** Build a rule inventory — board locks, timeouts, + output contracts, retry policy, config selection, forcing — and diff every place each + rule is stated. No per-file agent can do this pass; it is where the `hil-operator` + failure lived. +4. **Edits.** Delete only what is refuted by source, restates the command it precedes, or + duplicates a rule that has a canonical home elsewhere (keep one, reference it). Keep + every claim source confirms that changes behavior, every hard-earned observation, and + the "why" behind non-obvious rules. Structure stays as is. +5. **Gate.** Re-run the path and interface scans; `check.sh` on every workflow; `bash -n` + and `py_compile` on all 8 helper scripts; the four `test/hil` suites; + `pre-commit run --all-files`. + +## Extraction clusters + +| # | Cluster | Lines | +|---|---|---| +| 1 | `.claude/agents/*.md` (7 files) | 313 | +| 2 | `.claude/workflows/*.js` + `check.sh` | 659 | +| 3 | `hil`, `hil-pool-check` | 223 | +| 4 | `usb-kernel-recover`, `usb-kernel-debug` + 2 scripts | 253 + scripts | +| 5 | `target-debug`, `esp-target-debug` | 496 | +| 6 | `usbtest`, `usbmon`, `usb-sniffer` + `usbcap.sh` | 382 + script | +| 7 | `etm-trace` + `boards.md` + 2 scripts | 203 + files | +| 8 | `build-doc`, `code-size`, `pvs`, `make-release`, `read-doc`, `pre-pr` + 2 scripts | 345 + scripts | +| 9 | `CLAUDE.md` | 139 | + +## Deliverables + +Commits split by surface (agents / workflows / skills / CLAUDE.md) so review stays +tractable, on `claude/claude-doc-audit`. A findings report covering every REFUTED claim +with its citation, and every earned-knowledge-vs-code disagreement found in pass 2. + +A refuted claim whose *code* is the wrong half does not get a silent code edit: it becomes +a handoff doc under `docs/superpowers/followup/`, per the repo's deferred-work rule. + +## Success criteria + +- Every falsifiable claim in scope carries a verdict with a citation. +- No claim that current source refutes survives in the tree. +- No hard-earned observation is deleted; stale rig state is re-derived or turned into a + derivation recipe. +- No rule is stated in two places with two different meanings. +- The gate in pass 5 passes. |
