summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorhathach <[email protected]>2026-08-18 15:28:58 +0700
committerhathach <[email protected]>2026-08-18 16:58:58 +0700
commitf59c8948729debc6d57c4dfade5486176468edbf (patch)
tree4090122073c645ab5c73cf58cb01075950a2f88f
parent11bdbc3eac085e4a946e25a8a614bcbf62e41fa2 (diff)
skill(read-doc): search the Calibre database instead of the filesystem
Finding documents by walking the library tree misses anything the filename does not carry - Calibre stores only a truncated title and the author there, so the tags, series, publisher and description that hold most part numbers and errata IDs are invisible to it. A zero-result tree search then reads as "the document does not exist" rather than as a bad search; that happened here, and led to a confident claim that a fully populated 14,000-file library was empty. search.py queries metadata.db, ANDs its keywords across every metadata field (including the stored filename), and prints the best matches first with the exact path to read. Matching is NFKC + casefold, so a typed ASCII apostrophe or mu reaches the titles that store the typographic ones. Every printed path is checked on disk. Calibre renames <author>/<title> (<id>) when metadata is edited and leaves the old directory behind, so a miss retries by the stable book id before reporting MISSING - which distinguishes "the file is not here right now" from "no such document". The gate tests for metadata.db rather than the directory, since an unmounted or half-synced mountpoint is still a directory. Consumers that prescribed their own tree search - driver-reviewer, port-dev, the driver-review workflow, and the calibre-library references in CLAUDE.md, usbtest, etm-trace and target-debug - now point at the skill, which owns the library's location.
-rw-r--r--.claude/agents/driver-reviewer.md4
-rw-r--r--.claude/agents/port-dev.md2
-rw-r--r--.claude/skills/etm-trace/SKILL.md2
-rw-r--r--.claude/skills/etm-trace/boards.md2
-rw-r--r--.claude/skills/read-doc/SKILL.md60
-rwxr-xr-x.claude/skills/read-doc/search.py112
-rw-r--r--.claude/skills/target-debug/SKILL.md2
-rw-r--r--.claude/skills/usbtest/SKILL.md2
-rw-r--r--.claude/workflows/driver-review.js2
-rw-r--r--CLAUDE.md2
10 files changed, 165 insertions, 25 deletions
diff --git a/.claude/agents/driver-reviewer.md b/.claude/agents/driver-reviewer.md
index f45eca03e..9ce8b621f 100644
--- a/.claude/agents/driver-reviewer.md
+++ b/.claude/agents/driver-reviewer.md
@@ -1,7 +1,7 @@
---
name: driver-reviewer
description: Review one TinyUSB driver directory or one diff against one review dimension (correctness, ISR safety, datasheet/errata conformance, style) with coverage-first structured findings; or adversarially verify a single finding / fix. Read-only.
-tools: Bash, Read, Grep, Glob
+tools: Bash, Read, Grep, Glob, Skill
model: opus
---
@@ -9,7 +9,7 @@ You review exactly the scope given in your prompt (one driver directory, or one
## Datasheets & errata
-For register-use review, find the MCU/USB-IP reference manual in `$HOME/Documents/calibre-library` — and ALSO search the library for the part's errata / silicon-bug sheets (search terms: "errata" plus the MCU or USB-IP name). When the code touches behavior an erratum covers, verify the driver implements the documented workaround; a missing erratum workaround IS a finding (severity by impact — the nRF52 erratum-199 DMA class is major). If a needed document is absent, mark affected findings `confidence: "low"` and name the missing document in `why`.
+For register-use review, find the MCU/USB-IP reference manual with the `read-doc` skill — `python3 .claude/skills/read-doc/search.py <keywords>`, never `find`/`grep` over the library tree — and ALSO search for the part's errata / silicon-bug sheets (search terms: "errata" plus the MCU or USB-IP name). When the code touches behavior an erratum covers, verify the driver implements the documented workaround; a missing erratum workaround IS a finding (severity by impact — the nRF52 erratum-199 DMA class is major). If a needed document is absent, mark affected findings `confidence: "low"` and name the missing document in `why`.
## Reporting discipline
diff --git a/.claude/agents/port-dev.md b/.claude/agents/port-dev.md
index 76bafb39b..77a28bafa 100644
--- a/.claude/agents/port-dev.md
+++ b/.claude/agents/port-dev.md
@@ -16,7 +16,7 @@ You implement exactly one specified change in one assigned scope (a directory un
## Datasheets
-When changing dcd/hcd register logic, cross-check the MCU reference manual / datasheet / programming guide in `$HOME/Documents/calibre-library` (search by MCU or USB-IP name). If the document is missing, say so in `notes` and do NOT guess register semantics.
+When changing dcd/hcd register logic, cross-check the MCU reference manual / datasheet / programming guide with the `read-doc` skill — `python3 .claude/skills/read-doc/search.py <MCU or USB-IP name>`, never `find`/`grep` over the library tree. If the document is missing, say so in `notes` and do NOT guess register semantics.
## Finish checklist (in order)
diff --git a/.claude/skills/etm-trace/SKILL.md b/.claude/skills/etm-trace/SKILL.md
index 99e89729c..43cf6a2a5 100644
--- a/.claude/skills/etm-trace/SKILL.md
+++ b/.claude/skills/etm-trace/SKILL.md
@@ -147,7 +147,7 @@ request `itrace.csv`, `profile_lines.csv`, `profile_insts.csv`, `samples.csv`,
Bring-up ladder — each step gates the next:
-1. **Docs before hardware** (calibre library first, then vendor site): board
+1. **Docs before hardware** (`read-doc` skill first, then vendor site): board
manual, schematics, MCU reference manual. Establish the trace clock
source and max — chip side and probe side (J-Trace PRO Cortex-M tops out
at a 150 MHz trace clock) — the pins carrying TRACE_CLK/D0-D3 (read the board's
diff --git a/.claude/skills/etm-trace/boards.md b/.claude/skills/etm-trace/boards.md
index 4a5f297ae..044d4e0ee 100644
--- a/.claude/skills/etm-trace/boards.md
+++ b/.claude/skills/etm-trace/boards.md
@@ -52,7 +52,7 @@ Board caveats (beyond the table):
`AfterTargetConnect` hook; un-attachable after a killed session →
power-cycle.
- **metro_m7_1011** (RT1011): a custom Adafruit rev with a hand-added 2x10
- ETM header (KiCad schematic in the calibre library). No SEGGER RT1011
+ ETM header (KiCad schematic via the `read-doc` skill). No SEGGER RT1011
example exists — the committed .jdebug (tuned +50 ps) is the known-good
reference. BOARD_BootClockRUN sets the 132 MHz trace root but leaves it
gated; `trace_etm_init` ungates it. The first Ozone run after a fresh
diff --git a/.claude/skills/read-doc/SKILL.md b/.claude/skills/read-doc/SKILL.md
index df845e7b5..feaa914af 100644
--- a/.claude/skills/read-doc/SKILL.md
+++ b/.claude/skills/read-doc/SKILL.md
@@ -8,17 +8,22 @@ description: Use when you need authoritative hardware/protocol facts from a prim
## Overview
Some maintainers keep datasheets, manuals, and books in a Calibre library at
-`$HOME/Documents/calibre-library/`, laid out as
-`AUTHOR/TITLE (id)/TITLE - AUTHOR.pdf|.epub`. For hardware/protocol facts —
-registers, bitfields, memory maps, pinouts, electrical/timing specs, errata, USB
-spec — read the doc instead of answering from training knowledge or the web.
+`$HOME/Documents/calibre-library/`. For hardware/protocol facts — registers,
+bitfields, memory maps, pinouts, electrical/timing specs, errata, USB spec —
+read the doc instead of answering from training knowledge or the web.
+
+Search the library's `metadata.db`, never the filesystem. The database indexes
+title, authors, tags, series, publisher, description and the stored filename;
+most part numbers live in the tags, which the filesystem does not carry.
## Gate first
-The library is per-user. Check it exists before anything else:
+The library is per-user and usually on a network mount, so test the database
+file, not the directory — an unmounted or half-synced mountpoint is still a
+directory:
```bash
-[ -d "$HOME/Documents/calibre-library" ] && echo present || echo absent
+[ -f "${CALIBRE_LIBRARY:-$HOME/Documents/calibre-library}/metadata.db" ] && echo present || echo absent
```
Absent → the skill does not apply; fall back to normal sources silently (don't
@@ -35,27 +40,50 @@ Not for general concepts, repo/code questions, or when no such doc is likely.
## Find
-Keywords from `/read-doc <keywords>`, else derived from the question (part number,
-peripheral, spec name). AND them with chained case-insensitive grep:
+Keywords from `/read-doc <keywords>`, else derived from the question (part
+number, peripheral, spec name). `search.py` ANDs them across every metadata
+field and prints the best matches first — at most 40, and the header says when
+more matched:
```bash
-find "$HOME/Documents/calibre-library/" -maxdepth 3 \( -iname '*.pdf' -o -iname '*.epub' \) | grep -i "kw1" | grep -i "kw2"
+python3 .claude/skills/read-doc/search.py errata RT1064 # AND (default)
+python3 .claude/skills/read-doc/search.py RT1060 RT1064 --any
```
-One match → read it. Several → list and ask via AskUserQuestion. None → drop the
-weakest keyword and broaden (filenames hold title+author, not tags); still none →
-list the closest author/title matches.
+Exit 0 matched, 1 nothing matched, 2 bad usage or no library — 2 means the
+search never ran, so fix the invocation instead of broadening.
+
+One match → read it. Several → list and ask via AskUserQuestion. Nothing
+(exit 1) → retry with fewer keywords; the part number alone often works where
+`<part> datasheet` does not, because words like "datasheet" and "manual" are
+rarely in the metadata. `--any` only changes anything with two or more
+keywords. Still nothing → say the document is missing rather than answering
+from memory.
+
+Set `CALIBRE_LIBRARY` to search a library elsewhere.
## Read
-- **PDF:** Read with `pages`; for >10 pages start `pages: "1-20"` (TOC/overview),
+`search.py` prints one `FORMAT path` line per stored file:
+
+- **PDF** — Read with `pages`; for >10 pages start `pages: "1-20"` (TOC/overview),
report the page count, then read sections on demand.
-- **EPUB:** Read the path directly.
-- Summarize in one line (title, pages, coverage) and keep as reference context.
+- **Any other format** (EPUB, MOBI, CHM, ZIP…) — Read has no decoder for these
+ and returns mojibake rather than an error. Say the document is not in a
+ readable format; do not paste what Read returned.
+- **`MISSING`** — the metadata is real but the file is not on disk (library
+ mid-sync, or the file was deleted). Report the file as unavailable, not the
+ document as nonexistent.
+
+Summarize in one line (title, pages, coverage) and keep as reference context.
## Common mistakes
+- Searching with `find`/`grep` over the library tree. It sees only truncated
+ filenames, missing the tags, series and descriptions where part numbers and
+ errata IDs actually live. Query the database.
- Skipping the gate on a machine with no library.
- Answering a register/spec question from memory when the datasheet is on disk.
- Loading a 1000-page PDF up front instead of TOC-first.
-- Requiring all keywords to match — broaden on zero hits.
+- Requiring all keywords to match — broaden, or use `--any`, on zero hits.
+- Treating a `MISSING` file, or an exit 2, as proof the document is absent.
diff --git a/.claude/skills/read-doc/search.py b/.claude/skills/read-doc/search.py
new file mode 100755
index 000000000..c70d36805
--- /dev/null
+++ b/.claude/skills/read-doc/search.py
@@ -0,0 +1,112 @@
+#!/usr/bin/env python3
+"""Search the Calibre library by metadata and print matching document paths.
+
+Usage: search.py KEYWORD [KEYWORD...] all keywords must match (AND)
+ search.py --any KEYWORD [KEYWORD...] any keyword matches (OR)
+
+Matches title, authors, tags, series, publisher, description and stored
+filename, and prints the exact path to read, best match first.
+
+Exit 0 matched, 1 nothing matched, 2 bad usage or no library.
+"""
+import glob
+import os
+import sqlite3
+import sys
+import unicodedata
+import urllib.parse
+
+LIB = os.path.realpath(os.path.expanduser(os.environ.get("CALIBRE_LIBRARY") or "~/Documents/calibre-library"))
+DB = os.path.join(LIB, "metadata.db")
+LIMIT = 40
+
+QUERY = """
+SELECT b.id, b.title, b.path,
+ (SELECT group_concat(a.name, ', ') FROM authors a
+ JOIN books_authors_link l ON l.author = a.id WHERE l.book = b.id),
+ (SELECT group_concat(t.name, ', ') FROM tags t
+ JOIN books_tags_link l ON l.tag = t.id WHERE l.book = b.id),
+ (SELECT group_concat(s.name, ', ') FROM series s
+ JOIN books_series_link l ON l.series = s.id WHERE l.book = b.id),
+ (SELECT group_concat(p.name, ', ') FROM publishers p
+ JOIN books_publishers_link l ON l.publisher = p.id WHERE l.book = b.id),
+ (SELECT c.text FROM comments c WHERE c.book = b.id),
+ (SELECT group_concat(d.format || '/' || d.name, char(10)) FROM data d WHERE d.book = b.id)
+FROM books b
+"""
+
+_authors = None
+
+
+def norm(s):
+ # NFKC + casefold so MICRO SIGN/GREEK MU, curly quotes and dashes compare equal.
+ return unicodedata.normalize("NFKC", s).casefold()
+
+
+def resolve(bid, path, fmt, name):
+ """Absolute path of one format row, or None if the file is not on disk.
+
+ Calibre renames `<author>/<title> (<id>)` when metadata is edited and leaves
+ the old directory behind, so on a miss retry by the stable book id.
+ """
+ ext = "." + fmt.lower()
+ exact = os.path.join(LIB, path, name + ext)
+ if os.path.exists(exact):
+ return exact
+ global _authors
+ if _authors is None:
+ _authors = {}
+ for d in os.listdir(LIB): # case-only duplicates exist on a case-sensitive mount
+ _authors.setdefault(d.lower(), []).append(d)
+ for author in _authors.get(path.split("/")[0].lower(), ()):
+ for d in glob.glob(os.path.join(glob.escape(os.path.join(LIB, author)), "* (%d)" % bid)):
+ for f in sorted(glob.glob(os.path.join(glob.escape(d), "*" + ext))):
+ return f
+ return None
+
+
+def main(argv):
+ match_any = "--any" in argv
+ keywords = [norm(k) for k in argv if k != "--any"]
+ if not keywords:
+ print(__doc__, file=sys.stderr)
+ return 2
+
+ if not os.path.exists(DB):
+ print(f"no Calibre database at {DB}", file=sys.stderr)
+ return 2
+
+ db = sqlite3.connect("file:" + urllib.parse.quote(DB) + "?mode=ro", uri=True)
+ hits = []
+ for bid, title, path, authors, tags, series, publisher, comments, files in db.execute(QUERY):
+ entries = [e.split("/", 1) for e in (files or "").split("\n") if e]
+ hay = norm(" ".join(x for x in (title, authors, tags, series, publisher, comments) if x)
+ + " " + " ".join(n for _, n in entries))
+ found = sum(k in hay for k in keywords)
+ if not found or (not match_any and found < len(keywords)):
+ continue
+ in_title = sum(k in norm(title) for k in keywords)
+ hits.append((-found, -in_title, title, authors, tags, bid, path, entries))
+
+ if not hits:
+ print("no match")
+ return 1
+
+ hits.sort(key=lambda h: h[:3]) # authors/tags may be None and are not comparable
+ print(f"{len(hits)} book(s)" + (f", showing the {LIMIT} best" if len(hits) > LIMIT else ""))
+ for _, _, title, authors, tags, bid, path, entries in hits[:LIMIT]:
+ print(f"\n{title}" + (f" [{authors}]" if authors else "") + (f" tags: {tags}" if tags else ""))
+ if not entries:
+ print(" (no file in this library)")
+ for fmt, name in entries:
+ p = resolve(bid, path, fmt, name)
+ print(f" {fmt} {p}" if p else f" {fmt} MISSING (library mid-sync or file deleted)")
+ return 0
+
+
+if __name__ == "__main__":
+ try:
+ sys.exit(main(sys.argv[1:]))
+ except BrokenPipeError:
+ os.dup2(os.open(os.devnull, os.O_WRONLY), sys.stdout.fileno())
+ sys.exit(0)
diff --git a/.claude/skills/target-debug/SKILL.md b/.claude/skills/target-debug/SKILL.md
index 9a61a86c7..050a697b9 100644
--- a/.claude/skills/target-debug/SKILL.md
+++ b/.claude/skills/target-debug/SKILL.md
@@ -347,7 +347,7 @@ the wire itself: `usb-sniffer` skill (hardware tap, PID-level).
- J-Link (UM08001): <https://kb.segger.com/UM08001_J-Link_/_J-Trace_User_Guide> — flash breakpoints, RTT, SWO, monitor mode, Commander.
- OpenOCD: <https://openocd.org/doc/html/index.html> — `rtt`, `bp`/`wp`, `cortex_m vector_catch`/`maskisr`, `itm`/`tpiu`.
- "Debugging with GDB" (§5.1 = break/watch/dprintf): Tenth Edition (GDB 18)
- via calibre/`read-doc`, or
+ via the `read-doc` skill, or
`curl -sL -o /tmp/gdb.pdf https://sourceware.org/gdb/current/onlinedocs/gdb.pdf`
(the HTML mirror blocks fetchers). Installed `arm-none-eabi-gdb`
`help <cmd>` is authoritative here.
diff --git a/.claude/skills/usbtest/SKILL.md b/.claude/skills/usbtest/SKILL.md
index 6197ccd51..32c2be913 100644
--- a/.claude/skills/usbtest/SKILL.md
+++ b/.claude/skills/usbtest/SKILL.md
@@ -121,7 +121,7 @@ curl -sO "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/plain
3. **On-device gdb/openocd**: read the EP control registers and DCD structs at the hang.
4. Heisenbugs (vanish under logging): RAM ring-buffer trace dumped over openocd; for silent lockups
JLink PC-sampling (`halt`+`regs` repeatedly — a pinned PC names the spin).
-5. **Cross-check the reference manual** (calibre library) before changing any register-level code —
+5. **Cross-check the reference manual** (`read-doc` skill) before changing any register-level code —
per CLAUDE.md, and because comments/assumptions in DCDs have been wrong about hardware caps.
6. Check the vendor's **silicon errata** early for timing/DMA hangs (an unimplemented erratum
workaround caused a case-10 hang on one port).
diff --git a/.claude/workflows/driver-review.js b/.claude/workflows/driver-review.js
index 3638aa179..255b8ac74 100644
--- a/.claude/workflows/driver-review.js
+++ b/.claude/workflows/driver-review.js
@@ -16,7 +16,7 @@ if (!args || !Array.isArray(args.dirs) || args.dirs.length === 0) {
const DIMS = args.question ? [args.question] : (args.dimensions || [
'correctness: transfer state machines, endpoint bookkeeping, completion and error paths',
'ISR safety: work deferred to task context, shared-state races, register access ordering',
- 'register use vs datasheet and MCU errata: cross-check the reference manual AND errata sheets in $HOME/Documents/calibre-library; a missing erratum workaround is a finding',
+ 'register use vs datasheet and MCU errata: cross-check the reference manual AND errata sheets via the read-doc skill (python3 .claude/skills/read-doc/search.py <keywords>); a missing erratum workaround is a finding',
'style: repo conventions (TU_ASSERT, no dynamic allocation, include order, naming)',
])
if (!DIMS.length) {
diff --git a/CLAUDE.md b/CLAUDE.md
index 4198081fb..762473714 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -118,7 +118,7 @@ Cutting a release — version bump, regenerated files, the per-release changelog
## References
-- MCU reference manuals, datasheets, schematics: before answering register/bitfield/pinout/errata/timing questions from memory or the web — or changing a specific dcd/hcd driver — use the `read-doc` skill (`.claude/skills/read-doc/SKILL.md`) to cross-check against docs in `$HOME/Documents/calibre-library`; tell the user if the needed document is missing (skill no-ops if the library is absent).
+- MCU reference manuals, datasheets, schematics: before answering register/bitfield/pinout/errata/timing questions from memory or the web — or changing a specific dcd/hcd driver — use the `read-doc` skill (`.claude/skills/read-doc/SKILL.md`) to cross-check against the maintainer's document library; tell the user if the needed document is missing (skill no-ops if the library is absent). Never search the library tree directly — the skill owns its location and search.
- Linux kernel behaviour (usbfs, usbtest, sysfs attributes, device locks, D state): never
infer it from symptoms — read the source for the *running* version. It refutes as often
as it confirms: it has killed two plausible dcd theories and corrected a recovery skill's