Merge pull request #3829 from hathach/claude/read-doc-calibre-db

read-doc skill: search the document library's database instead of the filesystem

10 files changed, 165 insertions, 25 deletions

diff --git a/.claude/agents/driver-reviewer.md b/.claude/agents/driver-reviewer.md
index f45eca03e..9ce8b621f 100644
--- a/.claude/agents/driver-reviewer.md
+++ b/.claude/agents/driver-reviewer.md
@@ -1,7 +1,7 @@
 ---
 name: driver-reviewer
 description: Review one TinyUSB driver directory or one diff against one review dimension (correctness, ISR safety, datasheet/errata conformance, style) with coverage-first structured findings; or adversarially verify a single finding / fix. Read-only.
-tools: Bash, Read, Grep, Glob
+tools: Bash, Read, Grep, Glob, Skill
 model: opus
 ---
 
@@ -9,7 +9,7 @@ You review exactly the scope given in your prompt (one driver directory, or one
 
 ## Datasheets & errata
 
-For register-use review, find the MCU/USB-IP reference manual in `$HOME/Documents/calibre-library` — and ALSO search the library for the part's errata / silicon-bug sheets (search terms: "errata" plus the MCU or USB-IP name). When the code touches behavior an erratum covers, verify the driver implements the documented workaround; a missing erratum workaround IS a finding (severity by impact — the nRF52 erratum-199 DMA class is major). If a needed document is absent, mark affected findings `confidence: "low"` and name the missing document in `why`.
+For register-use review, find the MCU/USB-IP reference manual with the `read-doc` skill — `python3 .claude/skills/read-doc/search.py <keywords>`, never `find`/`grep` over the library tree — and ALSO search for the part's errata / silicon-bug sheets (search terms: "errata" plus the MCU or USB-IP name). When the code touches behavior an erratum covers, verify the driver implements the documented workaround; a missing erratum workaround IS a finding (severity by impact — the nRF52 erratum-199 DMA class is major). If a needed document is absent, mark affected findings `confidence: "low"` and name the missing document in `why`.
 
 ## Reporting discipline
 
diff --git a/.claude/agents/port-dev.md b/.claude/agents/port-dev.md
index 76bafb39b..77a28bafa 100644
--- a/.claude/agents/port-dev.md
+++ b/.claude/agents/port-dev.md
@@ -16,7 +16,7 @@ You implement exactly one specified change in one assigned scope (a directory un
 
 ## Datasheets
 
-When changing dcd/hcd register logic, cross-check the MCU reference manual / datasheet / programming guide in `$HOME/Documents/calibre-library` (search by MCU or USB-IP name). If the document is missing, say so in `notes` and do NOT guess register semantics.
+When changing dcd/hcd register logic, cross-check the MCU reference manual / datasheet / programming guide with the `read-doc` skill — `python3 .claude/skills/read-doc/search.py <MCU or USB-IP name>`, never `find`/`grep` over the library tree. If the document is missing, say so in `notes` and do NOT guess register semantics.
 
 ## Finish checklist (in order)
 
diff --git a/.claude/skills/etm-trace/SKILL.md b/.claude/skills/etm-trace/SKILL.md
index 99e89729c..43cf6a2a5 100644
--- a/.claude/skills/etm-trace/SKILL.md
+++ b/.claude/skills/etm-trace/SKILL.md
@@ -147,7 +147,7 @@ request `itrace.csv`, `profile_lines.csv`, `profile_insts.csv`, `samples.csv`,
 
 Bring-up ladder — each step gates the next:
 
-1. **Docs before hardware** (calibre library first, then vendor site): board
+1. **Docs before hardware** (`read-doc` skill first, then vendor site): board
    manual, schematics, MCU reference manual. Establish the trace clock
    source and max — chip side and probe side (J-Trace PRO Cortex-M tops out
    at a 150 MHz trace clock) — the pins carrying TRACE_CLK/D0-D3 (read the board's
diff --git a/.claude/skills/etm-trace/boards.md b/.claude/skills/etm-trace/boards.md
index 4a5f297ae..044d4e0ee 100644
--- a/.claude/skills/etm-trace/boards.md
+++ b/.claude/skills/etm-trace/boards.md
@@ -52,7 +52,7 @@ Board caveats (beyond the table):
   `AfterTargetConnect` hook; un-attachable after a killed session →
   power-cycle.
 - **metro_m7_1011** (RT1011): a custom Adafruit rev with a hand-added 2x10
-  ETM header (KiCad schematic in the calibre library). No SEGGER RT1011
+  ETM header (KiCad schematic via the `read-doc` skill). No SEGGER RT1011
   example exists — the committed .jdebug (tuned +50 ps) is the known-good
   reference. BOARD_BootClockRUN sets the 132 MHz trace root but leaves it
   gated; `trace_etm_init` ungates it. The first Ozone run after a fresh
diff --git a/.claude/skills/read-doc/SKILL.md b/.claude/skills/read-doc/SKILL.md
index df845e7b5..feaa914af 100644
--- a/.claude/skills/read-doc/SKILL.md
+++ b/.claude/skills/read-doc/SKILL.md
@@ -8,17 +8,22 @@ description: Use when you need authoritative hardware/protocol facts from a prim
 ## Overview
 
 Some maintainers keep datasheets, manuals, and books in a Calibre library at
-`$HOME/Documents/calibre-library/`, laid out as
-`AUTHOR/TITLE (id)/TITLE - AUTHOR.pdf|.epub`. For hardware/protocol facts —
-registers, bitfields, memory maps, pinouts, electrical/timing specs, errata, USB
-spec — read the doc instead of answering from training knowledge or the web.
+`$HOME/Documents/calibre-library/`. For hardware/protocol facts — registers,
+bitfields, memory maps, pinouts, electrical/timing specs, errata, USB spec —
+read the doc instead of answering from training knowledge or the web.
+
+Search the library's `metadata.db`, never the filesystem. The database indexes
+title, authors, tags, series, publisher, description and the stored filename;
+most part numbers live in the tags, which the filesystem does not carry.
 
 ## Gate first
 
-The library is per-user. Check it exists before anything else:
+The library is per-user and usually on a network mount, so test the database
+file, not the directory — an unmounted or half-synced mountpoint is still a
+directory:
 
 ```bash
-[ -d "$HOME/Documents/calibre-library" ] && echo present || echo absent
+[ -f "${CALIBRE_LIBRARY:-$HOME/Documents/calibre-library}/metadata.db" ] && echo present || echo absent
 ```
 
 Absent → the skill does not apply; fall back to normal sources silently (don't
@@ -35,27 +40,50 @@ Not for general concepts, repo/code questions, or when no such doc is likely.
 
 ## Find
 
-Keywords from `/read-doc <keywords>`, else derived from the question (part number,
-peripheral, spec name). AND them with chained case-insensitive grep:
+Keywords from `/read-doc <keywords>`, else derived from the question (part
+number, peripheral, spec name). `search.py` ANDs them across every metadata
+field and prints the best matches first — at most 40, and the header says when
+more matched:
 
 ```bash
-find "$HOME/Documents/calibre-library/" -maxdepth 3 \( -iname '*.pdf' -o -iname '*.epub' \) | grep -i "kw1" | grep -i "kw2"
+python3 .claude/skills/read-doc/search.py errata RT1064      # AND (default)
+python3 .claude/skills/read-doc/search.py RT1060 RT1064 --any
 ```
 
-One match → read it. Several → list and ask via AskUserQuestion. None → drop the
-weakest keyword and broaden (filenames hold title+author, not tags); still none →
-list the closest author/title matches.
+Exit 0 matched, 1 nothing matched, 2 bad usage or no library — 2 means the
+search never ran, so fix the invocation instead of broadening.
+
+One match → read it. Several → list and ask via AskUserQuestion. Nothing
+(exit 1) → retry with fewer keywords; the part number alone often works where
+`<part> datasheet` does not, because words like "datasheet" and "manual" are
+rarely in the metadata. `--any` only changes anything with two or more
+keywords. Still nothing → say the document is missing rather than answering
+from memory.
+
+Set `CALIBRE_LIBRARY` to search a library elsewhere.
 
 ## Read
 
-- **PDF:** Read with `pages`; for >10 pages start `pages: "1-20"` (TOC/overview),
+`search.py` prints one `FORMAT path` line per stored file:
+
+- **PDF** — Read with `pages`; for >10 pages start `pages: "1-20"` (TOC/overview),
   report the page count, then read sections on demand.
-- **EPUB:** Read the path directly.
-- Summarize in one line (title, pages, coverage) and keep as reference context.
+- **Any other format** (EPUB, MOBI, CHM, ZIP…) — Read has no decoder for these
+  and returns mojibake rather than an error. Say the document is not in a
+  readable format; do not paste what Read returned.
+- **`MISSING`** — the metadata is real but the file is not on disk (library
+  mid-sync, or the file was deleted). Report the file as unavailable, not the
+  document as nonexistent.
+
+Summarize in one line (title, pages, coverage) and keep as reference context.
 
 ## Common mistakes
 
+- Searching with `find`/`grep` over the library tree. It sees only truncated
+  filenames, missing the tags, series and descriptions where part numbers and
+  errata IDs actually live. Query the database.
 - Skipping the gate on a machine with no library.
 - Answering a register/spec question from memory when the datasheet is on disk.
 - Loading a 1000-page PDF up front instead of TOC-first.
-- Requiring all keywords to match — broaden on zero hits.
+- Requiring all keywords to match — broaden, or use `--any`, on zero hits.
+- Treating a `MISSING` file, or an exit 2, as proof the document is absent.
diff --git a/.claude/skills/read-doc/search.py b/.claude/skills/read-doc/search.py
new file mode 100755
index 000000000..c70d36805
--- /dev/null
+++ b/.claude/skills/read-doc/search.py
@@ -0,0 +1,112 @@
+#!/usr/bin/env python3
+"""Search the Calibre library by metadata and print matching document paths.
+
+Usage: search.py KEYWORD [KEYWORD...]        all keywords must match (AND)
+       search.py --any KEYWORD [KEYWORD...]  any keyword matches (OR)
+
+Matches title, authors, tags, series, publisher, description and stored
+filename, and prints the exact path to read, best match first.
+
+Exit 0 matched, 1 nothing matched, 2 bad usage or no library.
+"""
+import glob
+import os
+import sqlite3
+import sys
+import unicodedata
+import urllib.parse
+
+LIB = os.path.realpath(os.path.expanduser(os.environ.get("CALIBRE_LIBRARY") or "~/Documents/calibre-library"))
+DB = os.path.join(LIB, "metadata.db")
+LIMIT = 40
+
+QUERY = """
+SELECT b.id, b.title, b.path,
+       (SELECT group_concat(a.name, ', ') FROM authors a
+          JOIN books_authors_link l ON l.author = a.id WHERE l.book = b.id),
+       (SELECT group_concat(t.name, ', ') FROM tags t
+          JOIN books_tags_link l ON l.tag = t.id WHERE l.book = b.id),
+       (SELECT group_concat(s.name, ', ') FROM series s
+          JOIN books_series_link l ON l.series = s.id WHERE l.book = b.id),
+       (SELECT group_concat(p.name, ', ') FROM publishers p
+          JOIN books_publishers_link l ON l.publisher = p.id WHERE l.book = b.id),
+       (SELECT c.text FROM comments c WHERE c.book = b.id),
+       (SELECT group_concat(d.format || '/' || d.name, char(10)) FROM data d WHERE d.book = b.id)
+FROM books b
+"""
+
+_authors = None
+
+
+def norm(s):
+    # NFKC + casefold so MICRO SIGN/GREEK MU, curly quotes and dashes compare equal.
+    return unicodedata.normalize("NFKC", s).casefold()
+
+
+def resolve(bid, path, fmt, name):
+    """Absolute path of one format row, or None if the file is not on disk.
+
+    Calibre renames `<author>/<title> (<id>)` when metadata is edited and leaves
+    the old directory behind, so on a miss retry by the stable book id.
+    """
+    ext = "." + fmt.lower()
+    exact = os.path.join(LIB, path, name + ext)
+    if os.path.exists(exact):
+        return exact
+    global _authors
+    if _authors is None:
+        _authors = {}
+        for d in os.listdir(LIB):  # case-only duplicates exist on a case-sensitive mount
+            _authors.setdefault(d.lower(), []).append(d)
+    for author in _authors.get(path.split("/")[0].lower(), ()):
+        for d in glob.glob(os.path.join(glob.escape(os.path.join(LIB, author)), "* (%d)" % bid)):
+            for f in sorted(glob.glob(os.path.join(glob.escape(d), "*" + ext))):
+                return f
+    return None
+
+
+def main(argv):
+    match_any = "--any" in argv
+    keywords = [norm(k) for k in argv if k != "--any"]
+    if not keywords:
+        print(__doc__, file=sys.stderr)
+        return 2
+
+    if not os.path.exists(DB):
+        print(f"no Calibre database at {DB}", file=sys.stderr)
+        return 2
+
+    db = sqlite3.connect("file:" + urllib.parse.quote(DB) + "?mode=ro", uri=True)
+    hits = []
+    for bid, title, path, authors, tags, series, publisher, comments, files in db.execute(QUERY):
+        entries = [e.split("/", 1) for e in (files or "").split("\n") if e]
+        hay = norm(" ".join(x for x in (title, authors, tags, series, publisher, comments) if x)
+                   + " " + " ".join(n for _, n in entries))
+        found = sum(k in hay for k in keywords)
+        if not found or (not match_any and found < len(keywords)):
+            continue
+        in_title = sum(k in norm(title) for k in keywords)
+        hits.append((-found, -in_title, title, authors, tags, bid, path, entries))
+
+    if not hits:
+        print("no match")
+        return 1
+
+    hits.sort(key=lambda h: h[:3])  # authors/tags may be None and are not comparable
+    print(f"{len(hits)} book(s)" + (f", showing the {LIMIT} best" if len(hits) > LIMIT else ""))
+    for _, _, title, authors, tags, bid, path, entries in hits[:LIMIT]:
+        print(f"\n{title}" + (f"  [{authors}]" if authors else "") + (f"  tags: {tags}" if tags else ""))
+        if not entries:
+            print("  (no file in this library)")
+        for fmt, name in entries:
+            p = resolve(bid, path, fmt, name)
+            print(f"  {fmt} {p}" if p else f"  {fmt} MISSING (library mid-sync or file deleted)")
+    return 0
+
+
+if __name__ == "__main__":
+    try:
+        sys.exit(main(sys.argv[1:]))
+    except BrokenPipeError:
+        os.dup2(os.open(os.devnull, os.O_WRONLY), sys.stdout.fileno())
+        sys.exit(0)
diff --git a/.claude/skills/target-debug/SKILL.md b/.claude/skills/target-debug/SKILL.md
index 9a61a86c7..050a697b9 100644
--- a/.claude/skills/target-debug/SKILL.md
+++ b/.claude/skills/target-debug/SKILL.md
@@ -347,7 +347,7 @@ the wire itself: `usb-sniffer` skill (hardware tap, PID-level).
 - J-Link (UM08001): <https://kb.segger.com/UM08001_J-Link_/_J-Trace_User_Guide> — flash breakpoints, RTT, SWO, monitor mode, Commander.
 - OpenOCD: <https://openocd.org/doc/html/index.html> — `rtt`, `bp`/`wp`, `cortex_m vector_catch`/`maskisr`, `itm`/`tpiu`.
 - "Debugging with GDB" (§5.1 = break/watch/dprintf): Tenth Edition (GDB 18)
-  via calibre/`read-doc`, or
+  via the `read-doc` skill, or
   `curl -sL -o /tmp/gdb.pdf https://sourceware.org/gdb/current/onlinedocs/gdb.pdf`
   (the HTML mirror blocks fetchers). Installed `arm-none-eabi-gdb`
   `help <cmd>` is authoritative here.
diff --git a/.claude/skills/usbtest/SKILL.md b/.claude/skills/usbtest/SKILL.md
index 6197ccd51..32c2be913 100644
--- a/.claude/skills/usbtest/SKILL.md
+++ b/.claude/skills/usbtest/SKILL.md
@@ -121,7 +121,7 @@ curl -sO "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/plain
 3. **On-device gdb/openocd**: read the EP control registers and DCD structs at the hang.
 4. Heisenbugs (vanish under logging): RAM ring-buffer trace dumped over openocd; for silent lockups
    JLink PC-sampling (`halt`+`regs` repeatedly — a pinned PC names the spin).
-5. **Cross-check the reference manual** (calibre library) before changing any register-level code —
+5. **Cross-check the reference manual** (`read-doc` skill) before changing any register-level code —
    per CLAUDE.md, and because comments/assumptions in DCDs have been wrong about hardware caps.
 6. Check the vendor's **silicon errata** early for timing/DMA hangs (an unimplemented erratum
    workaround caused a case-10 hang on one port).
diff --git a/.claude/workflows/driver-review.js b/.claude/workflows/driver-review.js
index 3638aa179..255b8ac74 100644
--- a/.claude/workflows/driver-review.js
+++ b/.claude/workflows/driver-review.js
@@ -16,7 +16,7 @@ if (!args || !Array.isArray(args.dirs) || args.dirs.length === 0) {
 const DIMS = args.question ? [args.question] : (args.dimensions || [
   'correctness: transfer state machines, endpoint bookkeeping, completion and error paths',
   'ISR safety: work deferred to task context, shared-state races, register access ordering',
-  'register use vs datasheet and MCU errata: cross-check the reference manual AND errata sheets in $HOME/Documents/calibre-library; a missing erratum workaround is a finding',
+  'register use vs datasheet and MCU errata: cross-check the reference manual AND errata sheets via the read-doc skill (python3 .claude/skills/read-doc/search.py <keywords>); a missing erratum workaround is a finding',
   'style: repo conventions (TU_ASSERT, no dynamic allocation, include order, naming)',
 ])
 if (!DIMS.length) {
diff --git a/CLAUDE.md b/CLAUDE.md
index 4198081fb..762473714 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -118,7 +118,7 @@ Cutting a release — version bump, regenerated files, the per-release changelog
 
 ## References
 
-- MCU reference manuals, datasheets, schematics: before answering register/bitfield/pinout/errata/timing questions from memory or the web — or changing a specific dcd/hcd driver — use the `read-doc` skill (`.claude/skills/read-doc/SKILL.md`) to cross-check against docs in `$HOME/Documents/calibre-library`; tell the user if the needed document is missing (skill no-ops if the library is absent).
+- MCU reference manuals, datasheets, schematics: before answering register/bitfield/pinout/errata/timing questions from memory or the web — or changing a specific dcd/hcd driver — use the `read-doc` skill (`.claude/skills/read-doc/SKILL.md`) to cross-check against the maintainer's document library; tell the user if the needed document is missing (skill no-ops if the library is absent). Never search the library tree directly — the skill owns its location and search.
 - Linux kernel behaviour (usbfs, usbtest, sysfs attributes, device locks, D state): never
   infer it from symptoms — read the source for the *running* version. It refutes as often
   as it confirms: it has killed two plausible dcd theories and corrected a recovery skill's