diff --git a/.github/skills/lucid-docs/README.md b/.github/skills/lucid-docs/README.md new file mode 100644 index 00000000..022e3555 --- /dev/null +++ b/.github/skills/lucid-docs/README.md @@ -0,0 +1,63 @@ +# Lucid docs v0.1 + +Plain syntax. Precise terms. Necessary detail. + +Reduce reading effort without reducing technical depth. + +## Reader + +The skill defines a mid-level Python reader by assumed knowledge. + +It assumes comfort with idiomatic Python, common language features, standard library conventions, type hints, testing, and ordinary object-oriented design. It does not explain standard Python or established dependencies unless their local use is surprising. It explains repository-specific concepts, hidden constraints, surprising behaviour, and non-obvious interactions. + +The voice is direct when the repository supports a clear claim. Uncertainty is surfaced. + +## Repository glossary + +`docs/glossary.md` is the canonical source for repository-specific terminology. It is user-facing documentation and shared context for coding agents. + +The skill reads the glossary before editing documentation. It adds or updates terms that developers must understand as concepts, while excluding ordinary Python terms, obvious identifiers, and incidental implementation details. + +The glossary owns each canonical definition. Individual documents still provide enough local context to remain readable. Ambiguous or conflicting uses are reported rather than silently resolved. + +## Documentation locality + +Each explanation belongs at the narrowest scope that contains everyone who needs it: + +- module docstrings hold cross-cutting decisions, boundaries, and a short map +- class and function docstrings hold the unit's contract and local why +- inline comments hold the smallest-scope reason or constraint + +Shared reasoning is stated once and referred to by name. Shared contracts live at the abstraction that owns them. Implementations and overrides document only their differences. + +## Proportionality and comments + +Documentation depth follows conceptual weight, not line count. Simple units stay brief. `Args` and `Returns` sections appear only when they add information beyond the signature and type hints. + +A comment earns its place when it names a reasonable wrong turn, hidden constraint, or non-obvious consequence. The code remains responsible for routine mechanics. + +## Our style + +We want to establish a simple term to encapsulate our style as succinctly as possible. + +The established umbrella term is **plain language**, or **plain English** in a UK house style. + +Plain language is defined for an intended audience. It does not require replacing terms that the audience needs and understands. For developer documentation, this means simple sentence structure around exact technical vocabulary. + +The narrower linguistic operation is **syntactic simplification**: making sentence structure easier to parse while preserving meaning. + +The closest formal engineering analogue is **ASD-STE100 Simplified Technical English**. It constrains grammar and general vocabulary while permitting project-specific technical nouns and verbs. This skill borrows that distinction, but it does not claim ASD-STE100 compliance. The full standard is more restrictive and uses American English. + +For this project, the practical label is: + +> Plain English for technical readers. + +## Research links + +- ISO 24495-1 plain language standard: https://www.iso.org/obp/ui/#iso:std:iso:24495:-1:ed-1:v1:en +- GOV.UK guidance on clear language for specialists: https://guidance.publishing.service.gov.uk/writing-to-gov-uk-standards/writing-guidelines/clear-language/ +- IEEE guidance on plain language for engineers: https://procomm.ieee.org/communication-resources-for-engineers/other-topics/plain-language/ +- ASD-STE100 official overview: https://www.asd-ste100.org/about_STE.html +- Google guidance on short technical sentences: https://developers.google.com/tech-writing/one/short-sentences +- Matt Pocock's domain-modelling skill: https://github.com/mattpocock/skills/tree/main/skills/engineering/domain-modeling +- Martin Fowler on ubiquitous language: https://martinfowler.com/bliki/UbiquitousLanguage.html diff --git a/.github/skills/lucid-docs/SKILL.md b/.github/skills/lucid-docs/SKILL.md new file mode 100644 index 00000000..342ae68f --- /dev/null +++ b/.github/skills/lucid-docs/SKILL.md @@ -0,0 +1,141 @@ +--- +name: lucid-docs +description: Edit Python documentation, docstrings, comments, and repository terminology in plain English for technical readers. Reduce reading effort while preserving precise terms, contracts, design reasons, warnings, and non-obvious behaviour. +license: CC0-1.0 +compatibility: Designed for coding agents that can inspect a repository and edit text files. +--- + +# Lucid docs + +## Goal + +Rewrite the specified documentation to reduce reading effort. Do not cut technical depth. + +Write in plain English for technical readers: + +> Plain syntax. Precise terms. Necessary detail. + +Fluent, sophisticated prose is NOT plain English for technical readers. Rewrite sentence structure ruthlessly. Use short sentences and direct verbs. Reduce ideas per sentence, but keep precise technical terms. + +## Reader + +The reader is a mid-level Python developer, with a B2 CEFR reading level (Flesch-Kincaid Grade 8-9). + +Assume the reader can read idiomatic Python and understands common language features, standard library conventions, type hints, testing, and ordinary object-oriented design. + +Do not explain standard Python or established dependencies unless their use here differs from normal expectations. Explain repository-specific concepts, hidden constraints, surprising behaviour, and non-obvious interactions. + +State verified behaviour directly. Surface genuine uncertainty instead of weakening every claim. + +## Repository language + +Treat `docs/glossary.md` as the source of truth for repository-specific terms. Read it before editing documentation. + +Add or update a term when the reader must understand it as a concept. Include domain concepts, named workflows, architectural concepts, and important local distinctions. Exclude ordinary Python terms, obvious identifiers, and incidental implementation details. + +Write the glossary for developers as well as agents. Use the same plain, precise style as the rest of the documentation. Let the glossary own the canonical definition, while each document supplies enough local context to remain readable. + +Do not invent a definition or silently resolve conflicting uses. Report ambiguity when the repository does not support one clear meaning. + +## Editing + +Read enough code, tests, types, nearby documentation, and glossary entries to preserve the meaning. + +Keep information that helps a developer: + +- use the code correctly +- understand a design choice +- preserve an invariant +- predict a side effect or failure +- follow a non-obvious interaction +- avoid a footgun + +Then tighten the prose: + +1. Put the main point first. +2. Give each sentence one job, but clarity always outranks brevity. +3. Put the subject and verb early. +4. Prefer direct verbs to noun-heavy phrases. +5. Remove words that add no meaning. +6. Keep exact technical terms. +7. Check that no condition, warning, or consequence was lost. + +Do not mistake prose that reads fluently and is technically sophisticated for plain English for technical readers. + +Do not minimise the diff on the documentation you've been asked to review. Assume it has been written maximally, and your job is to rewrite minimally. Plain syntax. Precise terms. Necessary detail. + +## House style + +- Use British English. +- Use sentence case headings. +- Do not use semicolons. +- Remove colons and dashes unless they truly simplify. +- Do not start sentences with the FANBOYS set (for, and, nor, but, or, yet, so) unless the sentence stands alone, with its own subject, verb, and stated reason. +- Use present tense for current behaviour. +- Use one term for one concept. +- Describe earlier or planned code only in migration guides. + +## Python documentation + +Put each explanation at the narrowest scope that contains everyone who needs it. State shared reasoning once, then refer to it by name instead of repeating it. + +Use module docstrings for cross-cutting design decisions, boundaries, and a short map of what the module contains. Explain why the module has its shape. Do not restate its implementation. + +Use class and function docstrings for the unit's contract and local why. Record the invariant it relies on, the footgun it avoids, or the important alternative it rejects. Let names, signatures, and type hints carry what they already express. + +Use inline comments for the smallest-scope fact or reason. Place them beside the lines they explain. A comment earns its place when it names a reasonable wrong turn, hidden constraint, or non-obvious consequence. Let the code express routine mechanics. + +Match documentation to conceptual weight. Keep simple units brief. Do not add `Args` or `Returns` sections merely because a function has parameters or returns a value. Give fuller treatment to contracts, decisions, and risks that need it. + +Document a shared contract at the highest abstraction that owns it. Overrides and implementations document only their differences, added constraints, and surprising behaviour. + +Mention another part of the codebase only when the interaction affects correct use or safe maintenance. + +## Final check + +The result should be easier to read and no less exact. + +### Phase 1 + +Confirm that: + +- the first sentence carries the main point +- each sentence is easy to parse +- precise technical vocabulary remains +- repository-specific terms match `docs/glossary.md` +- new or changed concepts are reflected in the glossary +- each explanation sits at the narrowest useful scope +- documentation depth matches conceptual weight +- shared contracts are stated once at the abstraction that owns them +- comments add reasoning the code cannot show +- design reasons and warnings remain +- every claim is supported by the repository +- the prose describes the code that exists + +### Phase 2 + +Perform a final check of the readability score and house style: + +```bash +uv run .github/skills/lucid-docs/readability.py +``` + +This takes the following options: + +* `--top `: only show the N hardest-to-read blocks per file +* `--rules`: comma-separated rule ids to run (default: all) + * `pleng001` readability score + * `pleng002` colon/semicolon/dash + * `pleng003` FANBOYS sentence openers + +Ensure an acceptable reading level for the reader. You may make allowances for the requirements of technical writing. + +Colons and dashes are permissable only if they truly simplify. + +This script is a spot-check, not the target. It cannot detect banned words or fluent-but-wordy prose. + +## Output + +Edit the requested documentation and `docs/glossary.md` in place when tools allow it. + +Report the files changed, unresolved terminology, and any claim that could not be verified. Keep the report brief. diff --git a/.github/skills/lucid-docs/readability.py b/.github/skills/lucid-docs/readability.py new file mode 100644 index 00000000..9ae7cda5 --- /dev/null +++ b/.github/skills/lucid-docs/readability.py @@ -0,0 +1,310 @@ +#!/usr/bin/env python3 +# /// script +# requires-python = ">=3.11" +# dependencies = ["textstat", "typer"] +# /// +"""Run plain-English rules over .py or .md files, one line per finding. + +Rules (run all by default, or pick with --rules): + + pleng001 Flesch-Kincaid readability score per block + pleng002 sentences containing a colon, semicolon, or dash + pleng003 sentences opening with a FANBOYS conjunction + +A "block" is a docstring, comment run (for .py) or a paragraph (for .md). +Blocks below TRIVIAL_WORDS are skipped from scoring entirely (the maths is +meaningless on 1-3 word fragments). Blocks below MIN_WORDS are still scored +but flagged low_confidence. + +Usage: + python readability.py [ ...] + uv run readability.py [ ...] [--top N] [--rules pleng002] +""" + +import ast +import json +import re +import tokenize +from collections.abc import Iterator +from dataclasses import dataclass +from enum import StrEnum +from pathlib import Path + +import textstat +import typer + +MIN_WORDS = 30 +TRIVIAL_WORDS = 5 + +FANBOYS = {"for", "and", "nor", "but", "or", "yet", "so"} + +SENTENCE_SPLIT = re.compile(r"(?<=[.!?])\s+") +PUNCT_MARKS = re.compile(r"[;:]|—|–|(?<=\s)-(?=\s)") +LEADING_WORD = re.compile(r"^[\"'“‘(]*([A-Za-z]+)") +INLINE_CODE = re.compile(r"`([^`]*)`") + + +class Rule(StrEnum): + """Plain-English rule ids, in the order they run and print.""" + + READABILITY = "pleng001" + PUNCTUATION = "pleng002" + FANBOYS = "pleng003" + + +RULE_DESCRIPTIONS = { + Rule.READABILITY: "Flesch-Kincaid readability score per block", + Rule.PUNCTUATION: "sentences containing a colon, semicolon, or dash", + Rule.FANBOYS: "sentences opening with a FANBOYS conjunction", +} + + +def strip_inline_code(text: str) -> str: + """Drop backticks but keep the token inside, so identifiers stay readable.""" + return INLINE_CODE.sub(r"\1", text) + + +def sentences_of(text: str) -> list[str]: + """Split a block into sentences (naive, on . ! ?), dropping empty fragments.""" + parts = SENTENCE_SPLIT.split(text.replace("\n", " ")) + return [s.strip() for s in parts if s.strip()] + + +@dataclass +class BlockScore: + """The readability result for one block, or the reason it wasn't scored.""" + + label: str + words: int + status: str # "", "low_confidence", "trivial" + grade: float | None + ease: float | None + + +def score_block(label: str, text: str) -> BlockScore: + """Score a block, or mark it trivial instead of scoring meaningless text.""" + word_count = len(text.split()) + if word_count < TRIVIAL_WORDS: + return BlockScore(label, word_count, "trivial", None, None) + grade = textstat.flesch_kincaid_grade(text) + ease = textstat.flesch_reading_ease(text) + status = "low_confidence" if word_count < MIN_WORDS else "" + return BlockScore(label, word_count, status, grade, ease) + + +def format_score(score: BlockScore) -> str: + """Format a scored block as a single key=value line.""" + status = f" status={score.status}" if score.status else "" + return ( + f"{score.label} words={score.words}{status} " + f"flesch_kincaid_grade={score.grade} flesch_reading_ease={score.ease}" + ) + + +def punctuation_sentences(text: str) -> list[str]: + """Return each sentence in text that contains a colon, semicolon, or dash.""" + return [s for s in sentences_of(text) if PUNCT_MARKS.search(s)] + + +def fanboys_sentences(text: str) -> list[tuple[str, str]]: + """Return (word, sentence) pairs for sentences opening with a FANBOYS word.""" + found = [] + for s in sentences_of(text): + m = LEADING_WORD.match(s) + if m and m.group(1).lower() in FANBOYS: + found.append((m.group(1).lower(), s)) + return found + + +def blocks_from_comments(path: Path) -> Iterator[tuple[str, str]]: + """Yield (label, text) pairs for each run of consecutive whole-line comments.""" + with open(path, "rb") as f: + run: list = [] + start_line = None + prev_line = None + for tok in tokenize.tokenize(f.readline): + if tok.type != tokenize.COMMENT: + continue + text = tok.string.lstrip("#").strip() + if run and prev_line == tok.start[0] - 1: + run.append(text) + else: + if run: + yield f"line={start_line} block=comment", " ".join(run) + run = [text] + start_line = tok.start[0] + prev_line = tok.start[0] + if run: + yield f"line={start_line} block=comment", " ".join(run) + + +def blocks_from_py(path: Path) -> Iterator[tuple[str, str]]: + """Yield (label, text) pairs for each docstring and comment block in a .py file.""" + tree = ast.parse(path.read_text()) + doc_nodes = (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef, ast.Module) + for node in ast.walk(tree): + if not isinstance(node, doc_nodes): + continue + doc = ast.get_docstring(node) + if not doc: + continue + name = getattr(node, "name", "module") + line = getattr(node, "lineno", 1) + yield f"line={line} block={name}", strip_inline_code(doc) + for label, text in blocks_from_comments(path): + yield label, strip_inline_code(text) + + +def blocks_from_md(path: Path) -> Iterator[tuple[str, str]]: + """Yield (label, paragraph) pairs for each paragraph, with markup stripped.""" + text = path.read_text() + # Blank out code blocks but keep their newlines, so line numbers stay accurate. + text = re.sub( + r"```.*?```", lambda m: "\n" * m.group(0).count("\n"), text, flags=re.DOTALL + ) + text = strip_inline_code(text) + text = re.sub(r"!\[.*?\]\(.*?\)", "", text) # images + text = re.sub(r"\[([^\]]*)\]\([^)]*\)", r"\1", text) # links + text = re.sub(r"^#{1,6}\s*", "", text, flags=re.MULTILINE) # headings + text = re.sub(r"\*\*([^*]+)\*\*", r"\1", text) # bold + text = re.sub(r"\*([^*]+)\*", r"\1", text) # italic + # admonition marker lines (mkdocs-style), e.g. !!! warning "title" + text = re.sub(r'^!!!\s+\S+(\s+".*?")?\s*$', "", text, flags=re.MULTILINE) + + table_row = re.compile(r"^\s*\|.*\|\s*$") + para_lines: list = [] + start_line = None + for i, raw_line in enumerate(text.split("\n"), start=1): + if table_row.match(raw_line): + if para_lines: + yield f"line={start_line} block=paragraph", "\n".join(para_lines) + para_lines, start_line = [], None + continue + if raw_line.strip(): + if start_line is None: + start_line = i + para_lines.append(raw_line) + else: + if para_lines: + yield f"line={start_line} block=paragraph", "\n".join(para_lines) + para_lines, start_line = [], None + if para_lines: + yield f"line={start_line} block=paragraph", "\n".join(para_lines) + + +def score_file(path: Path, top: int | None, rules: list[Rule]) -> None: + """Extract blocks from a file, print a summary, then each active rule's findings.""" + if path.suffix == ".py": + blocks = list(blocks_from_py(path)) + elif path.suffix == ".md": + blocks = list(blocks_from_md(path)) + else: + typer.echo(f"Unsupported file type: {path.suffix}", err=True) + return + + if not blocks: + print(f"file={path} status=no_prose_found") + return + + summary = [f"file={path}"] + + scored: list[BlockScore] = [] + if Rule.READABILITY in rules: + scores = [score_block(label, text) for label, text in blocks] + scored = [s for s in scores if s.grade is not None] + trivial = sum(1 for s in scores if s.status == "trivial") + low_conf = sum(1 for s in scores if s.status == "low_confidence") + worst = max((s.grade for s in scored), default=None) + worst_str = f"{worst:.1f}" if worst is not None else "none" + summary.append(f"blocks={len(blocks)}") + summary.append(f"trivial={trivial}") + summary.append(f"low_confidence={low_conf}") + summary.append(f"worst_grade={worst_str}") + + punct_flags: list[tuple[str, str]] = [] + if Rule.PUNCTUATION in rules: + punct_flags = [ + (label, s) for label, text in blocks for s in punctuation_sentences(text) + ] + summary.append(f"punctuation_flags={len(punct_flags)}") + + fanboys_flags: list[tuple[str, str, str]] = [] + if Rule.FANBOYS in rules: + fanboys_flags = [ + (label, word, s) + for label, text in blocks + for word, s in fanboys_sentences(text) + ] + summary.append(f"fanboys_flags={len(fanboys_flags)}") + + print(" ".join(summary)) + + if Rule.READABILITY in rules: + shown = scored + if top is not None: + shown = sorted(scored, key=lambda s: s.grade, reverse=True)[:top] + for score in shown: + print(format_score(score)) + + for label, sentence in punct_flags: + print(f"{label} sentence={json.dumps(sentence)}") + + for label, word, sentence in fanboys_flags: + print(f"{label} word={word} sentence={json.dumps(sentence)}") + + +def parse_rules(value: str | None) -> list[Rule]: + """Parse a comma-separated --rules value, defaulting to every rule.""" + if not value: + return list(Rule) + selected: list[Rule] = [] + for token in value.split(","): + token = token.strip() + if not token: + continue + try: + selected.append(Rule(token)) + except ValueError: + valid = ", ".join(r.value for r in Rule) + typer.echo(f"Unknown rule: {token}. Valid: {valid}", err=True) + raise typer.Exit(1) from None + return selected + + +def main( + paths: list[Path] = typer.Argument( # noqa: B008 + ..., help="One or more .py/.md files or directories to score." + ), + top: int | None = typer.Option( + None, "--top", help="Show only the N worst-scoring blocks per file." + ), + rules: str | None = typer.Option( + None, + "--rules", + help="Comma-separated rule ids to run (default: all). " + + "; ".join(f"{r.value}={d}" for r, d in RULE_DESCRIPTIONS.items()), + ), +) -> None: + """Score .py/.md files or directories against one or more plain-English rules.""" + selected_rules = parse_rules(rules) + + files: list = [] + for path in paths: + if not path.exists(): + typer.echo(f"Path not found: {path}", err=True) + raise typer.Exit(1) + if path.is_dir(): + found = sorted(p for p in path.rglob("*") if p.suffix in (".py", ".md")) + if not found: + typer.echo(f"No .py or .md files found under {path}", err=True) + raise typer.Exit(1) + files.extend(found) + else: + files.append(path) + + for file in files: + score_file(file, top=top, rules=selected_rules) + + +if __name__ == "__main__": + typer.run(main) diff --git a/docs/glossary.md b/docs/glossary.md new file mode 100644 index 00000000..36681f03 --- /dev/null +++ b/docs/glossary.md @@ -0,0 +1,127 @@ +# Glossary + +A list of repository-specific term definitions, used consistently across matchlab's +code and documentation. + +## Adapter + +Where a plan's artifacts are stored and read back, keyed by [fingerprint](#fingerprint). +An adapter is storage, not an engine — it does not resolve anything itself. +`DuckDBAdapter` is the reference implementation, backing a single DuckDB database file +(or `:memory:`). + +## Artifact + +The stored output of one [step](#step): a source's extract and leaf assignment, a +view's materialised table, a model's edge list, or a resolver's complete resolution. A +[store](#store) keeps every artifact it is given until something explicitly +[trims](#trim) it. + +## Collect + +Run a plan. `collect()` walks a plan upstream-first and runs only the steps that aren't +already in the store, so re-collecting an unchanged plan does no work and adding a step +to a collected plan runs only the new one. See [content-addressed](#content-addressed). + +## Content-addressed + +Identified by what it is, not by when or how it was built. A step's +[fingerprint](#fingerprint) comes from its own configuration and its inputs' +fingerprints, so the same plan always keys the same artifact — collecting it twice +reads the cached one back rather than recomputing it. + +## Entity + +The real-world thing several records refer to. A [resolver](#resolver) groups records +under one [root](#root) ID when it decides they describe the same entity. + +## Fingerprint + +The 32-byte SHA-256 digest that keys a step's stored [artifact](#artifact). Two steps +with identical configuration and identical input fingerprints hash to the same +fingerprint, which is what makes a store [content-addressed](#content-addressed). + +## Label + +A pointer from a name someone chose to a resolution's [fingerprint](#fingerprint), +created by `publish()`. A label belongs to the store, not the plan — it can be re-aimed +at a different resolution, whereas a source's `name` is part of that source's own +output and never moves. + +## Leaf + +The stable ID of one record — a hash of its content, not its key. Identity coming from +content rather than key is why a leaf ID changes only when the underlying data does, +which is the basis matchlab anchors evaluation judgements to. Reading a source or view +directly, without going through a resolver, exposes the leaf as the `id` column. + +## Merge-forward + +The guarantee that a resolver's stored resolution carries forward every record +reachable from its inputs, not just the ones its own models formed edges over. A record +no model touches keeps its upstream grouping, or becomes a singleton, rather than +vanishing. matchlab's resolvers materialise this complete table once, at collect time, +rather than resolving on demand. + +## Model + +A step that scores candidate matches: `.dedupe()` within one view, or `.link()` between +two. A model produces edges, not clusters — turning edges into entities is a +[resolver's](#resolver) job. + +## Plan + +A tree of [steps](#step). Each step holds a reference to its own inputs, so the step +you are holding is the pipeline — there is no separate object to register steps with. +Nothing runs until you [collect](#collect) it. + +## Position + +A step's place in `collect()`'s run order, shown in brackets by `draw()` and quoted in +logs (`[step 5]`). Steps have no names, so position is how a log line, a drawn tree, and +`plan.lineage()` all refer to the same step. Positions are relative to the step a plan +or drawing starts from, so a sub-plan numbers its steps differently from the full plan +it came from. + +## Resolver + +A step that collapses a model's scored edges into clusters, one per [entity](#entity). +`.resolve()` defaults to connected components. A resolver's stored resolution is always +complete and [merge-forward](#merge-forward). + +## Root + +The ID of a cluster a resolver produces — a hash of the sorted set of [leaf](#leaf) IDs +it contains. Two runs that produce the same clustering produce the same root ID, +whatever order the underlying algorithm found its clusters in. Reading a view through a +resolver exposes the root as the `id` column, so several records can share one `id`. + +## Source + +Where a plan starts: a warehouse query, plus the column that keys it. Every column the +query returns is part of a record's identity, so two rows are the same record exactly +when the query returns identical values for both. + +## Step + +One node in a [plan](#plan): a source, view, model, or resolver. A step's kind decides +what it produces, and what [artifact](#artifact) a store keeps for it. + +## Store + +Where a plan's artifacts live once collected — the reference implementation is a +DuckDB database. A store keeps everything given to it until something explicitly +[trims](#trim) it or the file is deleted. + +## Trim + +Delete every artifact except the ones named or published, and reclaim the space that +frees. For a file-backed store this rewrites the file, because deleting rows inside it +does not return space to the operating system. + +## View + +A step that says which records a model matches over, and what shape they're in. Only +the columns named in its cleaning survive, plus `id` — the grouping the model matches +on. + diff --git a/docs/guide/build-a-plan.md b/docs/guide/build-a-plan.md index ccb5ac95..6f70e167 100644 --- a/docs/guide/build-a-plan.md +++ b/docs/guide/build-a-plan.md @@ -1,7 +1,7 @@ # Build a plan A plan is a tree of steps. Each step holds a reference to its inputs, so the node you -are holding *is* the pipeline — there is no separate container object to register +are holding *is* the pipeline. There is no separate container object to register things with. Nothing runs until you call `collect()`. @@ -24,17 +24,17 @@ crn = Source( `key_field` is the identifier you'll get back in results. It's read as a string whatever the warehouse stores it as, so an integer primary key needs no ceremony. -`name` qualifies every column this source contributes — `company` becomes -`crn_company` — and those names end up in cleaning SQL, so it has to work as the start -of a column name: a letter or underscore, then letters, digits and underscores. A -hyphen or a dot would parse as arithmetic or as a table reference, so matchlab rejects -them when you build the source rather than letting it fail three steps later. SQL +`name` qualifies every column this source contributes. `company` becomes +`crn_company`. Those names end up in cleaning SQL, so a name must start with a letter +or underscore, then hold only letters, digits and underscores. A hyphen or a dot would +parse as arithmetic or as a table reference. This means matchlab rejects an invalid +name when you build the source, rather than letting it fail three steps later. SQL keywords are fine, since the name is only ever a prefix. -**The `select` is the whole declaration.** Every other column it returns is part of the -record, and so part of that record's identity: two rows are the same record exactly +**The `select` is the whole declaration.** Every column it returns becomes part of the +record, and so part of that record's identity. Two rows are the same record exactly when the extract returns identical values for both. Above, a company appearing twice -with the same name and town is one record; change the town and it's two. +with the same name and town is one record. Change the town, and it's two. There's no separate list of fields to index. That means: @@ -45,9 +45,9 @@ There's no separate list of fields to index. That means: * **Changing the warehouse data behind any selected column invalidates the source**, and everything downstream of it. -You can still select a column purely to look at — `view_entity` and the evaluation -samplers show every column the extract returned, reading it back from the copy cached at -collect time — but be aware that selecting it makes it count. +You can still select a column purely to look at. `view_entity` and the evaluation +samplers show every column the extract returned, reading it back from the copy cached +at collect time, but selecting it still makes it count. ## Verbs @@ -69,9 +69,9 @@ crn.dedupe(model_class=NaiveDeduper, model_settings={...}) crn.link(dh, model_class=DeterministicLinker, model_settings={...}) ``` -Both sides of a link are covered — passing a `Source` where a view is expected views -it. Reach for `.view()` when you want to do one of the three things only a view can do: -clean columns, `group`, or read through a resolver. +Both sides of a link are covered. Passing a `Source` where a view is expected views +it. Reach for `.view()` when you want to do one of the three things only a view can +do: clean columns, `group`, or read through a resolver. ### Views @@ -87,7 +87,7 @@ cleaned = crn.view( ) ``` -Only the columns you name survive, plus `id` — the grouping the model matches on. +Only the columns you name survive, plus `id`, the grouping the model matches on. `source.f("field")` gives you the source-qualified column name (`crn_company`), which is how fields are named once a view is built. @@ -98,7 +98,7 @@ is how fields are named once a view is built. #### What `id` is, and when to group -Read a source directly and `id` is the record — one row each. Read it *through a +Read a source directly and `id` is the record, one row each. Read it *through a resolver* and `id` is the resolver's entity, so several records share one: ```python @@ -111,7 +111,7 @@ deduped.view(crn, cleaning={"name": "crn_company"}).data() ``` That's often what you want — more evidence per entity. When it isn't, `group=True` -collapses each `id` to one row, and every expression becomes an aggregate so you say +collapses each `id` to one row. Every expression then becomes an aggregate, so you say how each column combines: ```python @@ -130,7 +130,7 @@ deduped.view( ``` Any DuckDB aggregate works, `list` and `string_agg` included. A non-aggregate gets you -DuckDB's own error naming the column. `group=True` needs cleaning expressions — there's +DuckDB's own error naming the column. `group=True` needs cleaning expressions. There's no sensible default for how a column collapses. Grouping matters most when a view reads **several** sources through a resolver. Those @@ -147,7 +147,7 @@ resolver.view(crn, dh, cleaning={"c": "crn_company", "d": "dh_company"}).data() ``` A comparison on `l.d` is null on every crn row, so the entity can't be matched on its -combined evidence. Grouping puts it on one populated row — `any_value` skips nulls: +combined evidence. Grouping puts it on one populated row. `any_value` skips nulls: ```python resolver.view( @@ -164,8 +164,9 @@ resolver.view( # acme | ["london", "leeds", "bristol"] ``` -Grouping changes what the *model* sees, never the resolution: record identity travels -separately, so a resolver below a grouped view still carries every record forward. +Grouping changes what the *model* sees. It never changes the resolution. Record +identity travels separately, so a resolver below a grouped view still carries every +record forward. ### Deduplicating and linking @@ -197,8 +198,8 @@ entities = deduped.resolve() `resolve()` defaults to connected components. Pass `resolver_class` and `resolver_settings` for something else. -A resolver takes several models, so you can resolve multiple methodologies together — -and trust a strict one further than a loose one by giving each a score threshold: +A resolver takes several models, so you can resolve multiple methodologies together. +Give each a score threshold to trust a strict one further than a loose one: ```python entities = crn_dedupe.resolve( @@ -208,7 +209,8 @@ entities = crn_dedupe.resolve( ) ``` -Thresholds take the model itself, not its name — you're already holding it. Any model with no threshold will contribute every edge. +Thresholds take the model itself, not its name. You're already holding it. Any model +with no threshold will contribute every edge. ## Layering @@ -225,7 +227,7 @@ entities = ( ``` The link now sees crn's deduplicated clusters rather than its raw rows. Records the -link never matches keep their upstream grouping — a resolver always carries its inputs' +link never matches keep their upstream grouping. A resolver always carries its inputs' resolutions forward, so nothing silently reverts to singletons. ## Collecting @@ -237,22 +239,22 @@ entities.collect() `collect()` walks the plan upstream-first and runs only what isn't already stored. Steps are content-addressed by their configuration and their inputs' fingerprints, so: -* re-collecting an unchanged plan does no work; -* adding a step to a collected plan runs only the new step; +* re-collecting an unchanged plan does no work +* adding a step to a collected plan runs only the new step * rebuilding the same plan in a new process is a cache hit, provided the warehouse data - hasn't changed. + hasn't changed -Sources are the exception — they hash the data they read, which is how a plan notices -the warehouse moved. Constructing a *fresh* `Source` re-reads it; an existing `Source` -object remembers. +Sources are the exception. They hash the data they read, which is how a plan notices +the warehouse has moved. Constructing a *fresh* `Source` re-reads the data. An +existing `Source` object remembers it. !!! warning "Seed anything non-deterministic" A step's cache key comes from its configuration, not from its output. If a model can produce different results from the same settings, the first result is cached - and reused. In practice this means passing a `seed` to Splink training functions - that sample — otherwise re-running gives you the cache, not a second opinion. + and reused. In practice, this means passing a `seed` to Splink training functions + that sample. Otherwise, re-running gives you the cache, not a second opinion. -Because the key is configuration-derived, it is also conservative: editing a cleaning +Because the key is configuration-derived, it is also conservative. Editing a cleaning expression in a way that doesn't change the data still re-runs everything below it. To collect somewhere other than the default store: @@ -264,7 +266,7 @@ entities.collect(adapter=DuckDBAdapter("./run.duckdb")) ### Watching it run At a terminal, `collect()` draws the plan as a tree and redraws it in place as each -step settles — one frame, not one tree per step: +step settles. That's one frame, not one tree per step: ``` ○ [6] resolver(Components) @@ -278,26 +280,26 @@ step settles — one frame, not one tree per step: ○ waiting ◐ running ● ran ``` -The number in brackets is the step's **position**, and it is the same number -everywhere: `[5]` here is `[step 5]` in the log and `steps[5]` in a -[document](../api/steps.md). Since steps have no names, that cross-reference is how -you know which node a line is about — which is why every mode puts the tree somewhere. +The number in brackets is the step's **position**. It's the same number everywhere. +`[5]` here is `[step 5]` in the log and `steps[5]` in a +[document](../api/steps.md). Steps have no names, so that cross-reference is how +you know which node a line is about. That's why every mode puts the tree somewhere. Next to it is what the step *is*. A model and a resolver name the class implementing them in parentheses, so `[5]` reads as the naive dedupe rather than as another -anonymous `model` line; a source names itself in quotes, since a source is the one step -with a name. A view is just a view. +anonymous `model` line. A source names itself in quotes, since a source is the one +step with a name. A view is just a view. -The legend lists only what's on screen. A node feeding two branches is one node, so it -is drawn in full where you first meet it and marked `↑` after: `[3]` above feeds both -models but runs once — computed once and read back by each of them — and its inputs are -listed under its first appearance rather than repeated. On plans with a shared base that -is the difference between a readable tree and a few hundred lines. +The legend lists only what's on screen. A node feeding two branches is still one +node. It's drawn in full where you first meet it, and marked `↑` after. `[3]` above +feeds both models but runs once, computed once and read back by each of them. Its +inputs are listed only under its first appearance, not repeated. On plans with a +shared base, that's the difference between a readable tree and a few hundred lines. -`cached` is the one to watch: it is the plan telling you your edit didn't invalidate -that step, so nothing was recomputed. +`cached` is the one to watch. It tells you your edit didn't invalidate that step, so +nothing was recomputed. -Where nothing is drawn — a scheduler, CI, a redirected stream — the same tree is logged +Where nothing is drawn (a scheduler, CI, a redirected stream), the same tree is logged **once**, up front, with each step reporting beneath it: ``` @@ -324,16 +326,16 @@ INFO Collected 7 steps (7 ran, 0 cached) in 1.402s. Store 3.0 MB (+3.0 MB), 7 a ``` The summary also says what the store now costs, and what this run added to it. A store -keeps everything you collect into it, so editing a cleaning expression and re-collecting -leaves the old artifacts behind — the `(+3.0 MB)` is what tells you which edit did that, -while it is still a few megabytes rather than a full disk. A fully cached re-run reads -`(+0 B)`. See [Reclaiming storage](#reclaiming-storage). +keeps everything you collect into it, so editing a cleaning expression and +re-collecting leaves the old artifacts behind. The `(+3.0 MB)` is what tells you which +edit did that, while it's still a few megabytes rather than a full disk. A fully +cached re-run reads `(+0 B)`. See [Reclaiming storage](#reclaiming-storage). -Work done is `INFO`; skipping — cached is `DEBUG`, and the closing summary -totals it so an `INFO` reader still sees what the run avoided. Anything a step logs -while it runs is prefixed the same way, so a linker reporting its rounds lands under -the position it belongs to. Like any library logger it is silent until you configure -logging: +Work done is logged at `INFO`. Skipping a cached step is logged at `DEBUG`, and the +closing summary totals those so an `INFO` reader still sees what the run avoided. +Anything a step logs while it runs is prefixed the same way, so a linker reporting its +rounds lands under the position it belongs to. Like any library logger, it's silent +until you configure logging: ```python import logging @@ -341,17 +343,17 @@ import logging logging.basicConfig(level=logging.INFO) ``` -The plan is put in **one** place, never two: a drawn tree is already the key those -`[step N]` lines need — it is on screen throughout, and left there in full when the run -ends — so it isn't logged as well. The per-step records are the same either way. +The plan is put in **one** place, never two. A drawn tree is already the key those +`[step N]` lines need. It's on screen throughout, and left there in full when the run +ends, so it isn't logged as well. The per-step records are the same either way. That choice is `collect(interactive=...)`, named for the assumption it makes rather than the widget it produces. Drawing means *someone is watching*, so the plan can be a -thing on screen that the session throws away; `interactive=False` puts the tree in the -log instead, and is what a run whose output outlives the session wants. The default, -`interactive=None`, reads a terminal or a notebook as a yes and anything else as a no. +thing on screen that the session throws away. `interactive=False` puts the tree in the +log instead. That's what a run whose output outlives the session wants. The default, +`interactive=None`, reads a terminal or a notebook as a yes, and anything else as a no. -A plan taller than your window is windowed rather than dropped: the frame shows the +A plan taller than your window is windowed rather than dropped. The frame shows the rows around the running step and says how many are hidden either side, following the run down the tree. The last frame is the whole thing. @@ -369,10 +371,10 @@ run down the tree. The last frame is the whole thing. ``` That works alongside the live tree with nothing further to set up. `basicConfig` binds -whatever `sys.stderr` was at the time, which would otherwise write over the frame being -redrawn, so a running collection borrows handlers pointed at its terminal and routes -them through its console until it is finished. Records appear above the tree as they -arrive. +whatever `sys.stderr` was at the time. Left alone, that would write over the frame +being redrawn, so a running collection borrows handlers pointed at its terminal and +routes them through its console until it's finished. Records appear above the tree as +they arrive. ## Inspecting @@ -382,10 +384,10 @@ entities.lineage() # every step, inputs first ``` Both look *upstream* only. A source can't reach the resolver built on top of it, -because a step knows its inputs and nothing else — `crn.lineage()` is just `[crn]` +because a step knows its inputs and nothing else. `crn.lineage()` is just `[crn]`, however much is built above it. -There's no lookup-by-name: to hold on to a step, hold on to the variable. +There's no lookup-by-name. To hold on to a step, hold on to the variable. ```python cleaned = crn.view(cleaning={"name": f"lower({crn.f('company')})"}) @@ -394,24 +396,24 @@ entities = cleaned.dedupe(...).resolve().collect() cleaned.data() # still yours to inspect ``` -That goes for settings too: a resolver's per-model thresholds take the model itself, +That goes for settings too. A resolver's per-model thresholds take the model itself, not its name. -Steps have no names at all. To find a resolution later, **publish** it under a label — -an operation on the collected result, not a property of the plan: +Steps have no names at all. To find a resolution later, **publish** it under a +label. That's an operation on the collected result, not a property of the plan: ```python entities = crn_dedupe.resolve(dh_dedupe).collect().publish("entities") ``` -Republishing the same label for the same resolution is a no-op; aiming it at a -different one needs `overwrite=True`. A plan you never publish still runs — it is just +Republishing the same label for the same resolution is a no-op. Aiming it at a +different one needs `overwrite=True`. A plan you never publish still runs. It's just unlabelled. -A label is not a name. A *name* belongs to a source and is part of its output; a label -belongs to the store, and points at whichever resolution you last aimed it at. +A label is not a name. A *name* belongs to a source and is part of its output. A +label belongs to the store, and points at whichever resolution you last aimed it at. -Everything else goes by **position**: the order `collect` runs it in, which is what +Everything else goes by **position**. That's the order `collect` runs it in, which is what logs quote and what `draw()` shows in brackets. ``` @@ -432,16 +434,16 @@ print(entities.draw()) ``` Positions are relative to the apex you collected or drew from, so a plan and a -sub-plan of it number differently — but a run and that run's drawing always agree. +sub-plan of it number differently, but a run and that run's drawing always agree. ## Reclaiming storage **A store keeps everything you collect into it, until you delete the file.** matchlab -never removes an artifact on its own initiative. That is deliberate: an artifact's value -has nothing to do with whether your program still holds the variable that produced it, -and the next process to rebuild the same plan wants a cache hit rather than a rerun. +never removes an artifact on its own initiative. That's deliberate. An artifact's value +has nothing to do with whether your program still holds the variable that produced it. +The next process to rebuild the same plan wants a cache hit, not a rerun. -The cost of that is real, so every collect reports it — the `Store 3.0 MB (+3.0 MB), +The cost of that is real, so every collect reports it. That's the `Store 3.0 MB (+3.0 MB), 7 artifacts` clause above. You can also ask directly: ```python @@ -454,12 +456,12 @@ print(stats.artifacts) # {'source': 8, 'view': 40, 'model': 32, 'resolver': 24} print(stats.describe()) # 'Store 1.2 GB, 104 artifacts' ``` -Watch the artifact count rather than the size to see this happen: edit a cleaning +Watch the artifact count rather than the size to see this happen. Edit a cleaning expression, re-collect, and the count grows while the plan stays the same size. The old artifacts are still there, and nothing will remove them. -Each adapter reports what only it can measure, so a `DuckDBAdapter` hands back a -`DuckDBStoreStats` — with a `path` you can pass to `unlink()`, and a `free_bytes` for +Each adapter reports what only it can measure. A `DuckDBAdapter` hands back a +`DuckDBStoreStats`, with a `path` you can pass to `unlink()`, and a `free_bytes` for space already freed inside the file. ### Trimming @@ -474,7 +476,7 @@ print(result.describe()) # 'Removed 80 artifacts, kept 24, reclaimed 416.2 MB' ``` -`plan.fingerprints()` names every artifact a plan is made of — its own and its inputs'. +`plan.fingerprints()` names every artifact a plan is made of, its own and its inputs'. Which artifacts those are is the plan's business, not the store's, so the plan is what answers. `keep` also takes the name of a published label, which keeps that resolution and the sources it reads through: @@ -484,17 +486,17 @@ default_adapter().trim(keep=[*entities.fingerprints(), "production"]) ``` **Published labels are kept whether or not you list them**, because publishing is the -strongest way this library has of saying "keep this", and losing one to a forgotten +strongest way this library has of saying "keep this". Losing one to a forgotten argument would be indefensible. Trimming with nothing to keep and nothing published -raises rather than emptying the store. +raises, rather than emptying the store. Nothing is inferred about what you are still using. matchlab does not watch which -objects your program is holding and treat the rest as rubbish — a store outlives the +objects your program is holding and treat the rest as rubbish. A store outlives the process that wrote it, so what some interpreter happens to have in scope says nothing about what is worth keeping. You say what to keep. Trimming rewrites the store, which reopens its connection. Any session setting applied -through `adapter.conn` — see [Keeping memory bounded](#keeping-memory-bounded) — has to +through `adapter.conn` (see [Keeping memory bounded](#keeping-memory-bounded)) has to be applied again afterwards. ### Starting from cold @@ -507,15 +509,15 @@ from pathlib import Path Path("./run.duckdb").unlink() # start again from cold ``` -The default store lives in your user cache directory — -`default_adapter().stats().location` will tell you exactly where — and is safe to delete -at any time. You lose cache hits, not -results you can't rebuild, provided the warehouse data hasn't moved. +The default store lives in your user cache directory. +`default_adapter().stats().location` will tell you exactly where, and it's safe to +delete at any time. You lose cache hits, not results you can't rebuild, provided the +warehouse data hasn't moved. !!! warning "DuckDB files do not shrink" Deleting rows or dropping tables inside a DuckDB file does **not** return space to the operating system. DuckDB marks the blocks free and reuses them for later - writes, but the file stays the size of its high-water mark — there is no + writes, but the file stays the size of its high-water mark. There is no `VACUUM FULL`, and `CHECKPOINT` will not do it either: ``` @@ -525,18 +527,19 @@ results you can't rebuild, provided the warehouse data hasn't moved. This is why `trim()` **rewrites** the store rather than deleting inside it. Purging artifacts alone would buy reuse headroom while your disk usage stayed exactly the - same — on a real 575 MB store, deleting 77% of its artifacts freed nothing at all. + same. On a real 575 MB store, deleting 77% of its artifacts freed nothing at all. Copying what survives into a fresh file and swapping it in recovered 437 MB of that store, in half a second. It is the manual "collect what you want into a new store and delete the old one", done for you and without the re-collect. - A trim reports what it actually recovered, measured before and after — never what - it deleted. Those are different numbers, and only one of them is on your disk. + A trim reports what it actually recovered, measured before and after. It never + reports what it deleted. Those are different numbers, and only one of them is on + your disk. ### Keeping memory bounded An in-memory store (`DuckDBAdapter(":memory:")`) is not limited to RAM. DuckDB spills -table data to a temporary directory once it exceeds `memory_limit`, which defaults to +table data to a temporary directory once it exceeds `memory_limit`. That defaults to about 80% of your machine's memory: ```python @@ -545,8 +548,8 @@ adapter.conn.execute("SET memory_limit = '4GB'") adapter.conn.execute("SET temp_directory = '/fast/scratch'") ``` -That bounds the resident footprint without discarding anything, which is almost always -what you want from a cache: paged out is cheap to read back, deleted has to be +That bounds the resident footprint without discarding anything. That's almost always +what you want from a cache. Paged out is cheap to read back, deleted has to be recomputed. ## Next diff --git a/mkdocs.yml b/mkdocs.yml index db78d61e..b4752574 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -8,6 +8,7 @@ nav: - Welcome: index.md - Why matchlab: why-matchlab.md - Use cases: use-cases.md + - Glossary: glossary.md - Guide: - Install: guide/install.md - Build a plan: guide/build-a-plan.md diff --git a/src/matchlab/adapters/duckdb.py b/src/matchlab/adapters/duckdb.py index 50083ca5..7e5beaf8 100644 --- a/src/matchlab/adapters/duckdb.py +++ b/src/matchlab/adapters/duckdb.py @@ -1,9 +1,9 @@ """DuckDB storage adapter — the reference local backend for matchlab. A single DuckDB database (a file, or `:memory:`) holds every collected artifact, -keyed by step fingerprint. There is no resolution engine here: resolvers arrive -already materialised (merge-forward), and reads are plain table scans. Analysts can -point their own SQL at the `resolution` table — it is the whole point. +keyed by step fingerprint. There is no resolution engine here. Resolvers arrive +already materialised (merge-forward), so reads are plain table scans. Analysts can +point their own SQL at the `resolution` table. That is the whole point. """ import os @@ -25,32 +25,35 @@ ) from matchlab.eval.judgements import Judgement -#: Bumped whenever the stored shape changes, or whenever stored IDs stop meaning what -#: they did. A store written by an older matchlab is recreated rather than half-read, -# which is the honest failure for a cache. +#: Bumped whenever the stored shape changes, or stored IDs stop meaning what they did. +#: A store written by an older matchlab is recreated rather than half-read. That's the +#: honest failure mode for a cache. _SCHEMA_VERSION = 5 _SCHEMA_DDL = """ CREATE TABLE IF NOT EXISTS meta ( schema_version INTEGER ); --- `fp` is BLOB in every table below because a fingerprint is a 32-byte SHA-256 digest --- and DuckDB's widest integer holds 16. An integer key would mean truncating the hash, --- and it would buy nothing: every `fp` predicate here is scalar equality, a store holds --- few fingerprints against many rows so the column dictionary-compresses to a code per --- row, and the zonemaps prune whole row groups regardless (see `source_leaves` below). --- Measured against a UBIGINT key over 20M rows, the BLOB is no slower laid out that way --- and faster interleaved, where random 64-bit keys do not compress. Truncation is a bad --- trade for that: unlike a leaf ID collision, which shows up as a wrong merge in the --- data, a fingerprint collision makes `_ensure` skip the step and read back a different --- one's artifact, silently. +-- `fp` is BLOB in every table below. A fingerprint is a 32-byte SHA-256 digest, and +-- DuckDB's widest integer holds only 16 bytes, so an integer key would mean truncating +-- the hash. +-- +-- Truncating would buy nothing. Every `fp` predicate here is scalar equality. A store +-- holds few fingerprints against many rows, so the column dictionary-compresses to a +-- code per row. The zonemaps prune whole row groups regardless (see `source_leaves` +-- below). Measured against a UBIGINT key over 20M rows, the BLOB is no slower laid out +-- that way, and it is faster interleaved, where random 64-bit keys do not compress. +-- +-- Truncation is a bad trade too. A leaf ID collision shows up as a wrong merge in the +-- data. A fingerprint collision is worse: it is silent. It makes `_ensure` skip the +-- step and read back a different artifact. CREATE TABLE IF NOT EXISTS artifacts ( fp BLOB PRIMARY KEY, kind VARCHAR ); -- A label: a pointer from a string someone chose to the resolution they want to find --- again. Separate from `artifacts` because labelling is an act, not a property — most --- artifacts carry none, and a label can be moved to a newer fingerprint without --- disturbing the artifact it used to point at. +-- again. It is kept separate from `artifacts` because labelling is an act, not a +-- property. Most artifacts carry no label, and moving a label to a newer fingerprint +-- never disturbs the artifact it used to point at. CREATE TABLE IF NOT EXISTS labels ( label VARCHAR PRIMARY KEY, fp BLOB, published_at TIMESTAMP ); @@ -64,22 +67,24 @@ CREATE TABLE IF NOT EXISTS resolution_sources ( fp BLOB, source_name VARCHAR, source_fp BLOB ); --- The three tables below hold every artifact of their kind side by side, and are the --- only ones here that grow with the data rather than with the plan. Every read of them --- is `WHERE fp = ?` for one artifact, and none carries an index, because the write path --- already sorts them: each artifact arrives as exactly one INSERT, so its rows are --- appended as one contiguous run and no row group (~122k rows) straddles two artifacts --- of any size. DuckDB keeps min/max statistics per row group, so a scan for one --- fingerprint skips the others' groups without decompressing them — on a digest, whose --- leading bytes are already random, the 8-byte prefix those statistics truncate to --- discriminates as well as the whole value would. +-- The three tables below hold every artifact of their kind side by side. They are the +-- only ones here that grow with the data rather than with the plan. +-- +-- Every read of them is `WHERE fp = ?` for one artifact, and none carries an index. +-- That works because the write path already sorts them: each artifact arrives as +-- exactly one INSERT, so its rows are appended as one contiguous run, and no row group +-- (~122k rows) straddles two artifacts of any size. DuckDB keeps min/max statistics +-- per row group, so a scan for one fingerprint skips the others' groups without +-- decompressing them. On a digest, whose leading bytes are already random, the 8-byte +-- prefix those statistics truncate to discriminates as well as the whole value would. -- -- This is a property of how we write, not one the storage layer enforces. Writing an -- artifact in several statements, or interleaving two artifacts' writes, would scatter --- each across row groups whose min/max then span both fingerprints and prune nothing. --- Nothing else disturbs the layout: `_purge` marks rows deleted in place and a --- re-collect appends afresh, and `_rewrite` copies in scan order. Artifacts small enough --- to share a row group prune poorly, which costs what scanning them costs — nothing. +-- each across row groups whose min/max then span both fingerprints. That prunes +-- nothing. Nothing else disturbs the layout: `_purge` marks rows deleted in place, a +-- re-collect appends afresh, and `_rewrite` copies in scan order. Artifacts small +-- enough to share a row group prune poorly, which costs no more than scanning them +-- would anyway. CREATE TABLE IF NOT EXISTS source_leaves ( fp BLOB, key VARCHAR, leaf UBIGINT ); @@ -103,12 +108,12 @@ class DuckDBStoreStats(StoreStats): Attributes: path: The database file, or `None` for `:memory:`. `location` already names the - store; this is the file itself, for code that wants to `stat` or delete it. + store. This is the file itself, for code that wants to `stat` or delete it. free_bytes: Space already freed inside the file. DuckDB reuses those blocks for - later writes but never returns them to the OS, so this is the gap between - what the store weighs and what it holds — and the only figure that says - what a reclaim could recover without first deciding what to delete. It has - no meaning for a backend that is not a file of reusable blocks, which is + later writes, but never returns them to the OS. This is the gap between + what the store weighs and what it holds. It is the only figure that says + what a reclaim could recover, without first deciding what to delete. It has + no meaning for a backend that is not a file of reusable blocks. That is why it lives here rather than on `StoreStats`. """ @@ -119,9 +124,9 @@ class DuckDBStoreStats(StoreStats): def size(self) -> str: """Say when the bytes are resident rather than written. - `4.6 MB` reads as disk, and for `:memory:` it is not — the store vanishes with - the process. The distinction matters most in exactly the case a user is least - likely to be thinking about it. + `4.6 MB` reads as a disk size, but for `:memory:` it isn't one. The store + vanishes with the process. The distinction matters most exactly when a user is + least likely to be thinking about it. """ return super().size if self.path is not None else f"{super().size} in memory" @@ -133,9 +138,9 @@ def _mint_cluster_id(leaves: list[int]) -> int: (`process_judgements`) works without an explicit expansion row. A group uses `root_id`, the same function a resolver mints its roots with. That - is load-bearing rather than tidy: scoring compares a judged group against the - resolution's clusters by ID, so if the two ever disagree every comparison misses - and precision/recall is computed over an empty set. + match is load-bearing, not just tidy. Scoring compares a judged group against the + resolution's clusters by ID. If the two ever disagree, every comparison misses, + and precision and recall get computed over an empty set. """ if len(leaves) == 1: return int(leaves[0]) @@ -163,8 +168,9 @@ def _open_schema(self) -> None: if row and row[0] == _SCHEMA_VERSION: return - # Either a pre-versioned store or an older version: start clean. Artifacts are - # a cache — everything in here can be recomputed from the plan that made it. + # Either a pre-versioned store or an older version, so start clean. Artifacts + # are a cache, and everything in here can be recomputed from the plan that + # made it. for (table,) in self.conn.execute( "SELECT table_name FROM information_schema.tables " "WHERE table_schema = 'main'" @@ -205,9 +211,9 @@ def _purge(self, fp: Fingerprint) -> None: Only ever called immediately before storing that same fingerprint again. A fingerprint addresses content, so the replacement is the same data by - construction — which is why any **label** pointing at `fp` is left alone. The - label still resolves, to bytes indistinguishable from the ones it resolved to - before, and a publication is not something a re-collect should quietly revoke. + construction. That is why any **label** pointing at `fp` is left alone. It + still resolves, to bytes indistinguishable from the ones it resolved to before. + A re-collect should not quietly revoke a publication. """ kind = self._kind(fp) if kind is None: @@ -235,24 +241,24 @@ def has(self, fp: Fingerprint) -> bool: def stats(self) -> DuckDBStoreStats: """Report the store's size and contents. - **Checkpoints a file store before measuring it**, which makes this the one - method here that writes without being asked to. Without it the figure is not - merely imprecise, it is the wrong order of magnitude: recent writes sit in the + **Checkpoints a file store before measuring it.** That makes this the one + method here that writes without being asked to. Without it, the figure is not + just imprecise. It is the wrong order of magnitude. Recent writes sit in the write-ahead log as a compact journal, and settling them into 256 KB blocks can - turn 21 KB of log into 5.5 MB of file. Measured on `examples/companies` — 33 KB + turn 21 KB of log into 5.5 MB of file. Measured on `examples/companies`: 33 KB reported against a store that became 5.5 MB the moment it was closed. Reporting a size that a user's next `du` contradicts by 170x is worse than reporting none. - DuckDB's own block count is no help before that point: it reads zero until a + DuckDB's own block count is no help before that point. It reads zero until a checkpoint has happened. Afterwards the two agree to within the file header, so - `stat` is what is used — it counts the `.wal` sibling too, and it is the number + `stat` is used instead. It counts the `.wal` sibling too, and it is the number a user can actually check. - The checkpoint is cheap because DuckDB has usually already done most of it: + The checkpoint is cheap, because DuckDB has usually already done most of it: 1.4 ms after writing 10M rows, 0.08 ms when there is nothing pending. - `free_blocks` still comes from DuckDB, because nothing outside the file can see - how much of it is reusable. So does an in-memory store's size, which allocates + `free_blocks` still comes from DuckDB. Nothing outside the file can see how + much of it is reusable. So does an in-memory store's size, since it allocates no blocks and has no file to measure. """ if self.path == ":memory:": @@ -269,9 +275,9 @@ def stats(self) -> DuckDBStoreStats: "SELECT block_size, free_blocks FROM pragma_database_size()" ).fetchone() free = blocks[0] * blocks[1] if blocks else 0 - # Resolved because `self.path` is the constructor's argument verbatim: a - # store opened as "./run.duckdb" would otherwise report itself against a - # working directory that has since moved on. + # `self.path` is the constructor's argument verbatim, so it's resolved + # here. Otherwise a store opened as "./run.duckdb" would report itself + # against a working directory that has since moved on. path = Path(self.path).resolve() size = sum( candidate.stat().st_size @@ -359,8 +365,8 @@ def read_identifiers( if self._kind(resolver_fp) is not StepKind.RESOLVER: raise KeyError(f"No stored resolver for fingerprint {resolver_fp.hex()}") - # Both predicates in the query: `source` is what keeps this from scanning every - # source's rows, and `resolution` holds one generation per collect of the plan. + # Both predicates matter. `source` keeps this from scanning every source's + # rows, and `resolution` holds one generation per collect of the plan. return self.conn.execute( "SELECT root AS id, source, key, leaf " "FROM resolution WHERE fp = ? AND source = ?", @@ -372,10 +378,10 @@ def read_identifiers( def _keep_set(self, keep: Iterable[Fingerprint | str]) -> set[Fingerprint]: """Every fingerprint that must survive a trim. - Labels go in whether or not they were named, and a label drags in the sources - its resolution needs: reading a published resolution without a plan goes - through `resolution_sources` to each source's extract, so a label kept without - them resolves to a fingerprint whose data is gone. + Labels go in whether or not they were named. A label also drags in the sources + its resolution needs. Reading a published resolution without a plan goes + through `resolution_sources` to each source's extract. Without them, a kept + label resolves to a fingerprint whose data is gone. """ kept: set[Fingerprint] = set() for item in keep: @@ -399,22 +405,23 @@ def _label_closure(self, label: str) -> set[Fingerprint]: def trim(self, keep: Iterable[Fingerprint | str] = ()) -> TrimResult: """Delete every artifact except the ones named, and reclaim what that frees. - Deleting is only half of it. DuckDB marks freed blocks for reuse but never hands - them back to the OS, so purging alone moves the file size by nothing — measured - on a real 575 MB store, deleting 77% of its artifacts freed 0 bytes. The space - comes back only by rewriting the database: copy what is left into a fresh file - and swap it in. Purge and rewrite together recovered 437 MB of that store, in - half a second. + Deleting is only half of it. DuckDB marks freed blocks for reuse but never + hands them back to the OS, so purging alone does not shrink the file at all. + Measured on a real 575 MB store, deleting 77% of its artifacts freed 0 bytes. + The space comes back only by rewriting the database, copying what is left into a + fresh file and swapping it in. Purge and rewrite together recovered 437 MB of + that store, in half a second. - The swap is a rename over the original, so a failure anywhere leaves the store - exactly as it was and the half-written copy orphaned beside it. + The swap is a rename over the original. A failure anywhere leaves the store + exactly as it was, with the half-written copy orphaned beside it. - **This reopens the connection**, which is the one internal anything outside - reaches for: session settings applied through `adapter.conn` — `memory_limit` - and `temp_directory`, as the guide suggests — do not survive. The adapter itself - stays valid, so anything holding *it* rather than its connection is unaffected. + **This reopens the connection.** That is the one internal detail anything + outside this class needs to know about. Session settings applied through + `adapter.conn` (`memory_limit` and `temp_directory`, as the guide suggests) do + not survive. The adapter itself stays valid, so anything holding *it*, rather + than its connection, is unaffected. - An in-memory store is purged but not rewritten: it has no file, and reopening + An in-memory store is purged but not rewritten. It has no file, and reopening one would hand back an empty database rather than a smaller one. Its freed blocks return to the allocator anyway, so the reclaim is real regardless. """ @@ -584,11 +591,11 @@ def read_eval_data( ).pl() expansion = self.conn.execute("SELECT root, leaves FROM expansion").pl() - # Present empty results with the right columns/dtypes. We deliberately do NOT - # re-validate against the arrow transport schemas here: those pin `leaves` to a - # small `list`, whereas polars naturally emits `large_list` — a serialisation - # detail that is meaningless locally. Types are guaranteed by the table DDL and - # inputs are validated on write. + # Present empty results with the right columns/dtypes. We deliberately do not + # re-validate against the arrow transport schemas here. Those pin `leaves` to a + # small `list`, while polars naturally emits `large_list`. That's a + # serialisation detail that is meaningless locally. Types are guaranteed by the + # table DDL, and inputs are validated on write. if judgements.height == 0: judgements = pl.DataFrame(schema=pl.Schema(SCHEMA_JUDGEMENTS)) if expansion.height == 0: