Add cli

docs/_quarto.yml

@@ -58,6 +58,7 @@ website:
             - get-started/basic-docs.qmd
             - get-started/basic-content.qmd
             - get-started/basic-building.qmd
+            - get-started/qpyd-cli.qmd
             - get-started/crossrefs.qmd
             - get-started/sidebar.qmd
             - get-started/extending.qmd

docs/get-started/basic-building.qmd

@@ -9,6 +9,11 @@ jupyter:
 
 **tl;dr**: Once you've configured quartodoc in your `_quarto.yml` file, use the following commands to build and preview a documentation site.
 
+:::{.callout-tip}
+For a higher-level workflow that wraps these commands and also manages
+notebooks, previewing, and publishing, see [the qpyd CLI](qpyd-cli.qmd).
+:::
+
 ## `quartodoc build`: Create doc files
 
 Automatically generate `.qmd` files with reference api documentation.  This is written by default to the reference/ folder in your quarto project.

docs/get-started/overview.qmd

@@ -161,6 +161,8 @@ quartodoc:
 
 The functions listed in `contents` are assumed to be imported from the package.
 
+If no contents are provided, quartodoc will attempt to pull in all modules (i.e. `.py` files) from the package.
+
 
 ## Learning more
 

docs/get-started/qpyd-cli.qmd

@@ -0,0 +1,218 @@
+---
+title: The qpyd CLI
+---
+
+**tl;dr**: `qpyd` is a higher-level command-line workflow built on top of
+quartodoc. Where [`quartodoc build`](basic-building.qmd) generates your API
+reference pages, `qpyd` wraps the *whole* docs lifecycle — pre-render builds,
+rendering, previewing, publishing, scaffolding — and adds parallel notebook
+management.
+
+It installs two console scripts:
+
+| Command | Purpose |
+|---------|---------|
+| `qpyd`  | Build, render, preview, publish, and scaffold a docs site. |
+| `qpynb` | Run, check, convert, and clean notebooks (`.qmd` / `.ipynb`). Also available as `qpyd nb …`. |
+
+`qpyd` builds on [`sciris`](https://sciris.org), [`jupytext`](https://jupytext.readthedocs.io),
+and [`nbformat`](https://nbformat.readthedocs.io), and shells out to the
+external [`quarto`](https://quarto.org) binary, so make sure Quarto is installed
+and on your `PATH`.
+
+## Quick start
+
+Scaffold a docs folder, then render it:
+
+```bash
+# Create docs/ with a starter _quarto.yml, index.qmd, and _variables.py
+qpyd init docs --package your_package
+
+cd docs
+
+# Run pre-render steps, then `quarto render`
+qpyd render
+
+# Or preview with live reload
+qpyd preview
+```
+
+## `qpyd`: site commands
+
+### `qpyd prerender`
+
+Runs the pre-render build steps, in order:
+
+1. `quartodoc build` — generate the API reference pages.
+2. Customize aliases — add short cross-reference aliases (e.g. `pkg.Thing` for
+   `pkg.submodule.Thing`) to `objects.json`.
+3. `quartodoc interlinks` — build interlink inventories.
+4. Build a Sphinx-compatible `objects.inv` so other projects can resolve your
+   references via intersphinx.
+
+```bash
+qpyd prerender
+```
+
+The documented package name is read from the `quartodoc.package` key in your
+`_quarto.yml`. This is the command you typically wire into your project's
+`pre-render` hook (see [below](#configuring-_quartoyml)).
+
+### `qpyd render`
+
+Runs `qpyd prerender`, then `quarto render`, reporting the total build time.
+Any extra arguments are passed straight through to Quarto:
+
+```bash
+qpyd render                 # full build
+qpyd render --to html       # extra args forwarded to `quarto render`
+qpyd render --no-prerender  # skip the pre-render steps
+```
+
+### `qpyd preview`
+
+Like `qpyd render`, but launches `quarto preview` (a live-reloading server)
+after the pre-render steps:
+
+```bash
+qpyd preview
+qpyd preview --no-prerender
+```
+
+### `qpyd gh-publish`
+
+Renders with `--cache-refresh` and publishes to the `gh-pages` branch via
+`quarto publish`.
+
+```bash
+qpyd gh-publish
+```
+
+:::{.callout-warning}
+This pushes to a remote branch and updates your live site. It is never run as
+part of any other command.
+:::
+
+### `qpyd init`
+
+Scaffolds a docs folder with a starter `_quarto.yml`, `index.qmd`, and
+`_variables.py`. **Existing files are never overwritten**, so it is safe to run
+in an established project.
+
+```bash
+qpyd init docs --package your_package
+```
+
+### `qpyd clean`
+
+Deletes auto-generated scratch files after a build. This is **opt-in**: it only
+removes files matching the `qpyd.clean` glob patterns in your `_quarto.yml`, and
+it never deletes source files (`.qmd`, `.ipynb`, `.py`, `.md`).
+
+```bash
+qpyd clean            # delete configured scratch files
+qpyd clean --dry-run  # show what would be deleted
+```
+
+## `qpynb`: notebook commands
+
+A "notebook" is an `.ipynb` file, or a `.qmd` file containing a `{python}` code
+cell. Commands that take `paths` accept individual notebooks or folders; omit
+them to operate on the whole project.
+
+### `qpynb run` and `qpynb check`
+
+Both execute notebooks **in parallel**. The difference is what they do with the
+cache:
+
+```bash
+qpynb run            # execute via `quarto render`, updating the _freeze/ cache
+qpynb check          # execute to verify they run; touch no caches, leave no files
+qpynb run tutorials  # only the notebooks under tutorials/
+qpynb check --serial # one at a time (useful for debugging)
+```
+
+* **`run`** renders each notebook with `quarto render`, which executes it *and*
+  refreshes Quarto's freeze cache (`_freeze/`). Use it to pre-bake the cache in
+  parallel so a subsequent full-site render is fast.
+* **`check`** is a pure validation pass — it executes each notebook to confirm
+  it runs without error, but writes no cache and leaves nothing behind. This is
+  ideal for CI.
+
+During `run`, each per-notebook render sets `QPYD_SKIP_HOOKS=1`, so a project
+`pre-render: qpyd prerender` hook becomes a no-op rather than rebuilding the
+whole API reference (and racing on `objects.json`) once per notebook.
+
+### `qpynb refresh`
+
+Deletes the cached copies of notebooks — Quarto's freeze cache (`_freeze/`) and
+every nested jupyter cache (`.jupyter_cache/`) — so they re-execute on the next
+render.
+
+```bash
+qpynb refresh
+qpynb refresh --dry-run
+```
+
+### `qpynb to-py` / `to-qmd` / `to-ipynb`
+
+Convert a notebook between formats. `.qmd` ⇄ `.ipynb` conversions go through
+`quarto convert`; anything involving `.py` uses `jupytext`. The destination is
+not overwritten unless you pass `--force`.
+
+```bash
+qpynb to-py tutorials/intro.qmd      # -> tutorials/intro.py
+qpynb to-ipynb tutorials/intro.qmd   # -> tutorials/intro.ipynb
+qpynb to-qmd notebook.ipynb --force  # overwrite an existing notebook.qmd
+```
+
+### `qpynb clear`
+
+Strips saved outputs and execution counts from `.ipynb` notebooks and
+normalizes them. Files that are already clean are left untouched.
+
+```bash
+qpynb clear
+qpynb clear --dry-run
+```
+
+## `_variables.py`
+
+You can place an optional `_variables.py` file alongside `_quarto.yml`. Its
+public, non-callable, YAML-serializable values are passed to `quarto render` as
+`-M key:value` metadata by `qpyd render` and `qpyd preview`:
+
+```python
+# _variables.py
+import your_package
+version = your_package.__version__
+versiondate = your_package.__versiondate__
+```
+
+Values are YAML-encoded, so version strings survive intact (e.g. `"1.10"` is
+passed as the string `'1.10'`, not coerced to the number `1.1`). Reference them
+in documents with the [`meta` shortcode](https://quarto.org/docs/authoring/variables.html#meta):
+
+```markdown
+Docs for version {{{< meta version >}}} ({{{< meta versiondate >}}}).
+```
+
+## Configuring `_quarto.yml`
+
+The keys `qpyd` reads or writes:
+
+```yaml
+project:
+  pre-render: qpyd prerender   # build API docs etc. before each render
+  # post-render: qpyd clean    # optional; opt-in scratch cleanup
+
+quartodoc:
+  package: your_package        # used by prerender for aliases / objects.inv
+
+qpyd:
+  clean:                       # optional glob patterns for `qpyd clean`
+    - '**/my-*.png'            # source files are never deleted, even if matched
+
+execute:
+  freeze: auto                 # let `qpynb run` pre-bake the freeze cache
+```

pyproject.toml

@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
 [tool.setuptools_scm]
 
 [tool.setuptools.packages.find]
-include = ["quartodoc"]
+include = ["quartodoc", "quartodoc.*"]
 
 [tool.pytest.ini_options]
 markers = []
@@ -43,7 +43,10 @@ dependencies = [
     "requests",
     "typing-extensions >= 4.4.0",
     "watchdog >= 3.0.0",
-    "plum-dispatch > 2.0.0"
+    "plum-dispatch > 2.0.0",
+    "sciris >= 3.2.0",
+    "jupytext >= 1.16.0",
+    "nbformat >= 5.0.0"
 ]
 
 [project.urls]
@@ -54,6 +57,8 @@ ci = "https://github.com/machow/quartodoc/actions"
 
 [project.scripts]
 quartodoc = "quartodoc.__main__:cli"
+qpyd = "quartodoc.qpyd:cli"
+qpynb = "quartodoc.qpyd:nb_cli"
 
 [dependency-groups]
 dev = [

quartodoc/_pydantic_compat.py

@@ -5,6 +5,7 @@
         Extra,
         PrivateAttr,
         ValidationError,
+        validator,
     )  # noqa
 except ImportError:
-    from pydantic import BaseModel, Field, Extra, PrivateAttr, ValidationError  # noqa
+    from pydantic import BaseModel, Field, Extra, PrivateAttr, ValidationError, validator  # noqa

quartodoc/builder/blueprint.py

@@ -1,9 +1,14 @@
 from __future__ import annotations
 
+import importlib.util
 import logging
 import json
 import yaml
 
+from collections import OrderedDict
+from pathlib import Path
+from typing import Iterable, List, Optional, Set
+
 from .._griffe_compat import dataclasses as dc
 from .._griffe_compat import (
     GriffeLoader,
@@ -44,6 +49,47 @@
     from quartodoc._pydantic_compat import BaseModel
 
 
+def _identify_files_to_document(
+    path: Path,
+    file_patterns: List[str],
+    ignore: Optional[Iterable[str]] = None,
+) -> Set[Path]:
+    reversed_patterns = file_patterns.copy()
+    reversed_patterns.reverse()
+
+    files_to_document: dict = OrderedDict()
+    for pattern in reversed_patterns:
+        for file in path.rglob(pattern=pattern):
+            files_to_document[file.with_suffix("")] = file
+    result = set(files_to_document.values())
+
+    if ignore:
+        for pattern in ignore:
+            result = result.difference(set(path.glob(pattern=pattern)))
+
+    return {p.resolve() for p in result}
+
+
+def _auto_contents_from_package(package_name: str) -> list[Auto]:
+    """Return Auto entries for every .py file in *package_name*, excluding __init__ files."""
+    spec = importlib.util.find_spec(package_name)
+    if spec is None or not spec.submodule_search_locations:
+        return []
+
+    pkg_path = Path(list(spec.submodule_search_locations)[0])
+    files = _identify_files_to_document(
+        pkg_path,
+        file_patterns=["*.py"],
+        ignore=["**/__init__.py", "**/__pycache__/**"],
+    )
+
+    contents = []
+    for file in sorted(files):
+        parts = list(file.relative_to(pkg_path).with_suffix("").parts)
+        contents.append(Auto(name=".".join(parts)))
+    return contents
+
+
 def _auto_package(mod: dc.Module) -> list[Section]:
     """Create default sections for the given package."""
 
@@ -246,6 +292,34 @@ def enter(self, el: Layout):
 
         return super().enter(el)
 
+    @dispatch
+    def enter(self, el: Section):
+        if el.contents:
+            return el
+
+        package = self.crnt_package
+        label = el.title or el.subtitle or "(untitled)"
+
+        if not package:
+            _log.warning(
+                f"Section '{label}' has no contents and no package is configured."
+                " Cannot auto-populate contents."
+            )
+            return el
+
+        _log.warning(
+            f"Section '{label}' has no contents. Auto-populating from package '{package}'."
+        )
+
+        contents = _auto_contents_from_package(package)
+        if not contents:
+            _log.warning(f"No Python files found in package '{package}'.")
+            return el
+
+        new = el.copy()
+        new.contents = contents
+        return super().enter(new)
+
     @dispatch
     def exit(self, el: Section):
         """Transform top-level sections, so their contents are all Pages."""