Move Chrome extension into its product directory

AGENTS.md

@@ -29,10 +29,12 @@ These instructions apply to human contributors and automated coding agents.
 
 ## Project map
 
-- Chrome extension: root-level `manifest.json`, HTML, CSS, and JavaScript files.
-- Shared capture behavior and tests: `capture-utils.js` and `tests/`.
+- Chrome extension: `chrome-extension/sourcebraid/`.
+- Shared capture behavior and tests:
+  `chrome-extension/sourcebraid/capture-utils.js` and `tests/`.
 - PDF conversion: `scripts/`, `requirements-docling.txt`, and
-  `.github/workflows/convert-pdfs.yml`.
+  `.github/workflows/convert-pdfs.yml`; the Chrome extension contains verified
+  bundled copies of these support files for unpacked development and releases.
 - ChatGPT/Codex plugin: `codex-plugin/sourcebraid/`.
 - iOS app and Share Extension: `ios/`.
 - Public documentation and community policy: root Markdown files and `.github/`.

CONTRIBUTING.md

@@ -66,9 +66,10 @@ node --test tests/capture-utils.test.js
 python3 -m unittest discover -s tests -p "test_*.py"
 ```
 
-The Chrome extension uses Manifest V3 and has no build step. Load the repository
-as an unpacked extension for manual browser testing. The iOS app and Share
-Extension require Xcode; see [ios/README.md](ios/README.md) for build commands.
+The Chrome extension uses Manifest V3 and has no build step. Load
+`chrome-extension/sourcebraid` as an unpacked extension for manual browser
+testing. The iOS app and Share Extension require Xcode; see
+[ios/README.md](ios/README.md) for build commands.
 
 The website is maintained separately from the public release. Maintainers with
 access to its private source can validate it with:

README.de.md

@@ -159,7 +159,7 @@ Bei einem Gist wird eine einzelne Markdown-Datei direkt als Dokumentinhalt gespe
 1. `chrome://extensions` öffnen.
 2. **Entwicklermodus** aktivieren.
 3. **Entpackte Erweiterung laden** auswählen.
-4. Diesen Ordner auswählen.
+4. [`chrome-extension/sourcebraid`](chrome-extension/sourcebraid) auswählen.
 5. Eine unterstützte Quelle öffnen und auf das **SourceBraid**-Symbol klicken.
 6. GitHub-Repository konfigurieren, optional Tags oder Notizen ergänzen und **Save to GitHub** wählen.
 
@@ -248,4 +248,10 @@ Die native iOS-App und Share Extension liegen unter [`ios/`](ios/README.md). Nac
 
 ## Technische Hinweise
 
-Die Chrome-Erweiterung benötigt keinen Build-Schritt und bündelt keine Drittanbieter-Runtime. Docling läuft ausschließlich in der GitHub Action des Ziel-Repositorys. Die HTML-Konvertierung erfolgt lokal in der Erweiterung; API- und Bildzugriffe nutzen je nach Quelle entweder normale HTTP-Anfragen oder die vorhandene angemeldete Browser-Sitzung.
+Der Quellcode der Chrome-Erweiterung liegt unter
+[`chrome-extension/sourcebraid`](chrome-extension/sourcebraid). Die Erweiterung
+benötigt keinen Build-Schritt und bündelt keine Drittanbieter-Runtime. Docling
+läuft ausschließlich in der GitHub Action des Ziel-Repositorys. Die
+HTML-Konvertierung erfolgt lokal in der Erweiterung; API- und Bildzugriffe
+nutzen je nach Quelle entweder normale HTTP-Anfragen oder die vorhandene
+angemeldete Browser-Sitzung.

README.md

@@ -220,7 +220,7 @@ be loaded through the still-open Gist tab.
 1. Open `chrome://extensions`.
 2. Enable **Developer mode**.
 3. Select **Load unpacked**.
-4. Choose this repository folder.
+4. Choose [`chrome-extension/sourcebraid`](chrome-extension/sourcebraid).
 5. Open a supported source and select the **SourceBraid** icon.
 6. Configure the private GitHub repository, optionally add tags or notes, and
    choose **Save to GitHub**.
@@ -331,7 +331,9 @@ the local, user-controlled data flow and licensing boundaries.
 
 ## Technical notes
 
-The Chrome extension has no build step and bundles no third-party runtime.
-Docling runs only inside the target repository's GitHub Action. HTML conversion
-happens locally in the extension; API and image requests use either ordinary
-HTTP or the browser's existing authenticated session, depending on the source.
+The Chrome extension source lives under
+[`chrome-extension/sourcebraid`](chrome-extension/sourcebraid). It has no build
+step and bundles no third-party runtime. Docling runs only inside the target
+repository's GitHub Action. HTML conversion happens locally in the extension;
+API and image requests use either ordinary HTTP or the browser's existing
+authenticated session, depending on the source.

RELEASING.md

@@ -26,7 +26,7 @@ customer-facing version unless the maintainers document a different decision.
 Before creating the first release candidate, align these customer-facing
 versions:
 
-- Chrome: `version` in `manifest.json`.
+- Chrome: `version` in `chrome-extension/sourcebraid/manifest.json`.
 - ChatGPT/Codex skills package: `version` in
   `codex-plugin/sourcebraid/.codex-plugin/plugin.json`.
 - iOS app and Share Extension: `MARKETING_VERSION` in
@@ -71,7 +71,8 @@ changes.
 
 ## Chrome package
 
-Build the Manifest V3 archive from its explicit allowlist:
+The unpacked extension source lives in `chrome-extension/sourcebraid`. Build the
+Manifest V3 archive from its explicit allowlist:
 
 ```bash
 python3 scripts/build_chrome_package.py

chrome-extension/sourcebraid/.github/workflows/convert-pdfs.yml

@@ -0,0 +1,62 @@
+# Managed by SourceBraid PDF support.
+name: Convert PDFs to Markdown
+
+on:
+  push:
+    paths:
+      - "**/assets/*/source.pdf"
+  workflow_dispatch:
+
+permissions:
+  contents: write
+
+concurrency:
+  group: pdf-conversion-${{ github.ref }}
+  cancel-in-progress: false
+
+jobs:
+  convert:
+    runs-on: ubuntu-latest
+    timeout-minutes: 45
+
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+          cache-dependency-path: requirements-docling.txt
+
+      - name: Install Docling
+        run: python -m pip install --requirement requirements-docling.txt
+
+      - name: Convert queued PDFs
+        if: github.event_name == 'push'
+        run: >-
+          python scripts/convert_pdfs.py
+          --before "${{ github.event.before }}"
+          --after "${{ github.sha }}"
+
+      - name: Convert all queued PDFs
+        if: github.event_name == 'workflow_dispatch'
+        run: python scripts/convert_pdfs.py --all
+
+      - name: Commit converted Markdown
+        run: |
+          git add --all
+          if git diff --cached --quiet; then
+            echo "No converted files changed."
+            exit 0
+          fi
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          git commit -m "Convert queued PDFs to Markdown"
+          python scripts/push_with_retry.py \
+            --branch "${GITHUB_REF_NAME}" \
+            --attempts 5 \
+            --delay-seconds 2

chrome-extension/sourcebraid/requirements-docling.txt

@@ -0,0 +1,2 @@
+# Managed by SourceBraid PDF support.
+docling>=2.0,<3.0

chrome-extension/sourcebraid/scripts/convert_pdfs.py

@@ -0,0 +1,231 @@
+#!/usr/bin/env python3
+# Managed by SourceBraid PDF support.
+"""Convert queued web-clip PDFs with Docling and update their Markdown/index."""
+
+from __future__ import annotations
+
+import argparse
+import importlib.metadata
+import json
+import re
+import shutil
+import subprocess
+from datetime import datetime, timezone
+from pathlib import Path
+
+
+ZERO_SHA = "0" * 40
+NOTES_PATTERN = re.compile(
+    r"<!-- clipper-notes-start -->.*?<!-- clipper-notes-end -->\s*",
+    re.DOTALL,
+)
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--before", default="")
+    parser.add_argument("--after", default="HEAD")
+    parser.add_argument("--all", action="store_true")
+    args = parser.parse_args()
+
+    sources = discover_sources(args.before, args.after, args.all)
+    if not sources:
+        print("No queued source.pdf files found.")
+        return 0
+
+    for source in sources:
+        convert_source(source)
+    return 0
+
+
+def discover_sources(before: str, after: str, process_all: bool) -> list[Path]:
+    if process_all or not before or before == ZERO_SHA:
+        candidates = Path(".").glob("**/assets/*/source.pdf")
+    else:
+        result = subprocess.run(
+            ["git", "diff", "--name-only", "--diff-filter=AM", before, after, "--", "**/assets/*/source.pdf"],
+            check=True,
+            capture_output=True,
+            text=True,
+        )
+        candidates = (Path(line) for line in result.stdout.splitlines())
+
+    return sorted(
+        path for path in candidates
+        if path.is_file() and path.name == "source.pdf" and path.parent.parent.name == "assets"
+    )
+
+
+def convert_source(source: Path) -> None:
+    from docling.datamodel.base_models import InputFormat
+    from docling.datamodel.pipeline_options import PdfPipelineOptions
+    from docling.document_converter import DocumentConverter, PdfFormatOption
+    from docling_core.types.doc import ImageRefMode
+
+    source = source.resolve()
+    clip_slug = source.parent.name
+    target = source.parent.parent.parent / f"{clip_slug}.md"
+    pending = target.read_text(encoding="utf-8") if target.exists() else ""
+    frontmatter = extract_frontmatter(pending)
+    notes = extract_notes(pending)
+
+    pipeline_options = PdfPipelineOptions()
+    pipeline_options.images_scale = 1.5
+    pipeline_options.generate_picture_images = True
+    pipeline_options.do_ocr = True
+    pipeline_options.do_table_structure = True
+    converter = DocumentConverter(
+        format_options={
+            InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options)
+        }
+    )
+    result = converter.convert(source)
+
+    export_path = source.parent / "docling-output.md"
+    artifact_path = source.parent / "docling-output_artifacts"
+    if artifact_path.exists():
+        shutil.rmtree(artifact_path)
+    result.document.save_as_markdown(export_path, image_mode=ImageRefMode.REFERENCED)
+    body = export_path.read_text(encoding="utf-8").strip()
+    export_path.unlink(missing_ok=True)
+
+    artifact_prefix = source.parent.relative_to(target.parent).as_posix()
+    body = rewrite_artifact_links(body, artifact_prefix)
+
+    converted_at = datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
+    pages = len(result.document.pages)
+    version = importlib.metadata.version("docling")
+    frontmatter = update_frontmatter(
+        frontmatter,
+        {
+            "capture_method": json.dumps("pdf-docling"),
+            "conversion_status": json.dumps("complete"),
+            "converter": json.dumps("docling"),
+            "converter_version": json.dumps(version),
+            "pages": str(pages),
+            "ocr_enabled": "true",
+            "converted_at": json.dumps(converted_at),
+        },
+    )
+
+    title = frontmatter_value(frontmatter, "title") or clip_slug
+    if not re.match(r"^#\s+", body):
+        body = f"# {title}\n\n{body}"
+    sections = [frontmatter.strip(), notes.strip(), body]
+    target.write_text("\n\n".join(section for section in sections if section).rstrip() + "\n", encoding="utf-8")
+
+    images = sorted(
+        path.relative_to(Path.cwd()).as_posix()
+        for path in source.parent.glob("docling-output_artifacts/**/*")
+        if path.is_file()
+    )
+    update_index(target, source, images, pages, version, converted_at)
+    print(f"Converted {source.relative_to(Path.cwd())} -> {target.relative_to(Path.cwd())}")
+
+
+def extract_frontmatter(markdown: str) -> str:
+    match = re.match(r"\A---\s*\n.*?\n---\s*", markdown, re.DOTALL)
+    return match.group(0).strip() if match else "---\n---"
+
+
+def extract_notes(markdown: str) -> str:
+    match = NOTES_PATTERN.search(markdown)
+    return match.group(0).strip() if match else ""
+
+
+def rewrite_artifact_links(markdown: str, artifact_prefix: str) -> str:
+    """Make Docling image links relative, regardless of its exported path style."""
+    replacement = f"{artifact_prefix.rstrip('/')}/docling-output_artifacts/"
+    return re.sub(
+        r"(?<=\]\()[^)\n]*docling-output_artifacts/",
+        lambda _match: replacement,
+        markdown,
+    )
+
+
+def update_frontmatter(frontmatter: str, updates: dict[str, str]) -> str:
+    lines = frontmatter.splitlines()
+    if not lines or lines[0].strip() != "---":
+        lines = ["---", "---"]
+    if lines[-1].strip() != "---":
+        lines.append("---")
+
+    seen: set[str] = set()
+    output = [lines[0]]
+    for line in lines[1:-1]:
+        match = re.match(r"^([A-Za-z0-9_-]+):", line)
+        key = match.group(1) if match else ""
+        if key in updates:
+            output.append(f"{key}: {updates[key]}")
+            seen.add(key)
+        else:
+            output.append(line)
+    for key, value in updates.items():
+        if key not in seen:
+            output.append(f"{key}: {value}")
+    output.append("---")
+    return "\n".join(output)
+
+
+def frontmatter_value(frontmatter: str, key: str) -> str:
+    match = re.search(rf"^{re.escape(key)}:\s*(.+)$", frontmatter, re.MULTILINE)
+    if not match:
+        return ""
+    value = match.group(1).strip()
+    try:
+        decoded = json.loads(value)
+        return str(decoded)
+    except json.JSONDecodeError:
+        return value.strip('"\'')
+
+
+def update_index(
+    target: Path,
+    source: Path,
+    images: list[str],
+    pages: int,
+    version: str,
+    converted_at: str,
+) -> None:
+    root = target.parents[2]
+    index_paths = [root / "index.jsonl"]
+    shard_root = root / "index"
+    if shard_root.exists():
+        index_paths.extend(sorted(shard_root.glob("*.jsonl")))
+
+    repository_root = Path.cwd().resolve()
+    target_repo_path = target.resolve().relative_to(repository_root).as_posix()
+    source_repo_path = source.resolve().relative_to(repository_root).as_posix()
+    for index_path in index_paths:
+        if not index_path.exists():
+            continue
+        output: list[str] = []
+        changed = False
+        for line in index_path.read_text(encoding="utf-8").splitlines():
+            try:
+                entry = json.loads(line)
+            except json.JSONDecodeError:
+                output.append(line)
+                continue
+            if entry.get("path") == target_repo_path or entry.get("pdf_path") == source_repo_path:
+                entry.update(
+                    {
+                        "capture_method": "pdf-docling",
+                        "conversion_status": "complete",
+                        "converter": "docling",
+                        "converter_version": version,
+                        "pages": pages,
+                        "ocr_enabled": True,
+                        "converted_at": converted_at,
+                        "images": [{"path": image} for image in images],
+                    }
+                )
+                changed = True
+            output.append(json.dumps(entry, ensure_ascii=False, separators=(",", ":")))
+
+        if changed:
+            index_path.write_text("\n".join(output).rstrip() + "\n", encoding="utf-8")
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())