diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index cb86b89..8a05671 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -14,8 +14,6 @@ jobs: - uses: actions/checkout@v7 - uses: pnpm/action-setup@v6 - with: - version: 10 - uses: actions/setup-node@v7 with: @@ -23,5 +21,7 @@ jobs: cache: pnpm - run: pnpm install --frozen-lockfile - - run: pnpm build + - run: pnpm format:check - run: pnpm test + - run: pnpm test:package + - run: pnpm smoke:generated diff --git a/.gitignore b/.gitignore index de2ebbb..9700791 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,7 @@ # dependencies /node_modules +/.pnpm-store /.pnp .pnp.js .yarn/install-state.gz diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7306f5a..74b9aba 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -14,6 +14,9 @@ pnpm install pnpm dev # Watch mode (recompiles on changes) pnpm build # One-time compile pnpm test # Run tests +pnpm format:check # Verify formatting +pnpm test:package # Verify the npm package contents +pnpm smoke:generated # Generate, lint, type-check, and build a fixture site ``` To test your changes locally: @@ -26,13 +29,13 @@ node /path/to/cli/dist/index.js watch ## Project Structure -All CLI logic lives in `src/index.ts`. Template files under `src/templates/` are string constants that get written into the generated Next.js app. +The CLI entry point and generator orchestration live in `src/index.ts`. Reusable generator logic is under `src/lib/`, while template files under `src/templates/` are string constants written into the generated Next.js app. The output layout is registered centrally in `src/lib/structures.ts`. When adding a new template: 1. Create the template file in the appropriate `src/templates/` subdirectory 2. Export a named constant with the `Template` suffix (e.g., `export const myComponentTemplate = ...`) -3. Import it in `src/index.ts` and add it to the `structure` object in `createNextJSStructure()` +3. Import it in `src/lib/structures.ts` and add it to the `structure` object ## Code Conventions @@ -43,7 +46,7 @@ When adding a new template: ## Pull Requests 1. Fork the repo and create a branch from `main` -2. Make your changes and ensure `pnpm build && pnpm test` passes +2. Make your changes and ensure `pnpm build && pnpm test && pnpm format:check && pnpm test:package` passes 3. Write a clear PR description explaining the change and why 4. Keep PRs focused - one feature or fix per PR diff --git a/README.md b/README.md index f4b6bcf..558a2e9 100644 --- a/README.md +++ b/README.md @@ -35,11 +35,14 @@ Doccupine will prompt you for: It then scaffolds the app, installs dependencies, and starts the dev server. Open http://localhost:3000 to view your docs. +The source and output directories must not overlap. To prevent accidental data loss, Doccupine only claims an empty output directory, one containing harmless local metadata such as `.DS_Store` or `.env.local`, or an existing Doccupine-generated app. Run `doccupine config --reset` if an older configuration no longer passes validation. + ## CLI Commands ```bash doccupine watch [options] # Default. Watch MDX files and start dev server -doccupine build [options] # One-time build without starting the server +doccupine build [options] # Generate the site without installing or serving it +doccupine generate [options] # Alias for build doccupine config --show # Show current configuration doccupine config --reset # Re-prompt for configuration ``` @@ -53,6 +56,7 @@ doccupine config --reset # Re-prompt for configuration | `--port ` | Port for the dev server (default: `3000`). Auto-increments if taken. | | `--verbose` | Show all Next.js output including compilation details | | `--reset` | Re-prompt for watch/output directories | +| `--skip-install` | Generate and serve without running the dependency installer (installs are skipped automatically when `package.json` is unchanged) | | `--package-manager ` | Package manager for the generated app: `pnpm` or `npm` (default: auto-detect). Overrides the `packageManager` field in `doccupine.json`. | `build`: @@ -149,7 +153,7 @@ Each entry has: ## API Reference -Point Doccupine at an OpenAPI document (`.json`, `.yaml`, or `.yml`, OpenAPI 3.0/3.1) and it generates an interactive API reference: one page per operation, each with a live playground for sending requests. Set the `openapi` field in `doccupine.json`: +Point Doccupine at an OpenAPI document (`.json`, `.yaml`, or `.yml`, OpenAPI 3.0/3.1) and it generates an interactive API reference: a directory at `/api-reference` linking every operation, plus one page per operation with a live playground for sending requests. Set the `openapi` field in `doccupine.json`: ```json { @@ -209,7 +213,7 @@ Doccupine generates `robots.ts` automatically for every site. When you set a `ur } ``` -You can override the URL at deploy time by setting the `NEXT_PUBLIC_SITE_URL` environment variable. When no URL is configured (neither in `config.json` nor via env), the sitemap is skipped and `robots.txt` is emitted without a sitemap reference. +You can override the URL at deploy time by setting the `NEXT_PUBLIC_SITE_URL` environment variable. When no URL is configured, `/sitemap.xml` is still served but stays empty, and `robots.txt` omits its sitemap reference until a public URL is available. The variable is baked in at build time, so changing it needs a redeploy. ## llms.txt @@ -223,7 +227,7 @@ Doccupine generates [llms.txt](https://llmstxt.org) artifacts so AI agents can d The site name and description used in `llms.txt` come from `config.json` (`name`, `description`). Page URLs are absolute when `url` is set in `config.json` (or via `NEXT_PUBLIC_SITE_URL`), and root-relative otherwise. -A `.doccupine-llms-manifest.json` file in the generated app tracks which per-page mirrors were emitted so renamed or deleted pages get cleaned up on the next regeneration. Don't commit this file ā€” it's regenerated automatically. +A `.doccupine-artifacts.json` file in the generated app tracks generated route ownership and per-page mirrors so renamed or deleted sources clean up only their own outputs. Don't commit this file; it is regenerated automatically. ## AI Chat Setup @@ -250,10 +254,13 @@ LLM_EMBEDDING_MODEL=text-embedding-3-small # Override the default embedding mod LLM_TEMPERATURE=0 # Set temperature (0-1, default: 0) LLM_EMBEDDING_DIMS=512 # Dimensions for the prebuilt search index (default: 512) RAG_RUNTIME_EMBED_MAX_CHUNKS=400 # Max chunks embedded on demand in production (default: 400; 0 requires a prebuilt index) +# RAG_API_KEY=... # Optional bearer auth for direct /api/rag requests ``` `LLM_EMBEDDING_DIMS` Matryoshka-truncates document vectors so the prebuilt search index stays small; lower values shrink the index at a slight cost to recall. `RAG_RUNTIME_EMBED_MAX_CHUNKS` caps how many chunks the chat will embed on demand in production before requiring a prebuilt index (it's unlimited under `next dev`). +Public documentation leaves browser chat available by default. Setting `RAG_API_KEY` requires a bearer token for `/api/rag` and is intended for server-to-server use; the built-in browser cannot safely hold that secret. Use `SITE_PASSWORD` instead when authenticated browser visitors should retain chat access. + Default models per provider: | Provider | Chat model | Embedding model | @@ -270,7 +277,7 @@ The generated app exposes an MCP endpoint at `/api/mcp` with three tools: - `get_doc` - retrieve a specific document by path - `list_docs` - list all available documents -This lets AI agents (Claude, ChatGPT, etc.) query your docs programmatically. Requires the AI chat setup above for embeddings. +This lets AI agents (Claude, ChatGPT, etc.) query your docs programmatically. Semantic `search_docs` requires the AI setup above for embeddings; `get_doc` and `list_docs` work from the generated content manifest without an embedding provider. ## Password Protection @@ -283,12 +290,12 @@ SITE_PASSWORD=choose-a-strong-shared-password When set, every visitor sees a login screen until they enter the password. Protection is enforced across three layers: - **Pages** are gated behind the login screen. -- **Content APIs** (`/api/rag` chat and `/api/search`) return `401` without a valid session, so the docs can't be scraped around the login. +- **Content APIs** (`/api/rag` chat, `/api/search`, and the `/api/playground` proxy) return `401` without a valid session, so the docs can't be scraped and the proxy can't relay anonymous requests around the login. Each route re-checks the session itself, not just the middleware. - **Search engines and crawlers** are blocked: `robots.txt` disallows everything, pages carry a `noindex, nofollow` tag, and responses include an `X-Robots-Tag` header. A successful login sets a signed, `httpOnly` cookie that lasts 30 days. The cookie stores an HMAC of the password, never the password itself. Leave `SITE_PASSWORD` unset (the default) to keep the site fully public. Documentation pages stay statically rendered either way - the gate is enforced in middleware. -> **Note:** The [MCP endpoint](#mcp-server) uses its own `DOCS_API_KEY` bearer token and is not affected by `SITE_PASSWORD`. +> **Note:** The [MCP endpoint](#mcp-server) uses `DOCS_API_KEY` bearer authentication when configured. Without an API key, a password-protected site requires the normal gate session for MCP as well. ## License diff --git a/SECURITY.md b/SECURITY.md index 137e66f..20e4c7d 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -41,8 +41,9 @@ Please include: Doccupine is a CLI that generates a Next.js application you run and host yourself. A few points matter when assessing the security surface: -- **API keys and secrets** - the generated app reads provider keys (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY`), an optional `DOCS_API_KEY`, and an optional `SITE_PASSWORD` from environment variables. These belong in the generated app's `.env` file, which is git-ignored by default. Never commit real keys. -- **MCP endpoint** - the `/api/mcp` route requires a bearer token only when `DOCS_API_KEY` is set. If `DOCS_API_KEY` is not set, the endpoint is publicly accessible with no authentication. Set `DOCS_API_KEY` before exposing the generated site publicly. +- **API keys and secrets** - the generated app reads provider keys (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_API_KEY`), optional endpoint keys (`RAG_API_KEY`, `DOCS_API_KEY`), and an optional `SITE_PASSWORD` from environment variables. These belong in the generated app's `.env` file, which is git-ignored by default. Never commit real keys. +- **Paid RAG endpoint** - public documentation leaves `/api/rag` available to the browser assistant by default. Set `RAG_API_KEY` to require a bearer token and prevent anonymous model spend on a public site. Because the browser cannot safely hold this secret, use `SITE_PASSWORD` instead when authenticated browser visitors need the built-in assistant. +- **MCP endpoint** - the `/api/mcp` route requires a bearer token when `DOCS_API_KEY` is set. Without `DOCS_API_KEY`, it falls back to the site gate: a configured `SITE_PASSWORD` requires a valid gate session, while an intentionally public site leaves MCP public. Set `DOCS_API_KEY` for independent server-to-server authentication. - **Site password** - `SITE_PASSWORD` gates the whole site behind a single shared password. It is a lightweight access gate, not a substitute for per-user authentication. - **Generated output** - review generated code before deploying to production, especially if you customize templates. diff --git a/package.json b/package.json index cafa584..1a32a80 100644 --- a/package.json +++ b/package.json @@ -7,13 +7,21 @@ "doccupine": "dist/index.js" }, "type": "module", + "engines": { + "node": ">=22" + }, + "packageManager": "pnpm@11.18.0", "scripts": { - "build": "tsc", + "clean": "node scripts/clean.mjs", + "build": "pnpm clean && tsc", "dev": "tsc --watch", "start": "node dist/index.js", - "prepare": "tsc", - "test": "vitest run", - "format": "prettier --write ." + "prepare": "node scripts/clean.mjs && tsc", + "test": "pnpm build && vitest run", + "test:package": "pnpm build && node scripts/verify-package.mjs", + "smoke:generated": "node scripts/smoke-generated-site.mjs", + "format": "prettier --write .", + "format:check": "prettier --check ." }, "keywords": [ "doccupine", @@ -40,10 +48,7 @@ "commander": "^15.0.0", "fs-extra": "^11.4.0", "gray-matter": "^4.0.3", - "next": "^16.2.12", - "prompts": "^2.4.2", - "react": "^19.2.8", - "react-dom": "^19.2.8" + "prompts": "^2.4.2" }, "devDependencies": { "@types/fs-extra": "^11.0.4", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 62545f3..c6b39dc 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -26,18 +26,9 @@ importers: gray-matter: specifier: ^4.0.3 version: 4.0.3 - next: - specifier: ^16.2.12 - version: 16.2.12(react-dom@19.2.8(react@19.2.8))(react@19.2.8) prompts: specifier: ^2.4.2 version: 2.4.2 - react: - specifier: ^19.2.8 - version: 19.2.8 - react-dom: - specifier: ^19.2.8 - version: 19.2.8(react@19.2.8) devDependencies: '@types/fs-extra': specifier: ^11.0.4 diff --git a/scripts/clean.mjs b/scripts/clean.mjs new file mode 100644 index 0000000..a556d6a --- /dev/null +++ b/scripts/clean.mjs @@ -0,0 +1,9 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const scriptDir = path.dirname(fileURLToPath(import.meta.url)); +await fs.rm(path.resolve(scriptDir, "..", "dist"), { + recursive: true, + force: true, +}); diff --git a/scripts/smoke-generated-site.mjs b/scripts/smoke-generated-site.mjs new file mode 100644 index 0000000..72cd0b0 --- /dev/null +++ b/scripts/smoke-generated-site.mjs @@ -0,0 +1,130 @@ +import { spawn } from "node:child_process"; +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); +const projectDir = await fs.mkdtemp(path.join(os.tmpdir(), "doccupine-smoke-")); +const packageManager = process.platform === "win32" ? "pnpm.cmd" : "pnpm"; + +function run(command, args, cwd) { + return new Promise((resolve, reject) => { + const child = spawn(command, args, { cwd, stdio: "inherit" }); + child.on("error", reject); + child.on("close", (code) => { + if (code === 0) resolve(); + else + reject(new Error(`${command} ${args.join(" ")} exited with ${code}`)); + }); + }); +} + +try { + await fs.mkdir(path.join(projectDir, "docs"), { recursive: true }); + await fs.writeFile( + path.join(projectDir, "doccupine.json"), + JSON.stringify( + { + watchDir: "docs", + outputDir: "site", + port: "3000", + openapi: "openapi.json", + }, + null, + 2, + ), + ); + await fs.writeFile( + path.join(projectDir, "config.json"), + JSON.stringify({ name: "Smoke Docs", url: "https://docs.example.test" }), + ); + await fs.writeFile( + path.join(projectDir, "docs", "index.mdx"), + [ + "---", + 'title: "Using `widgets` and ${safeText}"', + 'description: "Quotes, `code`, and ${expressions} stay data."', + "---", + "", + "# Smoke test", + "", + "The generated site must compile.", + "", + ].join("\n"), + ); + await fs.writeFile( + path.join(projectDir, "docs", "guide.mdx"), + [ + "---", + 'title: "Guide"', + 'section: "Guides"', + "---", + "", + "# Guide", + "", + "A sectioned page.", + "", + ].join("\n"), + ); + await fs.writeFile( + path.join(projectDir, "openapi.json"), + JSON.stringify({ + openapi: "3.1.0", + info: { title: "Smoke API", version: "1.0.0" }, + servers: [{ url: "https://api.example.test/v1" }], + paths: { + "/widgets/{quoted}": { + get: { + operationId: "getWidget", + summary: "Get a widget", + parameters: [ + { + name: 'quoted"name', + in: "path", + required: true, + schema: { type: "string" }, + }, + ], + responses: { 200: { description: "OK" } }, + }, + }, + }, + }), + ); + + await run( + process.execPath, + [path.join(root, "dist", "index.js"), "build"], + projectDir, + ); + + const siteDir = path.join(projectDir, "site"); + await run(packageManager, ["install", "--frozen-lockfile=false"], siteDir); + await run(packageManager, ["run", "type-check"], siteDir); + await run(packageManager, ["run", "lint", "--max-warnings=0"], siteDir); + await run(packageManager, ["run", "build"], siteDir); + + for (const route of ["mcp", "rag"]) { + const tracePath = path.join( + siteDir, + ".next", + "server", + "app", + "api", + route, + "route.js.nft.json", + ); + const trace = JSON.parse(await fs.readFile(tracePath, "utf8")); + if ( + !Array.isArray(trace.files) || + !trace.files.some((file) => + file.endsWith("services/mcp/docs-content.json"), + ) + ) { + throw new Error(`${route} route did not trace docs-content.json`); + } + } +} finally { + await fs.rm(projectDir, { recursive: true, force: true }); +} diff --git a/scripts/verify-package.mjs b/scripts/verify-package.mjs new file mode 100644 index 0000000..68f1336 --- /dev/null +++ b/scripts/verify-package.mjs @@ -0,0 +1,268 @@ +import { spawn } from "node:child_process"; +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); +const npm = process.platform === "win32" ? "npm.cmd" : "npm"; +const packageJson = JSON.parse( + await fs.readFile(path.join(root, "package.json"), "utf8"), +); + +if (packageJson.scripts?.prepare !== "node scripts/clean.mjs && tsc") { + throw new Error( + 'prepare must remain package-manager-neutral: expected "node scripts/clean.mjs && tsc"', + ); +} + +async function filesBelow(directory) { + const result = []; + for (const entry of await fs.readdir(directory, { withFileTypes: true })) { + const fullPath = path.join(directory, entry.name); + if (entry.isDirectory()) result.push(...(await filesBelow(fullPath))); + else result.push(fullPath); + } + return result; +} + +function run(command, args, cwd) { + return new Promise((resolve, reject) => { + const child = spawn(command, args, { + cwd, + shell: false, + windowsHide: true, + }); + let stdout = ""; + let stderr = ""; + + child.stdout.setEncoding("utf8"); + child.stderr.setEncoding("utf8"); + child.stdout.on("data", (data) => { + stdout += data; + }); + child.stderr.on("data", (data) => { + stderr += data; + }); + child.on("error", reject); + child.on("close", (code) => { + if (code === 0) { + resolve({ stdout, stderr }); + return; + } + + reject( + new Error( + [ + `${command} ${args.join(" ")} exited with ${code}`, + stdout.trim(), + stderr.trim(), + ] + .filter(Boolean) + .join("\n"), + ), + ); + }); + }); +} + +function normalizePackagePath(filePath, label) { + if (typeof filePath !== "string" || filePath.length === 0) { + throw new Error(`${label} must be a non-empty package-relative path`); + } + + const normalized = filePath.replaceAll("\\", "/").replace(/^\.\//, ""); + if (path.posix.isAbsolute(normalized) || normalized.startsWith("../")) { + throw new Error(`${label} must stay within the package: ${filePath}`); + } + return normalized; +} + +const main = normalizePackagePath(packageJson.main, "main"); +const bin = packageJson.bin; +const doccupineBin = normalizePackagePath( + typeof bin === "string" ? bin : bin?.doccupine, + "bin.doccupine", +); + +const sourceFiles = new Set( + (await filesBelow(path.join(root, "src"))) + .filter((file) => file.endsWith(".ts") && !file.endsWith(".test.ts")) + .map((file) => path.relative(path.join(root, "src"), file)), +); + +const stale = [ + ...new Set( + (await filesBelow(path.join(root, "dist"))) + .filter((file) => file.endsWith(".js") || file.endsWith(".d.ts")) + .map((file) => + path + .relative(path.join(root, "dist"), file) + .replace(/\.d\.ts$/, ".ts") + .replace(/\.js$/, ".ts"), + ), + ), +].filter((file) => !sourceFiles.has(file)); + +if (stale.length > 0) { + throw new Error(`Stale compiled files found:\n${stale.join("\n")}`); +} + +const distFiles = (await filesBelow(path.join(root, "dist"))).map((file) => + path.relative(root, file).split(path.sep).join("/"), +); +if (distFiles.length === 0) { + throw new Error("dist is empty; build the package before verifying it"); +} + +const autoIncludedRootFiles = (await fs.readdir(root, { withFileTypes: true })) + .filter( + (entry) => + entry.isFile() && /^(?:readme|licen[cs]e)(?:\..+)?$/i.test(entry.name), + ) + .map((entry) => entry.name); +const allowedFiles = new Set([ + "package.json", + ...autoIncludedRootFiles, + ...distFiles, +]); + +const temporaryRoot = await fs.mkdtemp( + path.join(os.tmpdir(), "doccupine-package-"), +); + +try { + const packDir = path.join(temporaryRoot, "pack"); + const projectDir = path.join(temporaryRoot, "project"); + await Promise.all([fs.mkdir(packDir), fs.mkdir(projectDir)]); + + const packed = await run( + npm, + ["pack", root, "--ignore-scripts", "--json", "--pack-destination", packDir], + temporaryRoot, + ); + + let packResult; + try { + [packResult] = JSON.parse(packed.stdout); + } catch (error) { + throw new Error(`Could not parse npm pack output:\n${packed.stdout}`, { + cause: error, + }); + } + + if (!packResult?.filename || !Array.isArray(packResult.files)) { + throw new Error("npm pack did not report a tarball and its contents"); + } + + const packedFiles = new Set( + packResult.files.map(({ path: filePath }) => + filePath.replaceAll("\\", "/").replace(/^package\//, ""), + ), + ); + const requiredFiles = new Set([...allowedFiles, main, doccupineBin]); + const missingFiles = [...requiredFiles].filter( + (file) => !packedFiles.has(file), + ); + if (missingFiles.length > 0) { + throw new Error( + `Required files missing from npm tarball:\n${missingFiles.join("\n")}`, + ); + } + const unexpectedFiles = [...packedFiles].filter( + (file) => !allowedFiles.has(file), + ); + if (unexpectedFiles.length > 0) { + throw new Error( + `Unexpected files found in npm tarball:\n${unexpectedFiles.join("\n")}`, + ); + } + + const tarball = path.join(packDir, packResult.filename); + await fs.access(tarball); + await fs.writeFile( + path.join(projectDir, "package.json"), + `${JSON.stringify({ name: "doccupine-package-test", private: true }, null, 2)}\n`, + ); + + await run( + npm, + [ + "install", + "--ignore-scripts", + "--omit=dev", + "--no-audit", + "--no-fund", + "--no-package-lock", + tarball, + ], + projectDir, + ); + + const installedPackageDir = path.join( + projectDir, + "node_modules", + packageJson.name, + ); + const installedPackageJson = JSON.parse( + await fs.readFile(path.join(installedPackageDir, "package.json"), "utf8"), + ); + if ( + installedPackageJson.main !== packageJson.main || + JSON.stringify(installedPackageJson.bin) !== JSON.stringify(packageJson.bin) + ) { + throw new Error( + "Installed package main or bin metadata differs from source", + ); + } + + const binShim = path.join( + projectDir, + "node_modules", + ".bin", + process.platform === "win32" ? "doccupine.cmd" : "doccupine", + ); + await fs.access(binShim); + + await run( + process.execPath, + [ + "--input-type=module", + "--eval", + `await import(${JSON.stringify(packageJson.name)})`, + ], + projectDir, + ); + let version; + if (process.platform === "win32") { + version = await run( + npm, + [ + "exec", + "--offline", + "--yes=false", + "--ignore-scripts", + "--cache", + path.join(temporaryRoot, "exec-cache"), + `--package=${packageJson.name}@${packageJson.version}`, + "--", + "doccupine", + "--version", + ], + projectDir, + ); + } else { + version = await run(binShim, ["--version"], projectDir); + } + if (version.stdout.trim() !== packageJson.version) { + throw new Error( + `Installed binary reported ${JSON.stringify(version.stdout.trim())}; expected ${packageJson.version}`, + ); + } + + console.log( + `verified ${packResult.filename}: ${packedFiles.size} files, installed binary ${packageJson.version}`, + ); +} finally { + await fs.rm(temporaryRoot, { recursive: true, force: true }); +} diff --git a/src/cli.ts b/src/cli.ts new file mode 100644 index 0000000..73f953c --- /dev/null +++ b/src/cli.ts @@ -0,0 +1,277 @@ +import { spawn } from "child_process"; +import { createHash } from "crypto"; +import { fileURLToPath } from "url"; +import { Command } from "commander"; +import fs from "fs-extra"; +import path from "path"; +import chalk from "chalk"; + +import { ConfigManager, normalizeOpenApiConfig } from "./lib/config-manager.js"; +import { + findAvailablePort, + resolvePackageManager, + writeFileAtomic, +} from "./lib/utils.js"; +import { MDXToNextJSGenerator } from "./mdx-to-nextjs-generator.js"; + +async function dependencyFingerprint( + outputDir: string, + packageManager: string, +): Promise { + const packageJson = await fs.readFile( + path.join(outputDir, "package.json"), + "utf8", + ); + return createHash("sha256") + .update(packageManager) + .update("\0") + .update(packageJson) + .digest("hex"); +} + +async function needsDependencyInstall( + outputDir: string, + packageManager: string, +): Promise { + if (!(await fs.pathExists(path.join(outputDir, "node_modules")))) return true; + const stampPath = path.join(outputDir, ".doccupine-install"); + try { + const expected = await dependencyFingerprint(outputDir, packageManager); + return (await fs.readFile(stampPath, "utf8")).trim() !== expected; + } catch { + return true; + } +} + +async function recordDependencyInstall( + outputDir: string, + packageManager: string, +): Promise { + const fingerprint = await dependencyFingerprint(outputDir, packageManager); + await writeFileAtomic( + path.join(outputDir, ".doccupine-install"), + `${fingerprint}\n`, + ); +} + +export async function runCli(argv?: string[]): Promise { + const __dirname = path.dirname(fileURLToPath(import.meta.url)); + const packageJson = JSON.parse( + await fs.readFile(path.join(__dirname, "..", "package.json"), "utf8"), + ) as { version: string }; + const program = new Command(); + + program + .name("doccupine") + .description( + "Watch MDX files and generate Next.js documentation pages automatically", + ) + .version(packageJson.version); + + program + .command("watch", { isDefault: true }) + .description("Watch a directory for MDX changes and generate Next.js app") + .option("--port ", "Port for Next.js dev server") + .option("--verbose", "Show verbose output") + .option("--reset", "Reset configuration and prompt for new directories") + .option("--skip-install", "Skip dependency installation") + .option( + "--package-manager ", + "Package manager for the generated app: pnpm or npm (default: auto-detect)", + ) + .action(async (options) => { + const configManager = new ConfigManager(); + const config = await configManager.getConfig({ + reset: options.reset, + port: options.port, + }); + + const generator = new MDXToNextJSGenerator( + config.watchDir, + config.outputDir, + normalizeOpenApiConfig(config.openapi), + ); + + // Config paths are project-relative; child processes get an absolute cwd. + const outputDir = path.resolve(process.cwd(), config.outputDir); + + await generator.init(); + + let devServer: ReturnType | null = null; + + // Prefer pnpm (the generated app ships a pnpm workspace) and fall back to + // npm. A --package-manager flag or a "packageManager" field in + // doccupine.json overrides detection; the flag wins. + const packageManager = resolvePackageManager( + options.packageManager ?? config.packageManager, + ); + + console.log(chalk.blue(`šŸ“¦ Using ${packageManager.name}...`)); + + if (options.skipInstall) { + console.log(chalk.yellow("ā­ļø Skipping dependency installation")); + } else if (await needsDependencyInstall(outputDir, packageManager.name)) { + console.log(chalk.blue("šŸ“¦ Installing dependencies...")); + const install = spawn(packageManager.bin, ["install"], { + cwd: outputDir, + stdio: "inherit", + }); + + await new Promise((resolve, reject) => { + install.on("close", (code) => { + if (code === 0) { + resolve(void 0); + } else { + reject( + new Error( + `${packageManager.name} install failed with code ${code}`, + ), + ); + } + }); + install.on("error", reject); + }); + await recordDependencyInstall(outputDir, packageManager.name); + console.log(chalk.green("āœ… Dependencies installed")); + } else { + console.log(chalk.green("āœ… Dependencies are already up to date")); + } + + const requestedPort = Number(config.port); + const port = await findAvailablePort(requestedPort); + if (port !== requestedPort) { + console.log( + chalk.yellow( + `āš ļø Port ${config.port} is in use, using port ${port} instead`, + ), + ); + } + console.log( + chalk.blue(`šŸš€ Starting Next.js dev server on port ${port}...`), + ); + const portStr = String(port); + const devArgs = + packageManager.name === "npm" + ? ["run", "dev", "--", "--port", portStr] + : ["run", "dev", "--port", portStr]; + devServer = spawn(packageManager.bin, devArgs, { + cwd: outputDir, + stdio: ["ignore", "pipe", "pipe"], + }); + + devServer.stdout?.on("data", (data: Buffer) => { + const output = data.toString(); + if (output.includes("Ready") || output.includes("started")) { + console.log( + chalk.green(`šŸŒ Next.js ready at http://localhost:${port}`), + ); + } + if (options.verbose) { + process.stdout.write(chalk.gray("[Next.js] ") + output); + } else if ( + output.includes("compiled") || + output.includes("error") || + output.includes("Ready") + ) { + process.stdout.write(chalk.gray("[Next.js] ") + output); + } + }); + + devServer.stderr?.on("data", (data: Buffer) => { + const output = data.toString(); + if ( + options.verbose || + output.includes("Error") || + output.includes("error") + ) { + process.stderr.write(chalk.red("[Next.js] ") + output); + } + }); + + devServer.on("error", (error: Error) => { + console.error(chalk.red("āŒ Error starting dev server:"), error); + }); + + devServer.on("close", (code: number | null) => { + if (code && code !== 0) { + console.error( + chalk.red(`āŒ Next.js dev server exited with code ${code}`), + ); + } + }); + + await generator.startWatching(); + + process.on("SIGINT", async () => { + console.log(chalk.yellow("\nšŸ›‘ Shutting down...")); + await generator.stop(); + if (devServer) { + devServer.kill(); + } + process.exit(0); + }); + + console.log( + chalk.green("šŸŽ‰ Generator is running! Press Ctrl+C to stop."), + ); + console.log(chalk.cyan(`šŸ“ Edit your MDX files in: ${config.watchDir}`)); + }); + + program + .command("build") + .alias("generate") + .description("Generate the Next.js app once without running its build") + .option("--reset", "Reset configuration and prompt for new directories") + .action(async (options) => { + const configManager = new ConfigManager(); + const config = await configManager.getConfig({ + reset: options.reset, + }); + + const generator = new MDXToNextJSGenerator( + config.watchDir, + config.outputDir, + normalizeOpenApiConfig(config.openapi), + ); + await generator.init(); + console.log(chalk.green("šŸŽ‰ Generation complete!")); + }); + + program + .command("config") + .description("Show or reset configuration") + .option("--show", "Show current configuration") + .option("--reset", "Reset configuration") + .action(async (options) => { + const configManager = new ConfigManager(); + + if (options.show) { + const config = await configManager.loadConfig(); + if (config) { + console.log(chalk.blue("šŸ“„ Current configuration:")); + console.log( + chalk.white("Watch Directory:"), + chalk.cyan(path.relative(process.cwd(), config.watchDir)), + ); + console.log( + chalk.white("Output Directory:"), + chalk.cyan(path.relative(process.cwd(), config.outputDir)), + ); + console.log(chalk.white("Port:"), chalk.cyan(config.port || "3000")); + } else { + console.log(chalk.yellow("āš ļø No configuration file found")); + } + } else if (options.reset) { + await configManager.getConfig({ reset: true }); + console.log(chalk.green("āœ… Configuration reset")); + } else { + console.log( + chalk.blue( + "Use --show to display configuration or --reset to reset it", + ), + ); + } + }); + + await program.parseAsync(argv ?? process.argv); +} diff --git a/src/index.test.ts b/src/index.test.ts index 395809d..b4e4eab 100644 --- a/src/index.test.ts +++ b/src/index.test.ts @@ -20,6 +20,9 @@ import { docsTemplate } from "./templates/components/Docs.js"; const execFileAsync = promisify(execFile); const selfPath = fileURLToPath(import.meta.url); const distEntry = path.resolve(selfPath, "..", "..", "dist", "index.js"); +if (!fs.pathExistsSync(distEntry)) { + throw new Error("Compiled CLI is missing; run `pnpm build` before Vitest"); +} // A generated app formats with `prettier --write .` and pins prettier ^3.9.6. // Tests can't `npm install` the generated app, so we reuse this repo's matching @@ -188,8 +191,8 @@ describe("isProcessEntrypoint", () => { }); }); -// Runs against the compiled output, so it needs a build. CI always runs -// `pnpm build` before `pnpm test`, and `pnpm install` builds via `prepare`. +// Runs against compiled output. The test script builds first so these checks +// can never silently exercise stale output or disappear on a clean checkout. describe.skipIf(!fs.pathExistsSync(distEntry))( "compiled CLI entrypoint", () => { @@ -219,7 +222,7 @@ describe.skipIf(!fs.pathExistsSync(distEntry))( // A generated Doccupine app ships a `format` script (`prettier --write .`). // This builds a real app from the templates and asserts it is already // Prettier-clean - i.e. running that script is a no-op - so every template -// emits format-stable output. Skipped unless the CLI is built (dist present). +// emits format-stable output. The test script builds dist before Vitest runs. describe.skipIf(!fs.pathExistsSync(distEntry))( "generated site is prettier-clean", () => { @@ -243,7 +246,14 @@ describe.skipIf(!fs.pathExistsSync(distEntry))( schema: { type: "string" }, }, ], - responses: { "200": { description: "OK" } }, + responses: { + "200": { + description: "OK", + content: { + "application/json": { example: { id: "OL123W" } }, + }, + }, + }, }, }, "/notes": { @@ -297,19 +307,26 @@ describe.skipIf(!fs.pathExistsSync(distEntry))( ); // ...and the spec must have produced an endpoint page, so the format // check actually covers the synthetic API-page output. - expect( - await fs.pathExists( - path.join( - outDir, - "app", - "(site)", - "api-reference", - "admin", - "getworkbyid", - "page.tsx", - ), - ), - ).toBe(true); + const endpointPagePath = path.join( + outDir, + "app", + "(site)", + "api-reference", + "admin", + "getworkbyid", + "page.tsx", + ); + expect(await fs.pathExists(endpointPagePath)).toBe(true); + expect(await fs.readFile(endpointPagePath, "utf8")).toContain( + ' page with `rss: true`, so the // format check also covers the generated feed route and the RSS-button // page shape. @@ -340,6 +357,45 @@ describe.skipIf(!fs.pathExistsSync(distEntry))( await fs.remove(projectDir); } }, 120_000); + + it("keeps a one-page artifact manifest out of generated formatting", async () => { + const projectDir = await fs.mkdtemp( + path.join(os.tmpdir(), "doccupine-fmt-one-page-"), + ); + try { + await fs.writeJson(path.join(projectDir, "doccupine.json"), { + watchDir: "docs", + outputDir: "out", + port: "3000", + }); + await fs.outputFile( + path.join(projectDir, "docs", "index.mdx"), + "---\ntitle: Home\n---\n# Home\n", + ); + + await execFileAsync(process.execPath, [distEntry, "build"], { + cwd: projectDir, + maxBuffer: 20 * 1024 * 1024, + }); + + const outDir = path.join(projectDir, "out"); + await expect( + fs.readFile(path.join(outDir, ".doccupine-artifacts.json"), "utf8"), + ).resolves.toContain('"index.md"'); + await expect( + fs.readFile(path.join(outDir, ".prettierignore"), "utf8"), + ).resolves.toContain(".doccupine-*"); + + const { stdout } = await execFileAsync( + process.execPath, + [prettierCli, "--check", "."], + { cwd: outDir, maxBuffer: 20 * 1024 * 1024 }, + ); + expect(stdout).toContain("All matched files use Prettier code style!"); + } finally { + await fs.remove(projectDir); + } + }, 120_000); }, ); @@ -585,3 +641,87 @@ describe.skipIf(!fs.pathExistsSync(distEntry))("MDX error resilience", () => { } }, 120_000); }); + +describe.skipIf(!fs.pathExistsSync(distEntry))( + "generation safety invariants", + () => { + it("fails with both source paths when MDX routes collide", async () => { + const projectDir = await fs.mkdtemp( + path.join(os.tmpdir(), "doccupine-collision-"), + ); + try { + await fs.writeJson(path.join(projectDir, "doccupine.json"), { + watchDir: "docs", + outputDir: "out", + port: "3000", + }); + await fs.outputFile( + path.join(projectDir, "docs", "index.mdx"), + "---\ntitle: Home\n---\n# Home\n", + ); + await fs.outputFile( + path.join(projectDir, "docs", "guide.mdx"), + "---\ntitle: Guide\n---\n# Guide\n", + ); + await fs.outputFile( + path.join(projectDir, "docs", "guide", "index.mdx"), + "---\ntitle: Other guide\n---\n# Other\n", + ); + + let stderr = ""; + try { + await execFileAsync(process.execPath, [distEntry, "build"], { + cwd: projectDir, + }); + throw new Error("Expected route collision to fail generation"); + } catch (error) { + stderr = (error as { stderr?: string }).stderr ?? ""; + } + expect(stderr).toContain("Route collision"); + expect(stderr).toContain('"guide.mdx"'); + expect(stderr).toContain('"guide/index.mdx"'); + } finally { + await fs.remove(projectDir); + } + }); + + it("refuses to overwrite an unrelated non-empty output directory", async () => { + const projectDir = await fs.mkdtemp( + path.join(os.tmpdir(), "doccupine-output-refusal-"), + ); + try { + await fs.writeJson(path.join(projectDir, "doccupine.json"), { + watchDir: "docs", + outputDir: "existing", + port: "3000", + }); + await fs.outputFile( + path.join(projectDir, "docs", "index.mdx"), + "# Home\n", + ); + await fs.outputFile( + path.join(projectDir, "existing", "important.txt"), + "keep me", + ); + + await expect( + execFileAsync(process.execPath, [distEntry, "build"], { + cwd: projectDir, + }), + ).rejects.toMatchObject({ + stderr: expect.stringContaining( + "Refusing to overwrite non-empty directory", + ), + }); + expect( + await fs.readFile( + path.join(projectDir, "existing", "important.txt"), + "utf8", + ), + ).toBe("keep me"); + } finally { + await fs.remove(projectDir); + } + }); + }, +); diff --git a/src/index.ts b/src/index.ts index 37a2c8b..5771953 100644 --- a/src/index.ts +++ b/src/index.ts @@ -1,2560 +1,22 @@ #!/usr/bin/env node -import { program } from "commander"; -import chokidar, { FSWatcher } from "chokidar"; import fs from "fs-extra"; -import path from "path"; import { fileURLToPath } from "url"; -import chalk from "chalk"; - -import { - appStructure, - obsoleteFiles, - startingDocsStructure, -} from "./lib/structures.js"; -import { rootLayoutTemplate, siteLayoutTemplate } from "./lib/layout.js"; -import { ConfigManager, normalizeOpenApiConfig } from "./lib/config-manager.js"; -import { - OpenApiRegistry, - DEFAULT_API_BASE_SLUG, - buildEndpointDoc, -} from "./lib/openapi.js"; -import { - findAvailablePort, - generateSlug, - getFullSlug, - escapeTemplateContent, - toJsStringLiteral, - resolvePackageManager, - safeMatter, -} from "./lib/utils.js"; -import { - generateMetadataBlock, - generateRuntimeOnlyMetadataBlock, - generateJsonLdScript, -} from "./lib/metadata.js"; -import { parseUpdateBlocks } from "./lib/rss.js"; -import { nextConfigTemplate } from "./templates/next.config.js"; -import { pnpmWorkspaceTemplate } from "./templates/pnpmWorkspace.js"; -import { proxyTemplate } from "./templates/proxy.js"; -import { robotsTemplate } from "./templates/app/robots.js"; -import { rssRouteTemplate } from "./templates/app/rssRoute.js"; -import { sitemapTemplate, type SitemapEntry } from "./templates/app/sitemap.js"; -import { llmsIndexTemplate } from "./templates/llms/llmsIndex.js"; -import { - llmsFullTemplate, - type PageWithBody, -} from "./templates/llms/llmsFull.js"; -import { llmsPageTemplate } from "./templates/llms/llmsPage.js"; -import { siteDocsSlug, skillMdTemplate } from "./templates/llms/skillMd.js"; -import type { - DoccupineConfig, - MDXFile, - PageMeta, - SectionConfig, - FontConfig, - AnalyticsConfig, - NormalizedOpenApiSpec, -} from "./lib/types.js"; -import type { OperationDescriptor } from "./lib/openapi-types.js"; - export { generateSlug, getFullSlug, escapeTemplateContent, toJsStringLiteral, } from "./lib/utils.js"; +export { MDXToNextJSGenerator } from "./mdx-to-nextjs-generator.js"; const __filename = fileURLToPath(import.meta.url); -const __dirname = path.dirname(__filename); -const packageJson = JSON.parse( - fs.readFileSync(path.join(__dirname, "..", "package.json"), "utf8"), -); -const version = packageJson.version; - -class MDXToNextJSGenerator { - private watchDir: string; - private outputDir: string; - private rootDir: string; - private watcher: FSWatcher | null = null; - private configWatcher: FSWatcher | null = null; - private fontWatcher: FSWatcher | null = null; - private publicWatcher: FSWatcher | null = null; - private rootDirWatcher: FSWatcher | null = null; - private analyticsWatcher: FSWatcher | null = null; - private openApiWatcher: FSWatcher | null = null; - private doccupineConfigWatcher: FSWatcher | null = null; - private doccupineConfigFile = "doccupine.json"; - private configFiles = [ - "theme.json", - "navigation.json", - "config.json", - "links.json", - "sections.json", - ]; - private fontConfigFile = "fonts.json"; - private analyticsConfigFile = "analytics.json"; - private analyticsConfig: AnalyticsConfig | null = null; - private sectionsConfig: SectionConfig[] | null = null; - /** Guards against recursive reprocessing when maybeUpdateSections() triggers processAllMDXFiles() */ - private isReprocessing = false; - /** Tracks per-page .md files written under public/ so we can clean up stale ones on rename/delete */ - private generatedLlmsPagePaths = new Set(); - /** OpenAPI specs (build config) that drive the generated API reference. */ - private openApiSpecs: NormalizedOpenApiSpec[]; - private apiBaseSlug = DEFAULT_API_BASE_SLUG; - private apiRegistry = new OpenApiRegistry(); - /** Route slugs of the endpoint pages written last pass, for stale cleanup. */ - private generatedApiPageSlugs = new Set(); - /** Section slugs whose index redirect we wrote this session, for cleanup. */ - private generatedSectionIndexSlugs = new Set(); - - constructor( - watchDir: string, - outputDir: string, - openApiSpecs: NormalizedOpenApiSpec[] = [], - ) { - this.watchDir = path.resolve(watchDir); - this.outputDir = path.resolve(outputDir); - this.rootDir = process.cwd(); - this.openApiSpecs = openApiSpecs; - } - - async init() { - console.log(chalk.blue("šŸš€ Initializing MDX to Next.js generator...")); - - await fs.ensureDir(this.watchDir); - await fs.ensureDir(this.outputDir); - - this.sectionsConfig = await this.resolveSections(); - this.analyticsConfig = await this.loadAnalyticsConfig(); - - if (this.analyticsConfig) { - console.log( - chalk.blue(`šŸ“Š Analytics enabled: ${this.analyticsConfig.provider}`), - ); - } - - // Parse OpenAPI spec(s) before generating structure so the synthetic - // endpoint pages flow into the very first layout/sitemap/llms pass. - await this.loadOpenApiRegistry(); - - await this.createNextJSStructure(); - await this.createStartingDocs(); - await this.copyCustomConfigFiles(); - await this.copyFontConfig(); - await this.copyAnalyticsConfig(); - await this.copyPublicFiles(); - - // createStartingDocs() may have written the sample docs - which carry - // section frontmatter - after the initial resolveSections() ran against an - // empty watch dir. Re-resolve now that every MDX file is on disk so the - // build applies the correct sections in a single O(n) pass, instead of - // rediscovering them per file (the old O(nĀ²) behavior). - this.sectionsConfig = await this.resolveSections(); - if (this.sectionsConfig) { - console.log( - chalk.blue( - `šŸ“‘ Found ${this.sectionsConfig.length} section(s): ${this.sectionsConfig.map((s) => s.label).join(", ")}`, - ), - ); - } - - // Write the endpoint pages + request allowlist before the MDX pass so its - // aggregate refresh (nav/sitemap/llms) already sees them on disk and in the - // registry. - await this.writeApiPages(); - - await this.processAllMDXFiles(); - - console.log(chalk.green("āœ… Initial setup complete!")); - console.log(chalk.cyan("šŸ’” To start the Next.js dev server:")); - console.log( - chalk.white(` cd ${path.relative(process.cwd(), this.outputDir)}`), - ); - console.log(chalk.white(" npm install && npm run dev")); - } - - async createNextJSStructure() { - // Clear the generated app/ directory first so a fresh run never inherits - // stale routes from a previous version (e.g. pages left at their old paths - // after a route-group move would collide with the newly generated ones). - // Everything under app/ is regenerated below and by processAllMDXFiles / - // generateSectionIndexPages, so nothing here is user-authored. Config JSONs - // and other generated dirs live outside app/ and are untouched. - await fs.remove(path.join(this.outputDir, "app")); - - // Drop files that earlier CLI versions generated but no longer exist in - // the template set, so upgraded projects don't keep stale copies. - await Promise.all( - obsoleteFiles.map((file) => fs.remove(path.join(this.outputDir, file))), - ); - - const siteUrl = await this.loadSiteUrl(); - - const structure: Record> = { - ...appStructure, - "next.config.ts": nextConfigTemplate(this.analyticsConfig), - "pnpm-workspace.yaml": pnpmWorkspaceTemplate, - "proxy.ts": proxyTemplate(this.analyticsConfig), - "analytics.json": `{}\n`, - "config.json": `{}\n`, - "links.json": `[]\n`, - "navigation.json": `[]\n`, - "sections.json": `[]\n`, - "theme.json": `{}\n`, - "app/robots.ts": robotsTemplate(siteUrl !== null), - "app/layout.tsx": this.generateRootLayout(), - "app/(site)/layout.tsx": this.generateSiteLayout(), - }; - - for (const [filePath, content] of Object.entries(structure)) { - const fullPath = path.join(this.outputDir, filePath); - await fs.ensureDir(path.dirname(fullPath)); - await fs.writeFile(fullPath, String(await content), "utf8"); - } - - await this.updateSitemap(); - await this.updateLlmsFiles(); - } - - async createStartingDocs() { - const structure = startingDocsStructure; - - const indexMdxExists = await fs.pathExists( - path.join(this.watchDir, "index.mdx"), - ); - - if (!indexMdxExists) { - for (const [filePath, content] of Object.entries(structure)) { - const fullPath = path.join(this.watchDir, filePath); - await fs.ensureDir(path.dirname(fullPath)); - await fs.writeFile(fullPath, String(content), "utf8"); - } - } - } - - async copyCustomConfigFiles() { - console.log( - chalk.blue(`šŸ” Checking for config files in: ${this.watchDir}`), - ); - - for (const configFile of this.configFiles) { - const sourcePath = path.join(this.rootDir, configFile); - const destPath = path.join(this.outputDir, configFile); - - console.log(chalk.gray(` Checking ${configFile}...`)); - - if (await fs.pathExists(sourcePath)) { - await fs.copy(sourcePath, destPath); - console.log(chalk.green(` āœ“ Copied ${configFile} to Next.js app`)); - } else { - console.log(chalk.gray(` āœ— ${configFile} not found, skipping`)); - } - } - } - - async copyFontConfig() { - console.log(chalk.blue(`šŸ” Checking for font configuration...`)); - - const sourcePath = path.join(this.rootDir, this.fontConfigFile); - const destPath = path.join(this.outputDir, this.fontConfigFile); - - if (await fs.pathExists(sourcePath)) { - await fs.copy(sourcePath, destPath); - console.log( - chalk.green(` āœ“ Copied ${this.fontConfigFile} to Next.js app`), - ); - } else { - console.log(chalk.gray(` āœ— ${this.fontConfigFile} not found, skipping`)); - } - } - - async loadFontConfig(): Promise { - const fontPath = path.join(this.rootDir, this.fontConfigFile); - - try { - if (await fs.pathExists(fontPath)) { - const fontContent = await fs.readFile(fontPath, "utf8"); - return JSON.parse(fontContent) as FontConfig; - } - } catch (error) { - console.warn( - chalk.yellow(`āš ļø Error reading ${this.fontConfigFile}`), - error, - ); - } - - return null; - } - - async loadAnalyticsConfig(): Promise { - const analyticsPath = path.join(this.rootDir, this.analyticsConfigFile); - - try { - if (await fs.pathExists(analyticsPath)) { - const content = await fs.readFile(analyticsPath, "utf8"); - const parsed = JSON.parse(content); - if (parsed?.provider === "posthog" && parsed.posthog?.key) { - return parsed as AnalyticsConfig; - } - } - } catch (error) { - console.warn( - chalk.yellow(`āš ļø Error reading ${this.analyticsConfigFile}`), - error, - ); - } - - return null; - } - - async copyAnalyticsConfig() { - console.log(chalk.blue(`šŸ” Checking for analytics configuration...`)); - - const sourcePath = path.join(this.rootDir, this.analyticsConfigFile); - const destPath = path.join(this.outputDir, this.analyticsConfigFile); - - if (await fs.pathExists(sourcePath)) { - await fs.copy(sourcePath, destPath); - console.log( - chalk.green(` āœ“ Copied ${this.analyticsConfigFile} to Next.js app`), - ); - } else { - console.log( - chalk.gray(` āœ— ${this.analyticsConfigFile} not found, skipping`), - ); - } - } - - async loadSectionsConfig(): Promise { - const sectionsPath = path.join(this.rootDir, "sections.json"); - - try { - if (await fs.pathExists(sectionsPath)) { - const content = await fs.readFile(sectionsPath, "utf8"); - const parsed = JSON.parse(content) as SectionConfig[]; - if (Array.isArray(parsed) && parsed.length > 0) { - return parsed; - } - } - } catch (error) { - console.warn(chalk.yellow("āš ļø Error reading sections.json"), error); - } - - return null; - } - - async discoverSectionsFromFrontmatter(): Promise { - const files = await this.getAllMDXFiles(); - const sectionMap = new Map(); - let hasUnsectionedPages = false; - let defaultSectionLabel = "Docs"; - - for (const file of files) { - const fullPath = path.join(this.watchDir, file); - const content = await fs.readFile(fullPath, "utf8"); - const { data: frontmatter } = safeMatter(content, file); - - if (frontmatter.section) { - const label = frontmatter.section as string; - const order = (frontmatter.sectionOrder as number) || 0; - const existing = sectionMap.get(label); - if (!existing || order < existing.order) { - sectionMap.set(label, { label, order }); - } - } else { - hasUnsectionedPages = true; - } - - if ( - (file === "index.mdx" || file === "./index.mdx") && - frontmatter.sectionLabel - ) { - defaultSectionLabel = frontmatter.sectionLabel as string; - } - } - - if (sectionMap.size === 0) return null; - - const sorted = [...sectionMap.values()].sort((a, b) => a.order - b.order); - - const sections: SectionConfig[] = []; - - // Implicit root entry for pages without a section field - if (hasUnsectionedPages) { - sections.push({ label: defaultSectionLabel, slug: "" }); - } - - for (const s of sorted) { - sections.push({ - label: s.label, - slug: s.label.toLowerCase().replace(/\s+/g, "-"), - }); - } - - return sections; - } - - async resolveSections(): Promise { - const fromFile = await this.loadSectionsConfig(); - const base = fromFile ?? (await this.discoverSectionsFromFrontmatter()); - return this.withApiReferenceSection(base); - } - - /** - * Promotes the generated OpenAPI endpoints into a dedicated "API Reference" - * section so they get their own top-level nav, separate from hand-written - * docs. When the site had no sections, a root "Documentation" section is added - * for the existing pages so both appear in the section switcher. - */ - private withApiReferenceSection( - sections: SectionConfig[] | null, - ): SectionConfig[] | null { - if (this.apiRegistry.isEmpty) return sections; - const apiSection: SectionConfig = { - label: "API Reference", - slug: this.apiBaseSlug, - }; - if (!sections || sections.length === 0) { - return [{ label: "Documentation", slug: "" }, apiSection]; - } - if (sections.some((s) => s.slug === this.apiBaseSlug)) return sections; - return [...sections, apiSection]; - } - - private async reloadSections(): Promise { - console.log(chalk.cyan("šŸ“‘ Sections configuration changed")); - this.sectionsConfig = await this.resolveSections(); - await this.processAllMDXFiles(); - } - - private async maybeUpdateSections(): Promise { - if (this.isReprocessing) return; - - // Skip if sections.json exists (explicit config takes priority) - const fromFile = await this.loadSectionsConfig(); - if (fromFile) return; - - const newSections = this.withApiReferenceSection( - await this.discoverSectionsFromFrontmatter(), - ); - const changed = - JSON.stringify(newSections) !== JSON.stringify(this.sectionsConfig); - - if (changed) { - console.log( - chalk.cyan( - newSections - ? `šŸ“‘ Sections updated from frontmatter: ${newSections.map((s) => s.label).join(", ")}` - : "šŸ“‘ Sections cleared (no section frontmatter found)", - ), - ); - this.sectionsConfig = newSections; - this.isReprocessing = true; - try { - // processAllMDXFiles() already refreshes section index pages via its - // aggregate pass, so no separate generateSectionIndexPages() here. - await this.processAllMDXFiles(); - } finally { - this.isReprocessing = false; - } - } - } - - private determineSectionForFile( - filePath: string, - frontmatter: Record, - ): { sectionSlug: string; pageSlug: string } { - if (!this.sectionsConfig || this.sectionsConfig.length === 0) { - return { sectionSlug: "", pageSlug: generateSlug(filePath) }; - } - - const normalizedPath = filePath.replace(/\\/g, "/"); - - const firstDir = normalizedPath.includes("/") - ? normalizedPath.split("/")[0] - : ""; - - // Explicit directory matching (entries with a directory field) - for (const section of this.sectionsConfig) { - if (!section.directory) continue; - const dirPrefix = section.directory + "/"; - if (normalizedPath.startsWith(dirPrefix)) { - return { - sectionSlug: section.slug, - pageSlug: generateSlug(normalizedPath.slice(dirPrefix.length)), - }; - } - } - - // Directory matches section slug (auto-detect) - if (firstDir) { - const match = this.sectionsConfig.find((s) => s.slug === firstDir); - if (match) { - const pathForSlug = normalizedPath.slice(firstDir.length + 1); - return { - sectionSlug: match.slug, - pageSlug: generateSlug(pathForSlug), - }; - } - } - - // Frontmatter section field - if (frontmatter.section) { - const label = frontmatter.section as string; - const match = this.sectionsConfig.find((s) => s.label === label); - if (match) { - // Strip the directory if it matches the section slug - let pathForSlug = filePath; - if (firstDir && firstDir === match.slug) { - pathForSlug = normalizedPath.slice(firstDir.length + 1); - } - - return { - sectionSlug: match.slug, - pageSlug: generateSlug(pathForSlug), - }; - } - } - - // No section match - page stays at root - return { - sectionSlug: "", - pageSlug: generateSlug(filePath), - }; - } - - async handleConfigFileChange(filePath: string) { - const fileName = path.basename(filePath); - - if (this.configFiles.includes(fileName)) { - const sourcePath = path.join(this.rootDir, fileName); - const destPath = path.join(this.outputDir, fileName); - - try { - await fs.copy(sourcePath, destPath); - console.log(chalk.green(`šŸ“‹ Updated ${fileName} in Next.js app`)); - - if (fileName === "sections.json") { - await this.reloadSections(); - } - - if (fileName === "config.json") { - await this.updateSitemap(); - await this.updateRobots(); - await this.updateLlmsFiles(); - } - } catch (error) { - console.error(chalk.red(`āŒ Error copying ${fileName}:`), error); - } - } - } - - async handleConfigFileDelete(filePath: string) { - const fileName = path.basename(filePath); - - if (this.configFiles.includes(fileName)) { - const destPath = path.join(this.outputDir, fileName); - - try { - if (await fs.pathExists(destPath)) { - await fs.remove(destPath); - console.log(chalk.yellow(`šŸ—‘ļø Removed ${fileName} from Next.js app`)); - } - - if (fileName === "sections.json") { - await this.reloadSections(); - } - - if (fileName === "config.json") { - await this.updateSitemap(); - await this.updateRobots(); - await this.updateLlmsFiles(); - } - } catch (error) { - console.error(chalk.red(`āŒ Error removing ${fileName}:`), error); - } - } - } - - async handleFontConfigChange() { - console.log(chalk.cyan(`šŸ”¤ Font configuration changed`)); - - const sourcePath = path.join(this.rootDir, this.fontConfigFile); - const destPath = path.join(this.outputDir, this.fontConfigFile); - - try { - await fs.copy(sourcePath, destPath); - console.log( - chalk.green(`šŸ“‹ Updated ${this.fontConfigFile} in Next.js app`), - ); - - await this.updateRootLayout(); - console.log(chalk.green(`āœ… Layout updated with new font configuration`)); - } catch (error) { - console.error(chalk.red(`āŒ Error updating font configuration:`), error); - } - } - - async handleFontConfigDelete() { - console.log(chalk.red(`šŸ—‘ļø Font configuration deleted`)); - - const destPath = path.join(this.outputDir, this.fontConfigFile); - - try { - if (await fs.pathExists(destPath)) { - await fs.remove(destPath); - console.log( - chalk.yellow(`šŸ—‘ļø Removed ${this.fontConfigFile} from Next.js app`), - ); - - await this.updateRootLayout(); - console.log( - chalk.green(`āœ… Layout updated without font configuration`), - ); - } - } catch (error) { - console.error(chalk.red(`āŒ Error removing font configuration:`), error); - } - } - - async handleAnalyticsConfigChange() { - console.log(chalk.cyan(`šŸ“Š Analytics configuration changed`)); - - const sourcePath = path.join(this.rootDir, this.analyticsConfigFile); - const destPath = path.join(this.outputDir, this.analyticsConfigFile); - - try { - await fs.copy(sourcePath, destPath); - console.log( - chalk.green(`šŸ“‹ Updated ${this.analyticsConfigFile} in Next.js app`), - ); - - this.analyticsConfig = await this.loadAnalyticsConfig(); - - // Regenerate dynamic templates that depend on analytics config - await fs.writeFile( - path.join(this.outputDir, "next.config.ts"), - nextConfigTemplate(this.analyticsConfig), - "utf8", - ); - await fs.writeFile( - path.join(this.outputDir, "proxy.ts"), - proxyTemplate(this.analyticsConfig), - "utf8", - ); - await this.updateRootLayout(); - - console.log(chalk.green(`āœ… Analytics configuration updated`)); - - if (this.analyticsConfig) { - console.log( - chalk.yellow( - `āš ļø Next.js dev server restart may be required for analytics proxy changes`, - ), - ); - } - } catch (error) { - console.error( - chalk.red(`āŒ Error updating analytics configuration:`), - error, - ); - } - } - - async handleAnalyticsConfigDelete() { - console.log(chalk.red(`šŸ—‘ļø Analytics configuration deleted`)); - - const destPath = path.join(this.outputDir, this.analyticsConfigFile); - - try { - // Write empty analytics.json so runtime imports don't break - await fs.writeFile(destPath, `{}\n`, "utf8"); - - this.analyticsConfig = null; - - // Regenerate dynamic templates without analytics - await fs.writeFile( - path.join(this.outputDir, "next.config.ts"), - nextConfigTemplate(null), - "utf8", - ); - await fs.writeFile( - path.join(this.outputDir, "proxy.ts"), - proxyTemplate(null), - "utf8", - ); - await this.updateRootLayout(); - - console.log(chalk.green(`āœ… Analytics removed from Next.js app`)); - } catch (error) { - console.error( - chalk.red(`āŒ Error removing analytics configuration:`), - error, - ); - } - } - - async copyPublicFiles() { - const publicDir = path.join(this.rootDir, "public"); - const destDir = path.join(this.outputDir, "public"); - - console.log(chalk.blue(`šŸ” Checking for public directory...`)); - - if (await fs.pathExists(publicDir)) { - await fs.copy(publicDir, destDir); - console.log(chalk.green(` āœ“ Copied public directory to Next.js app`)); - } else { - console.log(chalk.gray(` āœ— public directory not found, skipping`)); - } - } - - async handlePublicFileChange(filePath: string) { - const publicDir = path.join(this.rootDir, "public"); - const relativePath = path.relative(publicDir, filePath); - const destPath = path.join(this.outputDir, "public", relativePath); - - try { - await fs.ensureDir(path.dirname(destPath)); - await fs.copy(filePath, destPath); - console.log( - chalk.green(`šŸ“‹ Updated public/${relativePath} in Next.js app`), - ); - } catch (error) { - console.error( - chalk.red(`āŒ Error copying public/${relativePath}:`), - error, - ); - } - } - - async handlePublicFileDelete(filePath: string) { - const publicDir = path.join(this.rootDir, "public"); - const relativePath = path.relative(publicDir, filePath); - const destPath = path.join(this.outputDir, "public", relativePath); - - try { - if (await fs.pathExists(destPath)) { - await fs.remove(destPath); - console.log( - chalk.yellow(`šŸ—‘ļø Removed public/${relativePath} from Next.js app`), - ); - } - } catch (error) { - console.error( - chalk.red(`āŒ Error removing public/${relativePath}:`), - error, - ); - } - } - - async startWatching() { - console.log(chalk.yellow(`šŸ‘€ Watching for changes in: ${this.watchDir}`)); - - this.watcher = chokidar.watch(this.watchDir, { - persistent: true, - ignoreInitial: true, - ignored: (filePath: string, stats?: fs.Stats) => { - const isFile = stats?.isFile() ?? path.extname(filePath) !== ""; - const fileName = path.basename(filePath); - - if (this.configFiles.includes(fileName)) { - return true; - } - - if (isFile && !filePath.endsWith(".mdx")) { - return true; - } - return false; - }, - }); - - this.watcher - .on("add", (filePath: string) => { - const relativePath = path.relative(this.watchDir, filePath); - this.handleFileChange("added", relativePath); - }) - .on("change", (filePath: string) => { - const relativePath = path.relative(this.watchDir, filePath); - this.handleFileChange("changed", relativePath); - }) - .on("unlink", (filePath: string) => { - const relativePath = path.relative(this.watchDir, filePath); - this.handleFileDelete(relativePath); - }) - .on("ready", () => { - console.log( - chalk.green("šŸ“ Initial scan complete. Ready for changes..."), - ); - }) - .on("error", (error: unknown) => { - console.error(chalk.red("āŒ Watcher error:"), error); - }); - - const configPaths = this.configFiles.map((f) => path.join(this.rootDir, f)); - - this.configWatcher = chokidar.watch(configPaths, { - persistent: true, - ignoreInitial: true, - }); - - this.configWatcher - .on("add", (filePath: string) => { - console.log( - chalk.cyan(`šŸ“ Config file added: ${path.basename(filePath)}`), - ); - this.handleConfigFileChange(filePath); - }) - .on("change", (filePath: string) => { - console.log( - chalk.cyan(`šŸ“ Config file changed: ${path.basename(filePath)}`), - ); - this.handleConfigFileChange(filePath); - }) - .on("unlink", (filePath: string) => { - console.log( - chalk.red(`šŸ—‘ļø Config file deleted: ${path.basename(filePath)}`), - ); - this.handleConfigFileDelete(filePath); - }) - .on("error", (error: unknown) => { - console.error(chalk.red("āŒ Config watcher error:"), error); - }); - - const fontPath = path.join(this.rootDir, this.fontConfigFile); - - this.fontWatcher = chokidar.watch(fontPath, { - persistent: true, - ignoreInitial: true, - }); - - this.fontWatcher - .on("add", () => { - console.log(chalk.cyan(`šŸ”¤ Font configuration added`)); - this.handleFontConfigChange(); - }) - .on("change", () => { - this.handleFontConfigChange(); - }) - .on("unlink", () => { - this.handleFontConfigDelete(); - }) - .on("error", (error: unknown) => { - console.error(chalk.red("āŒ Font watcher error:"), error); - }); - - const analyticsPath = path.join(this.rootDir, this.analyticsConfigFile); - - this.analyticsWatcher = chokidar.watch(analyticsPath, { - persistent: true, - ignoreInitial: true, - }); - - this.analyticsWatcher - .on("add", () => { - console.log(chalk.cyan(`šŸ“Š Analytics configuration added`)); - this.handleAnalyticsConfigChange(); - }) - .on("change", () => { - this.handleAnalyticsConfigChange(); - }) - .on("unlink", () => { - this.handleAnalyticsConfigDelete(); - }) - .on("error", (error: unknown) => { - console.error(chalk.red("āŒ Analytics watcher error:"), error); - }); - - await this.syncOpenApiSpecWatcher(); - - const doccupineConfigPath = path.join( - this.rootDir, - this.doccupineConfigFile, - ); - - this.doccupineConfigWatcher = chokidar.watch(doccupineConfigPath, { - persistent: true, - ignoreInitial: true, - }); - - this.doccupineConfigWatcher - .on("add", () => this.handleDoccupineConfigChange()) - .on("change", () => this.handleDoccupineConfigChange()) - .on("error", (error: unknown) => { - console.error(chalk.red("āŒ doccupine.json watcher error:"), error); - }); - - const publicDir = path.join(this.rootDir, "public"); - - if (await fs.pathExists(publicDir)) { - this.setupPublicWatcher(); - } - - // Watch rootDir for public directory creation - this.rootDirWatcher = chokidar.watch(this.rootDir, { - persistent: true, - ignoreInitial: true, - depth: 0, - }); - - this.rootDirWatcher - .on("addDir", async (dirPath: string) => { - if ( - path.basename(dirPath) === "public" && - path.dirname(dirPath) === this.rootDir && - !this.publicWatcher - ) { - console.log(chalk.cyan("šŸ“ Public directory created")); - await this.copyPublicFiles(); - this.setupPublicWatcher(); - } - }) - .on("error", (error: unknown) => { - console.error(chalk.red("āŒ Root dir watcher error:"), error); - }); - } - - private setupPublicWatcher() { - if (this.publicWatcher) { - return; - } - - const publicDir = path.join(this.rootDir, "public"); - - this.publicWatcher = chokidar.watch(publicDir, { - persistent: true, - ignoreInitial: true, - }); - - this.publicWatcher - .on("add", (filePath: string) => { - console.log( - chalk.cyan( - `šŸ“ Public file added: ${path.relative(publicDir, filePath)}`, - ), - ); - this.handlePublicFileChange(filePath); - }) - .on("change", (filePath: string) => { - console.log( - chalk.cyan( - `šŸ“ Public file changed: ${path.relative(publicDir, filePath)}`, - ), - ); - this.handlePublicFileChange(filePath); - }) - .on("unlink", (filePath: string) => { - console.log( - chalk.red( - `šŸ—‘ļø Public file deleted: ${path.relative(publicDir, filePath)}`, - ), - ); - this.handlePublicFileDelete(filePath); - }) - .on("error", (error: unknown) => { - console.error(chalk.red("āŒ Public watcher error:"), error); - }); - } - - private async parseMDXFile(file: string): Promise { - const fullPath = path.join(this.watchDir, file); - const content = await fs.readFile(fullPath, "utf8"); - const { data: frontmatter } = safeMatter(content, file); - - const { sectionSlug, pageSlug } = this.determineSectionForFile( - file, - frontmatter, - ); - const fullSlug = getFullSlug(pageSlug, sectionSlug); - - let lastModified: string | undefined; - if (frontmatter.date) { - const parsed = new Date(frontmatter.date); - if (!Number.isNaN(parsed.getTime())) { - lastModified = parsed.toISOString(); - } - } - if (!lastModified) { - try { - const stats = await fs.stat(fullPath); - lastModified = stats.mtime.toISOString(); - } catch { - // ignore - } - } - - // A hand-written page that embeds an endpoint via `openapi:` frontmatter - // gets the same method badge in the sidebar as a generated endpoint page. - let httpMethod: string | undefined; - if (frontmatter.openapi) { - const op = this.apiRegistry.lookup(String(frontmatter.openapi)); - if (op) httpMethod = op.method.toUpperCase(); - } - - return { - slug: fullSlug, - title: frontmatter.title || "Untitled", - description: frontmatter.description || "", - date: frontmatter.date || null, - category: frontmatter.category || "", - path: file, - categoryOrder: frontmatter.categoryOrder || 0, - order: frontmatter.order || 0, - section: sectionSlug, - // Sidebar icons (Lucide names). Kept separate from `icon`, which is - // reserved for the favicon/OG metadata. Only emitted when set so the - // generated page literal stays lean. - ...(frontmatter.navIcon ? { navIcon: String(frontmatter.navIcon) } : {}), - ...(frontmatter.categoryIcon - ? { categoryIcon: String(frontmatter.categoryIcon) } - : {}), - ...(httpMethod ? { httpMethod } : {}), - lastModified, - }; - } - - private async buildAllPagesMeta(): Promise { - const files = await this.getAllMDXFiles(); - const real = await Promise.all( - files.map((file) => this.parseMDXFile(file)), - ); - if (this.apiRegistry.isEmpty) return real; - - // Inject synthetic OpenAPI endpoint pages here - the single funnel every - // aggregate (nav, sitemap, llms) flows through - so they cannot be dropped - // by the .mdx-only disk scan. Hand-written pages win on any slug collision. - const realSlugs = new Set(real.map((page) => page.slug)); - const synthetic = this.apiRegistry.syntheticPages().filter((page) => { - if (realSlugs.has(page.slug)) { - console.log( - chalk.yellow( - `āš ļø API page ${page.slug} is shadowed by a hand-written page; skipping`, - ), - ); - return false; - } - return true; - }); - return [...real, ...synthetic]; - } - - /** - * Writes the generated page(s) for a single MDX file: the doc page and, for a - * section-index file, the section landing page. Deliberately does NOT run the - * site-wide aggregations (pages index, layout, sitemap, llms, section - * redirects) - the caller batches those so a bulk build runs them once at the - * end instead of once per file (which is what made large builds O(nĀ²)). - */ - private async writePageForFile(filePath: string): Promise { - const fullPath = path.join(this.watchDir, filePath); - const content = await fs.readFile(fullPath, "utf8"); - const { data: frontmatter, content: mdxContent } = safeMatter( - content, - filePath, - ); - - const { sectionSlug, pageSlug } = this.determineSectionForFile( - filePath, - frontmatter, - ); - const fullSlug = getFullSlug(pageSlug, sectionSlug); - - const isIndex = filePath === "index.mdx" || filePath === "./index.mdx"; - const isSectionIndex = - this.sectionsConfig && pageSlug === "" && sectionSlug !== ""; - - if (isIndex) { - // The homepage is emitted by updatePagesIndex() in the aggregate pass, so - // there is no per-file page to write here. - console.log(chalk.blue("šŸ  Updating homepage with index.mdx content")); - } else { - const mdxFile: MDXFile = { - path: filePath, - content: mdxContent, - frontmatter, - slug: fullSlug, - }; - - // `openapi: ` (or an operationId) in frontmatter renders - // that operation's playground inline with the author's prose. An unknown - // reference is logged and the page still renders its prose (graceful). - let apiOperation: OperationDescriptor | undefined; - if (frontmatter.openapi) { - apiOperation = this.apiRegistry.lookup(String(frontmatter.openapi)); - if (!apiOperation) { - console.error( - chalk.red( - `āŒ openapi frontmatter "${frontmatter.openapi}" in ${filePath} not found in any spec`, - ), - ); - } - } - - await this.generatePageFromMDX( - mdxFile, - apiOperation ? { apiOperation } : undefined, - ); - } - - if (isSectionIndex) { - await this.updateSectionIndex( - sectionSlug, - frontmatter, - mdxContent, - filePath, - ); - } - } - - /** - * Regenerates every file that depends on the full set of pages (pages index, - * root/site layout, sitemap, llms files, section redirects). Parses all MDX - * exactly once and threads the result through each generator, so one refresh - * is a single scan rather than one scan per generator. - */ - private async refreshSiteAggregates(): Promise { - const pages = await this.buildAllPagesMeta(); - await this.updatePagesIndex(); - await this.updateRootLayout(pages); - await this.updateSitemap(pages); - await this.updateLlmsFiles(pages); - await this.generateSectionIndexPages(pages); - } - - async handleFileChange(action: string, filePath: string) { - console.log(chalk.cyan(`šŸ“ File ${action}: ${filePath}`)); - - try { - await this.writePageForFile(filePath); - await this.refreshSiteAggregates(); - - console.log(chalk.green(`āœ… Generated page for: ${filePath}`)); - - await this.maybeUpdateSections(); - } catch (error) { - console.error(chalk.red(`āŒ Error processing ${filePath}:`), error); - } - } - - async handleFileDelete(filePath: string) { - console.log(chalk.red(`šŸ—‘ļø File deleted: ${filePath}`)); - - try { - if (filePath === "index.mdx" || filePath === "./index.mdx") { - console.log(chalk.blue("šŸ  Updating homepage - index.mdx deleted")); - } else { - // We don't have frontmatter for deleted files, so use directory-based matching - const { sectionSlug, pageSlug } = this.determineSectionForFile( - filePath, - {}, - ); - const fullSlug = getFullSlug(pageSlug, sectionSlug); - const pagePath = path.join(this.outputDir, "app", "(site)", fullSlug); - await fs.remove(pagePath); - } - - await this.updatePagesIndex(); - await this.updateRootLayout(); - await this.updateSitemap(); - await this.updateLlmsFiles(); - - console.log(chalk.green(`āœ… Removed page for: ${filePath}`)); - - await this.maybeUpdateSections(); - } catch (error) { - console.error( - chalk.red(`āŒ Error removing page for ${filePath}:`), - error, - ); - } - } - - async processAllMDXFiles() { - const files = await this.getAllMDXFiles(); - - // Write each page first (the only genuinely per-file work), then run the - // site-wide aggregations a single time. Doing the aggregations per file - // re-scanned and re-parsed every MDX file on each iteration, which made a - // full build O(nĀ²); batching them makes it O(n). A single bad file is - // logged and skipped so it never aborts the whole build. - for (const file of files) { - console.log(chalk.cyan(`šŸ“ Processing: ${file}`)); - try { - await this.writePageForFile(file); - } catch (error) { - console.error(chalk.red(`āŒ Error processing ${file}:`), error); - } - } - - await this.refreshSiteAggregates(); - } - - async getAllMDXFiles(): Promise { - const files: string[] = []; - - async function scanDir(dir: string, relativePath = "") { - const entries = await fs.readdir(dir, { withFileTypes: true }); - - for (const entry of entries) { - const fullPath = path.join(dir, entry.name); - const relPath = path.join(relativePath, entry.name); - - if (entry.isDirectory()) { - await scanDir(fullPath, relPath); - } else if (entry.name.endsWith(".mdx")) { - files.push(relPath); - } - } - } - - await scanDir(this.watchDir); - return files; - } - - async generateRootLayout(): Promise { - const fontConfig = await this.loadFontConfig(); - const analyticsEnabled = this.analyticsConfig !== null; - return rootLayoutTemplate(fontConfig, analyticsEnabled); - } - - async generateSiteLayout(pages?: PageMeta[]): Promise { - const resolvedPages = pages ?? (await this.buildAllPagesMeta()); - return siteLayoutTemplate(resolvedPages, this.sectionsConfig); - } - - async generateSectionIndexPages(pages?: PageMeta[]) { - const nextSlugs = new Set(); - - if (this.sectionsConfig && this.sectionsConfig.length > 0) { - const resolvedPages = pages ?? (await this.buildAllPagesMeta()); - - for (const section of this.sectionsConfig) { - if (section.slug === "") continue; - - // Check if a page already exists at the section root - const hasIndex = resolvedPages.some((p) => p.slug === section.slug); - if (hasIndex) continue; - - // Find the first page in this section - const sectionPages = resolvedPages - .filter((p) => p.section === section.slug) - .sort((a, b) => { - if (a.categoryOrder !== b.categoryOrder) - return a.categoryOrder - b.categoryOrder; - return a.order - b.order; - }); - - if (sectionPages.length === 0) continue; - - const firstPage = sectionPages[0]; - const redirectContent = `import { redirect } from "next/navigation"; - -export default function SectionIndex() { - redirect("/${firstPage.slug}"); -} -`; - - const pagePath = path.join( - this.outputDir, - "app", - "(site)", - section.slug, - "page.tsx", - ); - await fs.ensureDir(path.dirname(pagePath)); - await fs.writeFile(pagePath, redirectContent, "utf8"); - nextSlugs.add(section.slug); - console.log( - chalk.blue( - `šŸ”€ Generated section index redirect: /${section.slug} -> /${firstPage.slug}`, - ), - ); - } - } - - await this.cleanupStaleSectionIndexPages(nextSlugs); - } - - /** - * Removes section index redirects written earlier in this session whose - * section has since disappeared (e.g. the API Reference section after the - * `openapi` config is removed mid-watch). Only files that still contain the - * generated redirect are deleted, so a hand-written page that has taken - * over the slug is never touched. Fresh processes start clean anyway - - * init() wipes app/ - so in-session tracking is enough. - */ - private async cleanupStaleSectionIndexPages( - nextSlugs: Set, - ): Promise { - for (const stale of this.generatedSectionIndexSlugs) { - if (nextSlugs.has(stale)) continue; - const pagePath = path.join( - this.outputDir, - "app", - "(site)", - stale, - "page.tsx", - ); - try { - if (!(await fs.pathExists(pagePath))) continue; - const content = await fs.readFile(pagePath, "utf8"); - if (!content.includes("function SectionIndex()")) continue; - await fs.remove(pagePath); - await this.removeEmptyDirsUpTo( - path.dirname(pagePath), - path.join(this.outputDir, "app", "(site)"), - ); - console.log( - chalk.blue(`šŸ§¹ Removed stale section index redirect: /${stale}`), - ); - } catch { - // ignore - } - } - this.generatedSectionIndexSlugs = nextSlugs; - } - - /** Best-effort removal of now-empty directories up to (not incl.) stopDir. */ - private async removeEmptyDirsUpTo( - dir: string, - stopDir: string, - ): Promise { - const stop = path.resolve(stopDir); - let current = path.resolve(dir); - while (current !== stop && current.startsWith(stop + path.sep)) { - try { - const entries = await fs.readdir(current); - if (entries.length > 0) return; - await fs.remove(current); - } catch { - return; - } - current = path.dirname(current); - } - } - - async generatePageFromMDX( - mdxFile: MDXFile, - options?: { apiOperation?: OperationDescriptor }, - ) { - const fm = mdxFile.frontmatter; - const apiOperation = options?.apiOperation; - - // Pages containing blocks publish a subscribable changelog: an - // RSS feed at {page-url}/rss.xml. Synthetic OpenAPI pages never contain - // Update blocks, so skip the parse for them. - const isSynthetic = mdxFile.path.startsWith("@openapi/"); - const updates = isSynthetic ? [] : parseUpdateBlocks(mdxFile.content); - const hasFeed = updates.length > 0; - const feedPath = `/${mdxFile.slug}/rss.xml`; - - const metadataBlock = generateMetadataBlock({ - title: fm.title, - titleFallback: "Generated with Doccupine", - name: fm.name, - titleOrder: "page-first", - description: fm.description, - icon: fm.icon, - image: fm.image, - canonicalPath: mdxFile.slug, - rssPath: hasFeed ? feedPath : undefined, - }); - - const jsonLd = generateJsonLdScript({ - kind: "article", - canonicalPath: mdxFile.slug, - title: fm.title, - description: fm.description, - date: typeof fm.date === "string" ? fm.date : undefined, - updated: - typeof fm.updated === "string" - ? fm.updated - : typeof fm.date === "string" - ? fm.date - : undefined, - image: fm.image, - }); - - // For an OpenAPI-backed page, embed the operation descriptor as a JS string - // literal parsed at load. Serializing to JSON then re-`JSON.parse`ing is - // total escaping for arbitrary JSON - unlike `escapeTemplateContent`, which - // only guards backticks/`${`/backslashes for the MDX prose literal. - const apiImport = apiOperation - ? `\nimport { ApiPlayground } from "@/components/layout/ApiPlayground";` - : ""; - // The descriptor JSON always exceeds the 80-col print width, so emit the - // call pre-wrapped in the exact shape Prettier produces (argument on its own - // line with a trailing comma, single-quoted so the JSON's own double quotes - // need no escaping). Keeps generated endpoint pages Prettier-stable without - // running a formatter at build time. - const apiConst = apiOperation - ? (() => { - const arg = toJsStringLiteral(JSON.stringify(apiOperation)); - const inline = `const operation = JSON.parse(${arg});`; - const decl = - inline.length <= 80 - ? inline - : `const operation = JSON.parse(\n ${arg},\n);`; - return `\n${decl}\n`; - })() - : ""; - // The playground renders as a child of so it sits inside the docs - // content column (a sibling would escape the layout and overlap the nav). - // Synthetic endpoint pages pass no `sourcePath`: it only namespaces Mermaid - // diagrams (which endpoint docs never contain), and its long `@openapi/...` - // value would push the opening tag past 80 cols and make Prettier rewrap it. - const sourcePathLiteral = JSON.stringify(mdxFile.path); - // `rss: true` frontmatter opts the page into an RSS button in the action - // bar (only when a feed actually exists). The playground branch keeps its - // fixed JSX shape - a feed on an inline-playground page stays reachable - // via autodiscovery. The buttoned form usually exceeds the 80-col print - // width, so pre-wrap it in the shape Prettier produces (attributes on - // their own lines) relative to its 6-space insertion indent. - const showRssButton = hasFeed && fm.rss === true && !apiOperation; - const docsAttrs = [ - `content={content}`, - `sourcePath={${sourcePathLiteral}}`, - ...(showRssButton ? [`rssHref={${JSON.stringify(feedPath)}}`] : []), - ]; - const inlineDocs = ``; - const docsElement = apiOperation - ? ` - - ` - : inlineDocs.length + 6 <= 80 - ? inlineDocs - : ` ` ${attr}`).join("\n")}\n />`; - - const pageContent = `import { Metadata } from "next"; -import { Docs } from "@/components/Docs"; -import { config } from "@/utils/config";${apiImport} - -const content = \`${escapeTemplateContent(mdxFile.content)}\`; -${apiConst} -${metadataBlock} - -// Doc pages have no per-request data: theme resolves client-side via the -// "dark" class on (set before paint by the theme-init blocking -// script). Static rendering lets every response come from the edge cache. -export const dynamic = "force-static"; -export const revalidate = false; - -export default function Page() { - ${jsonLd.declarations} - - return ( - <> - ${jsonLd.element} - ${docsElement} - - ); -} -`; - - const pagePath = path.join( - this.outputDir, - "app", - "(site)", - mdxFile.slug, - "page.tsx", - ); - await fs.ensureDir(path.dirname(pagePath)); - await fs.writeFile(pagePath, pageContent, "utf8"); - - // The feed route lives inside the page's directory, so a deleted page - // takes its feed along (handleFileDelete removes the whole dir) and the - // else-branch prunes the route when a regenerated page no longer has - // Update blocks. Cross-run staleness is covered by the app/ wipe in - // createNextJSStructure. - if (!isSynthetic) { - const rssDir = path.join(path.dirname(pagePath), "rss.xml"); - if (hasFeed) { - await fs.ensureDir(rssDir); - await fs.writeFile( - path.join(rssDir, "route.ts"), - rssRouteTemplate({ - pagePath: mdxFile.slug, - title: typeof fm.title === "string" ? fm.title : null, - description: - typeof fm.description === "string" ? fm.description : null, - items: updates.map((update) => ({ - title: update.label, - anchor: update.anchor, - description: update.description, - })), - }), - "utf8", - ); - } else { - await fs.remove(rssDir); - } - } - } - - /** Parses the configured OpenAPI spec(s) into the shared registry. */ - private async loadOpenApiRegistry(): Promise { - if (this.openApiSpecs.length === 0) return; - await this.apiRegistry.load( - this.openApiSpecs, - this.rootDir, - this.apiBaseSlug, - ); - if (!this.apiRegistry.isEmpty) { - console.log( - chalk.blue( - `šŸ“˜ Loaded ${this.apiRegistry.all.length} API endpoint(s) from ${this.openApiSpecs.length} spec(s)`, - ), - ); - } - } - - /** - * Generates one page per OpenAPI operation, (re)writes the request-execution - * allowlist consumed by the playground proxy + component, and removes endpoint - * pages that no longer exist in the spec. Safe to call when there are no specs - * - it still emits an empty allowlist and prunes any previously generated - * pages (e.g. after the `openapi` config is removed). - */ - private async writeApiPages(): Promise { - const nextSlugs = new Set(); - - for (const op of this.apiRegistry.all) { - const methodUpper = op.method.toUpperCase(); - const mdxFile: MDXFile = { - path: `@openapi/${op.specName}/${op.method}${op.path}`, - content: buildEndpointDoc(op), - frontmatter: { - title: op.summary ?? `${methodUpper} ${op.path}`, - description: op.summary ?? "", - }, - slug: op.slug, - }; - try { - await this.generatePageFromMDX(mdxFile, { apiOperation: op }); - nextSlugs.add(op.slug); - } catch (error) { - console.error( - chalk.red(`āŒ Error generating API page ${op.slug}:`), - error, - ); - } - } - - await this.writeApiAllowlist(); - await this.cleanupStaleApiPages(nextSlugs); - - if (nextSlugs.size > 0) { - console.log( - chalk.green(`šŸ§© Generated ${nextSlugs.size} API reference page(s)`), - ); - } - } - - /** Writes the request-execution allowlist (overwrites the shipped stub). */ - private async writeApiAllowlist(): Promise { - const target = path.join( - this.outputDir, - "services", - "openapi", - "playground-allowlist.json", - ); - await fs.ensureDir(path.dirname(target)); - await fs.writeFile( - target, - `${JSON.stringify(this.apiRegistry.allowlist(), null, 2)}\n`, - "utf8", - ); - } - - private apiManifestPath(): string { - return path.join(this.outputDir, ".doccupine-api-manifest.json"); - } - - private async readApiManifest(): Promise> { - try { - const manifestPath = this.apiManifestPath(); - if (await fs.pathExists(manifestPath)) { - const raw = await fs.readFile(manifestPath, "utf8"); - const parsed = JSON.parse(raw) as { pageSlugs?: unknown }; - if (Array.isArray(parsed.pageSlugs)) { - return new Set( - parsed.pageSlugs.filter( - (entry): entry is string => typeof entry === "string", - ), - ); - } - } - } catch { - // ignore corrupted manifest - } - return new Set(); - } - - private async writeApiManifest(slugs: Set): Promise { - const payload = { pageSlugs: Array.from(slugs).sort() }; - await fs.writeFile( - this.apiManifestPath(), - `${JSON.stringify(payload, null, 2)}\n`, - "utf8", - ); - } - - /** Removes endpoint page directories that are no longer in the spec. */ - private async cleanupStaleApiPages(nextSlugs: Set): Promise { - const previous = new Set([ - ...this.generatedApiPageSlugs, - ...(await this.readApiManifest()), - ]); - - for (const stale of previous) { - if (nextSlugs.has(stale)) continue; - try { - const dir = path.join(this.outputDir, "app", "(site)", stale); - if (await fs.pathExists(dir)) { - await fs.remove(dir); - await this.removeEmptyDirsUpTo( - path.dirname(dir), - path.join(this.outputDir, "app", "(site)"), - ); - } - } catch { - // ignore - } - } - - this.generatedApiPageSlugs = nextSlugs; - await this.writeApiManifest(nextSlugs); - } - - /** - * (Re)points the spec-file watcher at the currently configured spec paths. - * Called at startup and whenever doccupine.json changes the `openapi` set, - * so specs added mid-session are watched without a restart. - */ - private async syncOpenApiSpecWatcher(): Promise { - if (this.openApiWatcher) { - await this.openApiWatcher.close(); - this.openApiWatcher = null; - } - if (this.openApiSpecs.length === 0) return; - - const specPaths = this.openApiSpecs.map((spec) => - path.resolve(this.rootDir, spec.file), - ); - - this.openApiWatcher = chokidar.watch(specPaths, { - persistent: true, - ignoreInitial: true, - }); - - this.openApiWatcher - .on("add", () => this.handleOpenApiChange()) - .on("change", () => this.handleOpenApiChange()) - .on("unlink", () => this.handleOpenApiChange()) - .on("error", (error: unknown) => { - console.error(chalk.red("āŒ OpenAPI watcher error:"), error); - }); - } - - /** - * Reparses the spec(s) and rewrites everything derived from them: the - * endpoint pages and allowlist, the sections (the "API Reference" section - * appears and disappears with the registry), and the site aggregates. - */ - private async rebuildApiReference(): Promise { - await this.apiRegistry.load( - this.openApiSpecs, - this.rootDir, - this.apiBaseSlug, - ); - this.sectionsConfig = await this.resolveSections(); - await this.writeApiPages(); - await this.refreshSiteAggregates(); - } - - /** Reparses the spec(s) and regenerates the API reference on a watch event. */ - async handleOpenApiChange() { - console.log( - chalk.cyan("šŸ“˜ OpenAPI spec changed - regenerating API reference"), - ); - try { - await this.rebuildApiReference(); - console.log(chalk.green("āœ… API reference updated")); - } catch (error) { - console.error(chalk.red("āŒ Error updating API reference:"), error); - } - } - - /** - * Applies `openapi` edits in doccupine.json without a restart: reloads the - * registry, regenerates or prunes the endpoint pages and allowlist, - * refreshes nav/sitemap/llms, and re-points the spec-file watcher. Other - * fields (watchDir, outputDir, port) cannot be hot-applied, so a change - * there only logs a restart hint. Invalid or missing JSON (e.g. a - * half-written editor save) keeps the current configuration. - */ - async handleDoccupineConfigChange() { - const configPath = path.join(this.rootDir, this.doccupineConfigFile); - let config: DoccupineConfig; - try { - config = JSON.parse( - await fs.readFile(configPath, "utf8"), - ) as DoccupineConfig; - } catch { - console.warn( - chalk.yellow( - "āš ļø doccupine.json is missing or invalid - keeping the current configuration", - ), - ); - return; - } - - if ( - (config.watchDir && - path.resolve(this.rootDir, config.watchDir) !== this.watchDir) || - (config.outputDir && - path.resolve(this.rootDir, config.outputDir) !== this.outputDir) - ) { - console.log( - chalk.yellow( - "āš ļø watchDir/outputDir changes in doccupine.json need a restart to apply", - ), - ); - } - - const nextSpecs = normalizeOpenApiConfig(config.openapi); - if (JSON.stringify(nextSpecs) === JSON.stringify(this.openApiSpecs)) { - return; - } - - console.log( - chalk.cyan("šŸ“˜ OpenAPI configuration changed - updating API reference"), - ); - this.openApiSpecs = nextSpecs; - try { - await this.syncOpenApiSpecWatcher(); - await this.rebuildApiReference(); - console.log(chalk.green("āœ… API reference updated")); - } catch (error) { - console.error(chalk.red("āŒ Error updating API reference:"), error); - } - } - - async updatePagesIndex() { - const files = await this.getAllMDXFiles(); - let indexMDX: { - content: string; - title: string; - description: string; - icon?: string; - image?: string; - name?: string; - date?: string; - updated?: string; - openapi?: string; - rss?: boolean; - } | null = null; - - for (const file of files) { - if (file === "index.mdx" || file === "./index.mdx") { - const fullPath = path.join(this.watchDir, file); - const content = await fs.readFile(fullPath, "utf8"); - const { data: frontmatter, content: mdxContent } = safeMatter( - content, - file, - ); - - indexMDX = { - content: mdxContent, - title: frontmatter.title || "Welcome", - description: frontmatter.description || "", - icon: frontmatter.icon, - image: frontmatter.image, - name: frontmatter.name, - date: - typeof frontmatter.date === "string" ? frontmatter.date : undefined, - updated: - typeof frontmatter.updated === "string" - ? frontmatter.updated - : undefined, - openapi: - typeof frontmatter.openapi === "string" - ? frontmatter.openapi - : undefined, - rss: frontmatter.rss === true, - }; - break; - } - } - - // The homepage publishes the same subscribable changelog as any other - // page (see generatePageFromMDX): blocks feed the site-root - // /rss.xml, and `rss: true` frontmatter opts into the RSS button. - const updates = indexMDX ? parseUpdateBlocks(indexMDX.content) : []; - const hasFeed = updates.length > 0; - const feedPath = "/rss.xml"; - - const metadataBlock = indexMDX - ? generateMetadataBlock({ - title: indexMDX.title, - titleFallback: "Welcome", - name: indexMDX.name, - titleOrder: "name-first", - description: indexMDX.description || undefined, - icon: indexMDX.icon, - image: indexMDX.image, - canonicalPath: "", - rssPath: hasFeed ? feedPath : undefined, - }) - : generateRuntimeOnlyMetadataBlock(); - - const homeJsonLd = generateJsonLdScript({ - kind: "homepage", - canonicalPath: "", - title: indexMDX?.title, - description: indexMDX?.description || undefined, - date: indexMDX?.date, - updated: indexMDX?.updated ?? indexMDX?.date, - image: indexMDX?.image, - }); - - // The homepage supports the same `openapi: ` frontmatter as - // any other page: look the operation up and embed its playground inline. - let apiOperation: OperationDescriptor | undefined; - if (indexMDX?.openapi) { - apiOperation = this.apiRegistry.lookup(indexMDX.openapi); - if (!apiOperation) { - console.error( - chalk.red( - `āŒ openapi frontmatter "${indexMDX.openapi}" in index.mdx not found in any spec`, - ), - ); - } - } - const apiImport = apiOperation - ? `\nimport { ApiPlayground } from "@/components/layout/ApiPlayground";` - : ""; - const apiConst = apiOperation - ? `\nconst operation = JSON.parse(${JSON.stringify( - JSON.stringify(apiOperation), - )});\n` - : ""; - // Same gating as generatePageFromMDX: the playground branch keeps its - // fixed JSX shape, so a feed on a playground homepage stays reachable via - // autodiscovery. The buttoned inline form stays within the 80-col print - // width at its 6-space insertion indent, so it is Prettier-stable as is. - const showRssButton = hasFeed && indexMDX?.rss === true && !apiOperation; - const docsElement = apiOperation - ? ` - - ` - : showRssButton - ? `` - : ``; - - const indexContent = `import { Metadata } from "next"; -import { Docs } from "@/components/Docs"; -import { config } from "@/utils/config";${apiImport} - -${indexMDX ? `const content = \`${escapeTemplateContent(indexMDX.content)}\`;` : `const content = null;`} -${apiConst} -${metadataBlock} - -export const dynamic = "force-static"; -export const revalidate = false; - -export default function Home() { - ${homeJsonLd.declarations} - - return ( - <> - ${homeJsonLd.element} - ${docsElement} - - ); -} -`; - - const homePath = path.join(this.outputDir, "app", "(site)", "page.tsx"); - await fs.ensureDir(path.dirname(homePath)); - await fs.writeFile(homePath, indexContent, "utf8"); - - // Same lifecycle as the per-page feeds in generatePageFromMDX: write the - // root feed route while the homepage has Update blocks, prune it when - // they go away or index.mdx is deleted (this runs on every aggregate - // refresh, including the delete path). - const rssDir = path.join(this.outputDir, "app", "(site)", "rss.xml"); - if (hasFeed && indexMDX) { - await fs.ensureDir(rssDir); - await fs.writeFile( - path.join(rssDir, "route.ts"), - rssRouteTemplate({ - pagePath: "", - title: indexMDX.title, - description: indexMDX.description || null, - items: updates.map((update) => ({ - title: update.label, - anchor: update.anchor, - description: update.description, - })), - }), - "utf8", - ); - } else { - await fs.remove(rssDir); - } - } - - async updateSectionIndex( - sectionSlug: string, - frontmatter: Record, - mdxContent: string, - sourcePath?: string, - ) { - // This overwrites the page generatePageFromMDX just wrote for the same - // slug (section landings compose their metadata name-first), so the RSS - // state must be re-derived here or the overwrite silently drops the - // button and autodiscovery - the feed route survives either way since it - // lives in a sibling rss.xml/ dir. - const updates = parseUpdateBlocks(mdxContent); - const hasFeed = updates.length > 0; - const feedPath = `/${sectionSlug}/rss.xml`; - const showRssButton = hasFeed && frontmatter.rss === true; - - const metadataBlock = generateMetadataBlock({ - title: frontmatter.title, - titleFallback: "Section", - name: frontmatter.name, - titleOrder: "name-first", - description: frontmatter.description || undefined, - icon: frontmatter.icon, - image: frontmatter.image, - canonicalPath: sectionSlug, - rssPath: hasFeed ? feedPath : undefined, - }); - - const sectionJsonLd = generateJsonLdScript({ - kind: "article", - canonicalPath: sectionSlug, - title: frontmatter.title, - description: frontmatter.description, - date: typeof frontmatter.date === "string" ? frontmatter.date : undefined, - updated: - typeof frontmatter.updated === "string" - ? frontmatter.updated - : typeof frontmatter.date === "string" - ? frontmatter.date - : undefined, - image: frontmatter.image, - }); - - // Same Prettier pre-wrap contract as generatePageFromMDX: the buttoned - // form usually pushes the line past the 80-col print width, so emit it - // with attributes on their own lines relative to the 6-space indent. - const docsAttrs = [ - `content={content}`, - `sourcePath={${JSON.stringify(sourcePath ?? `${sectionSlug}/index.mdx`)}}`, - ...(showRssButton ? [`rssHref={${JSON.stringify(feedPath)}}`] : []), - ]; - const inlineDocs = ``; - const docsElement = - inlineDocs.length + 6 <= 80 - ? inlineDocs - : ` ` ${attr}`).join("\n")}\n />`; - - const indexContent = `import { Metadata } from "next"; -import { Docs } from "@/components/Docs"; -import { config } from "@/utils/config"; - -const content = \`${escapeTemplateContent(mdxContent)}\`; - -${metadataBlock} - -export const dynamic = "force-static"; -export const revalidate = false; - -export default function Page() { - ${sectionJsonLd.declarations} - - return ( - <> - ${sectionJsonLd.element} - ${docsElement} - - ); -} -`; - - const pagePath = path.join( - this.outputDir, - "app", - "(site)", - sectionSlug, - "page.tsx", - ); - await fs.ensureDir(path.dirname(pagePath)); - await fs.writeFile(pagePath, indexContent, "utf8"); - } - - async updateRootLayout(pages?: PageMeta[]) { - await fs.writeFile( - path.join(this.outputDir, "app", "layout.tsx"), - await this.generateRootLayout(), - "utf8", - ); - const siteLayoutPath = path.join( - this.outputDir, - "app", - "(site)", - "layout.tsx", - ); - await fs.ensureDir(path.dirname(siteLayoutPath)); - await fs.writeFile( - siteLayoutPath, - await this.generateSiteLayout(pages), - "utf8", - ); - } - - async loadSiteUrl(): Promise { - const configPath = path.join(this.rootDir, "config.json"); - - try { - if (await fs.pathExists(configPath)) { - const content = await fs.readFile(configPath, "utf8"); - const parsed = JSON.parse(content) as { url?: unknown }; - if (typeof parsed.url === "string" && parsed.url.trim() !== "") { - return parsed.url.trim().replace(/\/$/, ""); - } - } - } catch (error) { - console.warn(chalk.yellow("āš ļø Error reading config.json"), error); - } - - return null; - } - - private buildSitemapEntries(pages: PageMeta[]): SitemapEntry[] { - const sectionSlugs = new Set( - (this.sectionsConfig || []) - .map((s) => s.slug) - .filter((s): s is string => typeof s === "string" && s !== ""), - ); - - const entries: SitemapEntry[] = pages.map((page) => { - let priority = 0.5; - if (page.slug === "") { - priority = 1.0; - } else if (sectionSlugs.has(page.slug)) { - priority = 0.8; - } - return { - slug: page.slug, - lastModified: page.lastModified, - changeFrequency: "weekly", - priority, - }; - }); - - if (!entries.some((entry) => entry.slug === "")) { - entries.unshift({ - slug: "", - changeFrequency: "weekly", - priority: 1.0, - }); - } - - return entries; - } - - async updateSitemap(pages?: PageMeta[]) { - const sitemapPath = path.join(this.outputDir, "app", "sitemap.ts"); - const siteUrl = await this.loadSiteUrl(); - - if (!siteUrl) { - if (await fs.pathExists(sitemapPath)) { - await fs.remove(sitemapPath); - console.log( - chalk.yellow("šŸ—‘ļø Removed sitemap.ts (no site URL configured)"), - ); - } - return; - } - - const resolvedPages = pages ?? (await this.buildAllPagesMeta()); - const entries = this.buildSitemapEntries(resolvedPages); - await fs.writeFile(sitemapPath, sitemapTemplate(entries), "utf8"); - console.log( - chalk.green( - `šŸ—ŗļø Generated sitemap.ts with ${entries.length} page(s) using ${siteUrl}`, - ), - ); - } - - async updateRobots() { - const siteUrl = await this.loadSiteUrl(); - await fs.writeFile( - path.join(this.outputDir, "app", "robots.ts"), - robotsTemplate(siteUrl !== null), - "utf8", - ); - console.log( - chalk.green( - siteUrl - ? `šŸ¤– Regenerated robots.ts with sitemap link` - : `šŸ¤– Regenerated robots.ts (no sitemap link)`, - ), - ); - } - - private async loadSiteMetadata(): Promise<{ - url: string | null; - name: string; - description: string; - }> { - const configPath = path.join(this.rootDir, "config.json"); - let url: string | null = null; - let name = "Documentation"; - let description = ""; - - try { - if (await fs.pathExists(configPath)) { - const content = await fs.readFile(configPath, "utf8"); - const parsed = JSON.parse(content) as { - url?: unknown; - name?: unknown; - title?: unknown; - description?: unknown; - }; - if (typeof parsed.url === "string" && parsed.url.trim() !== "") { - url = parsed.url.trim().replace(/\/$/, ""); - } - if (typeof parsed.name === "string" && parsed.name.trim() !== "") { - name = parsed.name.trim(); - } else if ( - typeof parsed.title === "string" && - parsed.title.trim() !== "" - ) { - name = parsed.title.trim(); - } - if ( - typeof parsed.description === "string" && - parsed.description.trim() !== "" - ) { - description = parsed.description.trim(); - } - } - } catch (error) { - console.warn( - chalk.yellow("āš ļø Error reading config.json for llms metadata"), - error, - ); - } - - return { url, name, description }; - } - - private async readPageWithBody(page: PageMeta): Promise { - // Synthetic OpenAPI pages have no backing .mdx file; their markdown body - // comes from the registry instead of disk. - if (!page.path.endsWith(".mdx")) { - return { ...page, body: this.apiRegistry.bodyForSlug(page.slug) ?? "" }; - } - const fullPath = path.join(this.watchDir, page.path); - const raw = await fs.readFile(fullPath, "utf8"); - const { content: body } = safeMatter(raw, page.path); - return { ...page, body }; - } - - private llmsManifestPath(): string { - return path.join(this.outputDir, ".doccupine-llms-manifest.json"); - } - - private async readLlmsManifest(): Promise> { - const manifestPath = this.llmsManifestPath(); - try { - if (await fs.pathExists(manifestPath)) { - const raw = await fs.readFile(manifestPath, "utf8"); - const parsed = JSON.parse(raw) as { pageFiles?: unknown }; - if (Array.isArray(parsed.pageFiles)) { - return new Set( - parsed.pageFiles.filter( - (entry): entry is string => typeof entry === "string", - ), - ); - } - } - } catch { - // ignore corrupted manifest - } - return new Set(); - } - - private async writeLlmsManifest(pageFiles: Set): Promise { - const manifestPath = this.llmsManifestPath(); - const payload = { pageFiles: Array.from(pageFiles).sort() }; - const json = JSON.stringify(payload, null, 2) + "\n"; - await fs.writeFile(manifestPath, json, "utf8"); - } - - async updateLlmsFiles(pages?: PageMeta[]) { - const publicDir = path.join(this.outputDir, "public"); - await fs.ensureDir(publicDir); - - const { url: baseUrl, name, description } = await this.loadSiteMetadata(); - const resolvedPages = pages ?? (await this.buildAllPagesMeta()); - const pagesWithBodies = await Promise.all( - resolvedPages.map((page) => this.readPageWithBody(page)), - ); - - const indexContent = llmsIndexTemplate({ - siteName: name, - siteDescription: description, - baseUrl, - pages: resolvedPages, - sectionsConfig: this.sectionsConfig, - }); - const fullContent = llmsFullTemplate({ - siteName: name, - siteDescription: description, - baseUrl, - pages: pagesWithBodies, - sectionsConfig: this.sectionsConfig, - }); - - await fs.writeFile(path.join(publicDir, "llms.txt"), indexContent, "utf8"); - await fs.writeFile( - path.join(publicDir, "llms-full.txt"), - fullContent, - "utf8", - ); - - const skillContent = skillMdTemplate({ - siteName: name, - siteDescription: description, - baseUrl, - pages: resolvedPages, - sectionsConfig: this.sectionsConfig, - }); - await fs.writeFile(path.join(publicDir, "skill.md"), skillContent, "utf8"); - - // MCP discovery manifest. Needs an absolute URL, so it only exists when - // config.json declares the site url; it is pruned if the url is removed. - const mcpJsonPath = path.join(publicDir, ".well-known", "mcp.json"); - if (baseUrl) { - const mcpJson = - JSON.stringify( - { - mcpServers: { - [siteDocsSlug(name)]: { - url: `${baseUrl}/api/mcp`, - transport: "streamable-http", - }, - }, - }, - null, - 2, - ) + "\n"; - await fs.ensureDir(path.dirname(mcpJsonPath)); - await fs.writeFile(mcpJsonPath, mcpJson, "utf8"); - } else if (await fs.pathExists(mcpJsonPath)) { - await fs.remove(mcpJsonPath); - } - - const nextRelativePaths = new Set(); - await Promise.all( - pagesWithBodies.map(async (page) => { - const relPath = page.slug === "" ? "index.md" : `${page.slug}.md`; - const targetPath = path.join(publicDir, relPath); - await fs.ensureDir(path.dirname(targetPath)); - await fs.writeFile(targetPath, llmsPageTemplate(page, baseUrl), "utf8"); - nextRelativePaths.add(relPath); - }), - ); - - const previousRelativePaths = new Set([ - ...this.generatedLlmsPagePaths, - ...(await this.readLlmsManifest()), - ]); - - for (const stale of previousRelativePaths) { - if (!nextRelativePaths.has(stale)) { - try { - const stalePath = path.join(publicDir, stale); - if (await fs.pathExists(stalePath)) { - await fs.remove(stalePath); - } - } catch { - // ignore - } - } - } - this.generatedLlmsPagePaths = nextRelativePaths; - await this.writeLlmsManifest(nextRelativePaths); - - console.log( - chalk.green( - `šŸ¤– Generated llms.txt and llms-full.txt with ${resolvedPages.length} page(s)${ - baseUrl ? ` using ${baseUrl}` : " (relative URLs)" - }`, - ), - ); - } - - async stop() { - if (this.watcher) { - await this.watcher.close(); - console.log(chalk.yellow("šŸ‘‹ Stopped watching for MDX changes")); - } - if (this.configWatcher) { - await this.configWatcher.close(); - console.log(chalk.yellow("šŸ‘‹ Stopped watching for config changes")); - } - if (this.fontWatcher) { - await this.fontWatcher.close(); - console.log(chalk.yellow("šŸ‘‹ Stopped watching for font config changes")); - } - if (this.analyticsWatcher) { - await this.analyticsWatcher.close(); - console.log( - chalk.yellow("šŸ‘‹ Stopped watching for analytics config changes"), - ); - } - if (this.openApiWatcher) { - await this.openApiWatcher.close(); - console.log(chalk.yellow("šŸ‘‹ Stopped watching for OpenAPI spec changes")); - } - if (this.doccupineConfigWatcher) { - await this.doccupineConfigWatcher.close(); - console.log( - chalk.yellow("šŸ‘‹ Stopped watching for doccupine.json changes"), - ); - } - if (this.publicWatcher) { - await this.publicWatcher.close(); - console.log( - chalk.yellow("šŸ‘‹ Stopped watching for public directory changes"), - ); - } - if (this.rootDirWatcher) { - await this.rootDirWatcher.close(); - } - } -} - -program - .name("doccupine") - .description( - "Watch MDX files and generate Next.js documentation pages automatically", - ) - .version(version); - -program - .command("watch", { isDefault: true }) - .description("Watch a directory for MDX changes and generate Next.js app") - .option("--port ", "Port for Next.js dev server", "3000") - .option("--verbose", "Show verbose output") - .option("--reset", "Reset configuration and prompt for new directories") - .option( - "--package-manager ", - "Package manager for the generated app: pnpm or npm (default: auto-detect)", - ) - .action(async (options) => { - const configManager = new ConfigManager(); - const config = await configManager.getConfig({ - reset: options.reset, - port: options.port, - }); - - const generator = new MDXToNextJSGenerator( - config.watchDir, - config.outputDir, - normalizeOpenApiConfig(config.openapi), - ); - - // Config paths are project-relative; child processes get an absolute cwd. - const outputDir = path.resolve(process.cwd(), config.outputDir); - - await generator.init(); - - let devServer: ReturnType | null = null; - - console.log(chalk.blue("šŸ“¦ Installing dependencies...")); - const { spawn } = await import("child_process"); - - // Prefer pnpm (the generated app ships a pnpm workspace) and fall back to - // npm. A --package-manager flag or a "packageManager" field in - // doccupine.json overrides detection; the flag wins. - const packageManager = resolvePackageManager( - options.packageManager ?? config.packageManager, - ); - - console.log(chalk.blue(`šŸ“¦ Using ${packageManager.name}...`)); - - const install = spawn(packageManager.bin, ["install"], { - cwd: outputDir, - stdio: "inherit", - }); - - await new Promise((resolve, reject) => { - install.on("close", (code) => { - if (code === 0) { - console.log(chalk.green("āœ… Dependencies installed")); - resolve(void 0); - } else { - reject( - new Error( - `${packageManager.name} install failed with code ${code}`, - ), - ); - } - }); - install.on("error", reject); - }); - - const port = await findAvailablePort(parseInt(config.port, 10)); - if (port !== parseInt(config.port, 10)) { - console.log( - chalk.yellow( - `āš ļø Port ${config.port} is in use, using port ${port} instead`, - ), - ); - } - console.log( - chalk.blue(`šŸš€ Starting Next.js dev server on port ${port}...`), - ); - const portStr = String(port); - const devArgs = - packageManager.name === "npm" - ? ["run", "dev", "--", "--port", portStr] - : ["run", "dev", "--port", portStr]; - devServer = spawn(packageManager.bin, devArgs, { - cwd: outputDir, - stdio: ["ignore", "pipe", "pipe"], - }); - - devServer.stdout?.on("data", (data: Buffer) => { - const output = data.toString(); - if (output.includes("Ready") || output.includes("started")) { - console.log( - chalk.green(`šŸŒ Next.js ready at http://localhost:${port}`), - ); - } - if (options.verbose) { - process.stdout.write(chalk.gray("[Next.js] ") + output); - } else if ( - output.includes("compiled") || - output.includes("error") || - output.includes("Ready") - ) { - process.stdout.write(chalk.gray("[Next.js] ") + output); - } - }); - - devServer.stderr?.on("data", (data: Buffer) => { - const output = data.toString(); - if ( - options.verbose || - output.includes("Error") || - output.includes("error") - ) { - process.stderr.write(chalk.red("[Next.js] ") + output); - } - }); - - devServer.on("error", (error: Error) => { - console.error(chalk.red("āŒ Error starting dev server:"), error); - }); - - devServer.on("close", (code: number | null) => { - if (code && code !== 0) { - console.error( - chalk.red(`āŒ Next.js dev server exited with code ${code}`), - ); - } - }); - - await generator.startWatching(); - - process.on("SIGINT", async () => { - console.log(chalk.yellow("\nšŸ›‘ Shutting down...")); - await generator.stop(); - if (devServer) { - devServer.kill(); - } - process.exit(0); - }); - - console.log(chalk.green("šŸŽ‰ Generator is running! Press Ctrl+C to stop.")); - console.log(chalk.cyan(`šŸ“ Edit your MDX files in: ${config.watchDir}`)); - }); - -program - .command("build") - .description("One-time build of Next.js app from MDX files") - .option("--reset", "Reset configuration and prompt for new directories") - .action(async (options) => { - const configManager = new ConfigManager(); - const config = await configManager.getConfig({ - reset: options.reset, - }); - - const generator = new MDXToNextJSGenerator( - config.watchDir, - config.outputDir, - normalizeOpenApiConfig(config.openapi), - ); - await generator.init(); - console.log(chalk.green("šŸŽ‰ Build complete!")); - }); - -program - .command("config") - .description("Show or reset configuration") - .option("--show", "Show current configuration") - .option("--reset", "Reset configuration") - .action(async (options) => { - const configManager = new ConfigManager(); - - if (options.show) { - const config = await configManager.loadConfig(); - if (config) { - console.log(chalk.blue("šŸ“„ Current configuration:")); - console.log( - chalk.white("Watch Directory:"), - chalk.cyan(path.relative(process.cwd(), config.watchDir)), - ); - console.log( - chalk.white("Output Directory:"), - chalk.cyan(path.relative(process.cwd(), config.outputDir)), - ); - console.log(chalk.white("Port:"), chalk.cyan(config.port || "3000")); - } else { - console.log(chalk.yellow("āš ļø No configuration file found")); - } - } else if (options.reset) { - await configManager.getConfig({ reset: true }); - console.log(chalk.green("āœ… Configuration reset")); - } else { - console.log( - chalk.blue( - "Use --show to display configuration or --reset to reset it", - ), - ); - } - }); /** * True when this file is what node was asked to run, rather than a module some - * other process imported. The test suite imports this file for its exported - * helpers; without this guard, that import parses argv, falls through to the - * default `watch` command, and prompts for config / starts file watchers. - * - * Compares realpaths because npm installs the bin as a symlink - * (`node_modules/.bin/doccupine` -> `dist/index.js`), so `process.argv[1]` and - * `import.meta.url` can name the same file by different paths. + * other process imported. Compares realpaths because npm installs the bin as a + * symlink (`node_modules/.bin/doccupine` -> `dist/index.js`). */ export function isProcessEntrypoint( entry: string | undefined = process.argv[1], @@ -2577,5 +39,6 @@ export function isProcessEntrypoint( } if (isProcessEntrypoint()) { - program.parse(); + const { runCli } = await import("./cli.js"); + await runCli(); } diff --git a/src/lib/config-manager.test.ts b/src/lib/config-manager.test.ts index fc5bc73..7ef7c0b 100644 --- a/src/lib/config-manager.test.ts +++ b/src/lib/config-manager.test.ts @@ -7,10 +7,51 @@ import { ConfigManager, normalizeConfigPaths, toProjectRelativePath, + validateConfig, } from "./config-manager.js"; const ROOT = "/home/dev/project"; +describe("validateConfig", () => { + it("accepts distinct project directories and defaults the port", () => { + expect( + validateConfig({ watchDir: "docs", outputDir: "site" }, ROOT), + ).toMatchObject({ watchDir: "docs", outputDir: "site", port: "3000" }); + }); + + it("rejects the project root and overlapping directories", () => { + expect(() => + validateConfig({ watchDir: "docs", outputDir: "." }, ROOT), + ).toThrow("project root"); + expect(() => + validateConfig( + { watchDir: ".", outputDir: "generated", port: "3000" }, + ROOT, + ), + ).toThrow("must not overlap"); + }); + + it("rejects invalid ports and OpenAPI shapes", () => { + expect(() => + validateConfig( + { watchDir: "docs", outputDir: "site", port: "70000" }, + ROOT, + ), + ).toThrow("1 to 65535"); + expect(() => + validateConfig( + { + watchDir: "docs", + outputDir: "site", + port: "3000", + openapi: [{ name: "Missing file" }] as never, + }, + ROOT, + ), + ).toThrow("openapi"); + }); +}); + describe("toProjectRelativePath", () => { it("rewrites an absolute path inside the root as relative", () => { expect(toProjectRelativePath(`${ROOT}/docs`, ROOT)).toBe("docs"); @@ -194,4 +235,16 @@ describe("ConfigManager migration", () => { expect(config.port).toBe("4000"); expect(config.packageManager).toBe("npm"); }); + + it("points invalid existing configurations to the reset command", async () => { + await fs.writeJSON(path.join(tempDir, "doccupine.json"), { + watchDir: ".", + outputDir: "nextjs-app", + port: "3000", + }); + + await expect(new ConfigManager().loadConfig()).rejects.toThrow( + 'Run "doccupine config --reset" to repair the configuration.', + ); + }); }); diff --git a/src/lib/config-manager.ts b/src/lib/config-manager.ts index 0be2789..47ab1e0 100644 --- a/src/lib/config-manager.ts +++ b/src/lib/config-manager.ts @@ -4,6 +4,80 @@ import chalk from "chalk"; import prompts from "prompts"; import type { DoccupineConfig, NormalizedOpenApiSpec } from "./types.js"; +import { isPathInside } from "./output-safety.js"; + +function pathsOverlap(a: string, b: string): boolean { + return isPathInside(a, b) || isPathInside(b, a); +} + +function isValidOpenApiConfig(value: unknown): boolean { + if (value === undefined || typeof value === "string") return true; + if (!Array.isArray(value)) return false; + return value.every( + (entry) => + typeof entry === "string" || + (entry !== null && + typeof entry === "object" && + typeof (entry as { file?: unknown }).file === "string" && + ((entry as { name?: unknown }).name === undefined || + typeof (entry as { name?: unknown }).name === "string")), + ); +} + +export function validateConfig( + value: unknown, + rootDir: string = process.cwd(), +): DoccupineConfig { + if (!value || typeof value !== "object" || Array.isArray(value)) { + throw new Error("doccupine.json must contain a JSON object."); + } + + const config = value as Partial; + if (typeof config.watchDir !== "string" || !config.watchDir.trim()) { + throw new Error("doccupine.json watchDir must be a non-empty string."); + } + if (typeof config.outputDir !== "string" || !config.outputDir.trim()) { + throw new Error("doccupine.json outputDir must be a non-empty string."); + } + + const watchDir = path.resolve(rootDir, config.watchDir); + const outputDir = path.resolve(rootDir, config.outputDir); + if (outputDir === path.resolve(rootDir)) { + throw new Error("outputDir cannot be the project root."); + } + if (pathsOverlap(watchDir, outputDir)) { + throw new Error("watchDir and outputDir must not overlap."); + } + + const port = config.port ?? "3000"; + if ( + typeof port !== "string" || + !/^\d+$/.test(port) || + Number(port) < 1 || + Number(port) > 65535 + ) { + throw new Error("doccupine.json port must be an integer from 1 to 65535."); + } + if ( + config.packageManager !== undefined && + config.packageManager !== "npm" && + config.packageManager !== "pnpm" + ) { + throw new Error('packageManager must be either "npm" or "pnpm".'); + } + if (!isValidOpenApiConfig(config.openapi)) { + throw new Error( + "openapi must be a path or an array of paths/{ name, file } objects.", + ); + } + + return { + ...config, + watchDir: config.watchDir, + outputDir: config.outputDir, + port, + }; +} /** Strips a `.json`/`.yaml`/`.yml` extension to derive a spec's default name. */ function basenameWithoutExt(file: string): string { @@ -149,16 +223,20 @@ export class ConfigManager { try { if (await fs.pathExists(this.configPath)) { const configContent = await fs.readFile(this.configPath, "utf8"); - const config = JSON.parse(configContent) as DoccupineConfig; + const config = validateConfig(JSON.parse(configContent)); console.log( chalk.blue("šŸ“„ Using existing configuration from doccupine.json"), ); return config; } } catch (error) { - console.warn( - chalk.yellow("āš ļø Error reading config file, will create new one"), - ); + if (await fs.pathExists(this.configPath)) { + throw new Error( + `Unable to use ${this.configPath}: ${ + error instanceof Error ? error.message : String(error) + }. Run "doccupine config --reset" to repair the configuration.`, + ); + } } return null; } @@ -251,7 +329,7 @@ export class ConfigManager { if (!config || options.reset) { console.log(chalk.blue("šŸ”§ Setting up Doccupine configuration...")); - config = await this.promptForConfig(config || {}); + config = validateConfig(await this.promptForConfig(config || {})); dirty = true; } else { // Configs written before 0.0.129 stored absolute paths; rewrite them in @@ -271,6 +349,8 @@ export class ConfigManager { dirty = true; } + config = validateConfig(config); + if (dirty) { await this.saveConfig(config); } diff --git a/src/lib/generated-artifacts.test.ts b/src/lib/generated-artifacts.test.ts new file mode 100644 index 0000000..ec0cbea --- /dev/null +++ b/src/lib/generated-artifacts.test.ts @@ -0,0 +1,143 @@ +import fs from "fs-extra"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; + +import { GeneratedArtifacts } from "./generated-artifacts.js"; + +const temporaryDirectories: string[] = []; + +async function temporaryDirectory(): Promise { + const directory = await fs.mkdtemp( + path.join(os.tmpdir(), "doccupine-artifacts-"), + ); + temporaryDirectories.push(directory); + return directory; +} + +async function makeDirectoryLink(target: string, linkPath: string) { + await fs.symlink( + target, + linkPath, + process.platform === "win32" ? "junction" : "dir", + ); +} + +afterEach(async () => { + await Promise.all( + temporaryDirectories.splice(0).map((dir) => fs.remove(dir)), + ); +}); + +describe("GeneratedArtifacts", () => { + it("persists source ownership and safe llms paths", async () => { + const outputDir = await temporaryDirectory(); + const artifacts = new GeneratedArtifacts(outputDir); + artifacts.replaceRoutes("mdx", [ + { source: "guide.mdx", slug: "api/guide" }, + ]); + artifacts.replaceLlmsPageFiles(["api/guide.md"]); + await artifacts.save(); + + const reloaded = new GeneratedArtifacts(outputDir); + await reloaded.load(); + + expect(reloaded.routeFor("mdx", "guide.mdx")).toBe("api/guide"); + expect(reloaded.llmsPageFiles()).toEqual(new Set(["api/guide.md"])); + }); + + it("rejects unsafe paths written through its API", async () => { + const outputDir = await temporaryDirectory(); + const artifacts = new GeneratedArtifacts(outputDir); + + expect(() => + artifacts.replaceRoutes("mdx", [ + { source: "guide.mdx", slug: "../../outside" }, + ]), + ).toThrow("unsafe mdx route ownership"); + expect(() => artifacts.replaceLlmsPageFiles(["../../outside.md"])).toThrow( + "unsafe llms page path", + ); + }); + + it("ignores traversal entries while safely migrating the legacy manifest", async () => { + const outputDir = await temporaryDirectory(); + await fs.writeJson(path.join(outputDir, ".doccupine-llms-manifest.json"), { + pageFiles: ["guide.md", "../../victim.md", "/absolute.md", "guide.txt"], + }); + await fs.writeJson(path.join(outputDir, ".doccupine-api-manifest.json"), { + pageSlugs: ["api-reference/users"], + }); + + const artifacts = new GeneratedArtifacts(outputDir); + await artifacts.load(); + + expect(artifacts.llmsPageFiles()).toEqual(new Set(["guide.md"])); + expect( + await fs.pathExists( + path.join(outputDir, ".doccupine-llms-manifest.json"), + ), + ).toBe(false); + expect( + await fs.pathExists(path.join(outputDir, ".doccupine-api-manifest.json")), + ).toBe(false); + }); + + it("rejects a symlinked current manifest on load and save", async () => { + const parent = await temporaryDirectory(); + const outputDir = path.join(parent, "output"); + const external = path.join(parent, "external"); + await fs.ensureDir(outputDir); + await fs.ensureDir(external); + await fs.writeFile(path.join(external, "important.txt"), "keep"); + await makeDirectoryLink( + external, + path.join(outputDir, ".doccupine-artifacts.json"), + ); + + const artifacts = new GeneratedArtifacts(outputDir); + await expect(artifacts.load()).rejects.toThrow("is a symbolic link"); + await expect(artifacts.save()).rejects.toThrow("is a symbolic link"); + await expect( + fs.readFile(path.join(external, "important.txt"), "utf8"), + ).resolves.toBe("keep"); + }); + + it.each([".doccupine-llms-manifest.json", ".doccupine-api-manifest.json"])( + "refuses to remove a symlinked legacy manifest %s", + async (fileName) => { + const parent = await temporaryDirectory(); + const outputDir = path.join(parent, "output"); + const external = path.join(parent, "external"); + await fs.ensureDir(outputDir); + await fs.ensureDir(external); + await fs.writeFile(path.join(external, "important.txt"), "keep"); + await makeDirectoryLink(external, path.join(outputDir, fileName)); + + const artifacts = new GeneratedArtifacts(outputDir); + await expect(artifacts.load()).rejects.toThrow("is a symbolic link"); + await expect( + fs.readFile(path.join(external, "important.txt"), "utf8"), + ).resolves.toBe("keep"); + }, + ); + + it("rejects an output root replaced with a symlink after construction", async () => { + const parent = await temporaryDirectory(); + const outputDir = path.join(parent, "output"); + const originalOutput = path.join(parent, "original-output"); + const external = path.join(parent, "external"); + await fs.ensureDir(outputDir); + await fs.ensureDir(external); + const artifacts = new GeneratedArtifacts(outputDir); + await fs.rename(outputDir, originalOutput); + await makeDirectoryLink(external, outputDir); + + await expect(artifacts.save()).rejects.toThrow( + "outputDir is not a real directory", + ); + await expect( + fs.pathExists(path.join(external, ".doccupine-artifacts.json")), + ).resolves.toBe(false); + }); +}); diff --git a/src/lib/generated-artifacts.ts b/src/lib/generated-artifacts.ts new file mode 100644 index 0000000..bead0b4 --- /dev/null +++ b/src/lib/generated-artifacts.ts @@ -0,0 +1,190 @@ +import fs from "fs-extra"; +import path from "node:path"; + +import { readOutputFileIfPresent, resolveOutputPath } from "./output-safety.js"; +import { writeFileAtomic } from "./utils.js"; + +export type RouteOwnerKind = "mdx" | "openapi"; + +interface RouteArtifact { + kind: RouteOwnerKind; + source: string; + slug: string; +} + +interface ArtifactManifest { + schemaVersion: 1; + routes: RouteArtifact[]; + llmsPageFiles: string[]; +} + +const MANIFEST_FILE = ".doccupine-artifacts.json"; +const LEGACY_LLMS_MANIFEST = ".doccupine-llms-manifest.json"; +const LEGACY_API_MANIFEST = ".doccupine-api-manifest.json"; + +function normalizeRelativePath( + value: string, + allowEmpty = false, +): string | null { + const normalized = value.replace(/\\/g, "/").replace(/^\.\//, ""); + if ((allowEmpty && normalized === "") || normalized === "") { + return allowEmpty ? "" : null; + } + if (path.posix.isAbsolute(normalized)) return null; + const parts = normalized.split("/"); + if (parts.some((part) => part === "" || part === "." || part === "..")) { + return null; + } + return normalized; +} + +function normalizeRoute(value: unknown): RouteArtifact | null { + if (!value || typeof value !== "object") return null; + const route = value as Record; + if (route.kind !== "mdx" && route.kind !== "openapi") return null; + if (typeof route.source !== "string" || typeof route.slug !== "string") { + return null; + } + const source = normalizeRelativePath(route.source); + const slug = normalizeRelativePath(route.slug, true); + return source === null || slug === null + ? null + : { kind: route.kind, source, slug }; +} + +function normalizeLlmsPageFile(value: unknown): string | null { + if (typeof value !== "string") return null; + const normalized = normalizeRelativePath(value); + return normalized?.endsWith(".md") ? normalized : null; +} + +export class GeneratedArtifacts { + private routes = new Map(); + private llmsFiles = new Set(); + + constructor(private readonly outputDir: string) {} + + private manifestPath(fileName: string): string { + return resolveOutputPath(this.outputDir, fileName); + } + + private routeKey(kind: RouteOwnerKind, source: string): string { + return `${kind}:${source.replace(/\\/g, "/")}`; + } + + async load(): Promise { + this.routes.clear(); + this.llmsFiles.clear(); + + const manifestContent = await readOutputFileIfPresent( + this.outputDir, + MANIFEST_FILE, + ); + if (manifestContent !== null) { + try { + const parsed = JSON.parse(manifestContent) as { + routes?: unknown; + llmsPageFiles?: unknown; + }; + if (Array.isArray(parsed.routes)) { + for (const value of parsed.routes) { + const route = normalizeRoute(value); + if (route) + this.routes.set(this.routeKey(route.kind, route.source), route); + } + } + if (Array.isArray(parsed.llmsPageFiles)) { + for (const value of parsed.llmsPageFiles) { + const file = normalizeLlmsPageFile(value); + if (file) this.llmsFiles.add(file); + } + } + } catch { + // A corrupt manifest owns nothing. Generated output is rebuilt safely. + } + } + + // Import only valid paths from the old llms manifest. This preserves stale + // mirror cleanup across upgrades without trusting its historical contents. + const legacyContent = await readOutputFileIfPresent( + this.outputDir, + LEGACY_LLMS_MANIFEST, + ); + if (legacyContent !== null) { + try { + const parsed = JSON.parse(legacyContent) as { + pageFiles?: unknown; + }; + if (Array.isArray(parsed.pageFiles)) { + for (const value of parsed.pageFiles) { + const file = normalizeLlmsPageFile(value); + if (file) this.llmsFiles.add(file); + } + } + } catch { + // Ignore corrupt legacy state. + } + } + + await Promise.all([ + fs.remove(this.manifestPath(LEGACY_LLMS_MANIFEST)), + fs.remove(this.manifestPath(LEGACY_API_MANIFEST)), + ]); + } + + routeFor(kind: RouteOwnerKind, source: string): string | undefined { + return this.routes.get(this.routeKey(kind, source))?.slug; + } + + routesFor(kind: RouteOwnerKind): RouteArtifact[] { + return [...this.routes.values()].filter((route) => route.kind === kind); + } + + replaceRoutes( + kind: RouteOwnerKind, + routes: Iterable<{ source: string; slug: string }>, + ): void { + for (const [key, route] of this.routes) { + if (route.kind === kind) this.routes.delete(key); + } + for (const value of routes) { + const route = normalizeRoute({ kind, ...value }); + if (!route) { + throw new Error(`Refusing to record unsafe ${kind} route ownership`); + } + this.routes.set(this.routeKey(kind, route.source), route); + } + } + + removeRoute(kind: RouteOwnerKind, source: string): void { + this.routes.delete(this.routeKey(kind, source)); + } + + llmsPageFiles(): Set { + return new Set(this.llmsFiles); + } + + replaceLlmsPageFiles(files: Iterable): void { + const next = new Set(); + for (const value of files) { + const file = normalizeLlmsPageFile(value); + if (!file) throw new Error("Refusing to record an unsafe llms page path"); + next.add(file); + } + this.llmsFiles = next; + } + + async save(): Promise { + const manifest: ArtifactManifest = { + schemaVersion: 1, + routes: [...this.routes.values()].sort((a, b) => + `${a.kind}:${a.source}`.localeCompare(`${b.kind}:${b.source}`), + ), + llmsPageFiles: [...this.llmsFiles].sort(), + }; + await writeFileAtomic( + this.manifestPath(MANIFEST_FILE), + `${JSON.stringify(manifest, null, 2)}\n`, + ); + } +} diff --git a/src/lib/layout.ts b/src/lib/layout.ts index 0522f68..17c826a 100644 --- a/src/lib/layout.ts +++ b/src/lib/layout.ts @@ -107,24 +107,25 @@ function fontDeclLine(fontConfig: FontConfig | null): string { : 'const font = Inter({ subsets: ["latin"] });'; } -// The inline blocking script that resolves dark mode before first paint. -// Adds the "dark" class to (so the CSS variables emitted by -// GlobalStyles paint dark immediately) and, on dark visits, injects a -// "#__theme-init" style that hides the body AND forces a dark html -// background: the server always renders with the light theme (pages are -// force-static, so no cookie read), and Cherry's ClientThemeProvider -// briefly re-syncs the "dark" class off the initial light theme during -// mount, so without the forced background the html would flash white for a -// frame. A second "__theme-transitions" style disables all transitions so -// the light-to-dark swap on mount snaps instantly instead of animating every -// element that declares a color transition. It can't live in "__theme-init": -// the provider removes that in the same frame the dark colors commit, so the -// browser would compute one light-to-dark style diff with transitions -// re-enabled and animate it anyway. Instead a MutationObserver waits for the -// provider to remove "__theme-init" (themeDark committed, body unhidden), -// forces a reflow, and drops the transition suppression two painted frames -// later ā€” restoring transitions for user-initiated toggles. -const THEME_INIT_SCRIPT = `(function(){try{var c=document.cookie.split(";").map(function(s){return s.trim();}).find(function(s){return s.indexOf("theme=")===0;});var v=c?c.split("=")[1]:null;var d=v?v==="dark":(window.matchMedia&&window.matchMedia("(prefers-color-scheme:dark)").matches);if(!v){document.cookie="theme="+(d?"dark":"light")+";path=/;max-age=31536000;SameSite=Lax";}if(d){document.documentElement.classList.add("dark");document.documentElement.style.colorScheme="dark";var s=document.createElement("style");s.id="__theme-init";s.textContent="html{background:#000!important;color-scheme:dark}body{visibility:hidden}";document.head.appendChild(s);var t=document.createElement("style");t.id="__theme-transitions";t.textContent="*,*::before,*::after{transition:none!important}";document.head.appendChild(t);var f=function(){var e=document.getElementById("__theme-transitions");if(!e)return;void document.body.offsetHeight;requestAnimationFrame(function(){requestAnimationFrame(function(){e.remove();});});};var o=new MutationObserver(function(){if(!document.getElementById("__theme-init")){o.disconnect();f();}});o.observe(document.head,{childList:true});setTimeout(f,10000);}else{document.documentElement.style.colorScheme="light";}}catch(e){}})();`; +// The inline blocking script that resolves dark mode before the first paint. +// It only has to decide the mode and record it on : every painted color +// in the app resolves through the CSS custom properties that GlobalStyles +// emits under :root and :root[data-theme="dark"], so that one attribute is +// enough to make a statically rendered page arrive dark. Nothing is hidden and +// nothing waits for hydration ā€” the markup streams and paints in the right +// theme exactly like a light visit does. +// +// The mode lives on our own data-theme attribute rather than on the "dark" +// class, because that class belongs to Cherry's ClientThemeProvider, which +// syncs it from the theme object: releases before 0.2.12 strip it for a frame +// on mount, since the server render is always the light theme, and every color +// on the page would flash light with it. The class is still set here so CSS +// written against it ā€” Cherry's own, or a user's ā€” keeps working. +// +// The mode comes from the "theme" cookie, falling back to the OS preference +// and seeding the cookie so Cherry's ClientThemeProvider reconciles against +// the same answer on mount instead of flipping the class back. +const THEME_INIT_SCRIPT = `(function(){try{var c=document.cookie.split(";").map(function(s){return s.trim();}).find(function(s){return s.indexOf("theme=")===0;});var v=c?c.split("=")[1]:null;var d=v?v==="dark":(window.matchMedia&&window.matchMedia("(prefers-color-scheme:dark)").matches);if(!v){document.cookie="theme="+(d?"dark":"light")+";path=/;max-age=31536000;SameSite=Lax";}var r=document.documentElement;r.dataset.theme=d?"dark":"light";if(d){r.classList.add("dark");}}catch(e){}})();`; /** * Root layout ("app/layout.tsx"). Minimal shell: html/body, fonts, the theme @@ -182,16 +183,13 @@ export default function RootLayout({ return ( - {/* Resolves dark mode before first paint: adds the "dark" class to - (flipping the CSS variables in GlobalStyles) and hides the - body until Cherry's ClientThemeProvider has swapped in themeDark - on mount ā€” the server always renders the light theme since pages - are force-static. Inlined as a plain