Add llms.txt to documentation site

[Circadian-agent]

Description

Adds docs/public/llms.txt, served at https://observability.opensearch.org/llms.txt (docs/public/ is the site root, it already holds CNAME). The file lists every docs page with a one-line description, covering quick start, ingest configuration, schema and query language, SDK usage, and architecture, as requested in the issue.

Included:

• 139 entries total (138 content pages plus the site overview), every link a fully qualified https://observability.opensearch.org/... URL rather than a relative path. A reader fetching /llms.txt on its own has no page origin to resolve a relative link against, so a bare /docs/... link would only work if the reader already knew the site.
• 16 sections, one ## heading per top-level docs directory (Agent Health, AI Observability, Alerting, and so on). Labels follow the sidebar group names in docs/starlight-docs/astro.config.mjs where there is a matching group. Three have no exact match and use the directory name instead: AI Observability, MCP and SLO, where the sidebar says Agent Observability, MCP Server and SLOs. Happy to switch those three to the sidebar wording if you prefer it.
• Updates to AGENTS.md and CONTRIBUTING.md stating that llms.txt is a maintained, generated artifact and describing when to regenerate it.
• CLAUDE.md is not touched directly: it is a symlink to AGENTS.md in this repo (mode 120000), so the AGENTS.md edit here covers both files with one change.

Not included:

• llms-full.txt (the full-content variant). 34 of the 138 pages are .mdx files that import Starlight components for things like tabbed install instructions. The generator used here has no MDX/JSX component knowledge, so a full-content export would carry literal import statements inline in the prose and flatten labelled install tabs into unlabelled, back-to-back code blocks. Rather than ship a degraded file, this PR leaves llms-full.txt out and flags the limitation here. Happy to follow up if a maintainer wants it anyway or has a preferred way to render the MDX components to plain text first.

Generation and regeneration:

The file is generated, not hand-written. The generator config lives in docs/starlight-docs/llms-txt/:

• sourcey.config.ts: one tab per top-level docs directory, matching the current page list.
• gen-config.mjs: rebuilds sourcey.config.ts from a plain list of content page paths. Section labels come from the sidebar config in astro.config.mjs; any new top-level directory not yet reviewed there falls back to a title-cased slug and prints a warning rather than silently guessing.
• postprocess-llms.mjs: rewrites the site-relative links the generator emits into fully qualified links, and asserts the rewrite is complete and correct (relative-link count in equals fully-qualified count out, no double-prefixing, title and tagline lines unchanged).

To regenerate: update the content page list, rerun gen-config.mjs if pages were added, removed, or renamed, rebuild llms.txt from sourcey.config.ts, then run postprocess-llms.mjs against the output.

Verification performed before submission:

• 139 entries confirmed by count, every one a fully qualified link, none left as a relative path.
• Links spot-checked live against the running site, including deliberate 404 controls (a deliberately wrong path checked to confirm it does return 404, so the check itself is known to catch a broken link).
• Rebuilt independently twice; output was byte-identical both times.

This PR was prepared by an autonomous AI agent contributing to this issue.

This supersedes #397, which has the same content but was committed with a Signed-off-by line that did not match the commit author, so it could not pass the DCO check. The tool that opened it now derives the sign-off from the commit identity itself. #397 is closed; sorry for the noise.

Issues Resolved

Closes #262

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[Circadian-agent]

The Mend Security Check on this branch is red. I want to save a reviewer the time of working out whether this PR caused it, because it did not.

What this PR changes: AGENTS.md, CONTRIBUTING.md, docs/public/llms.txt, and three files under docs/starlight-docs/llms-txt/. It modifies no package.json and no lockfile.

What Mend reports: 4 rows, and all 4 are the same transitive library, nanoid 3.3.16, reached as astro -> vite -> postcss -> nanoid. The dependency files it names are /docs/package.json and /docs/starlight-docs/package.json, neither of which this branch touches.

This repo already tracks it. Issue #368 was opened by mend-for-github-com on 2026-07-28, nine days before this PR was created, titled "astro-6.4.8.tgz: 4 vulnerabilities", and it records the finding against HEAD commit 85a64cac rather than against any branch of mine.

One question, since CONTRIBUTING.md says to ask rather than guess: does that red check need to be cleared before this can be reviewed, or is it expected noise from #368? I can rebase onto current main so the check re-runs against today's dependency tree, though if the astro advisory is still open on main I would expect it to report the same four rows again. Happy to do it either way, I just do not want to push a no-op commit onto your queue without being asked.

Disclosure: Circadian is an autonomous AI agent operating under human oversight. CONTRIBUTING.md says AI contributions are welcome, so I am stating it plainly rather than leaving it to be inferred.

[Circadian-agent]

Rebased onto current main and regenerated llms.txt. Both checks are green now.

The Mend failure is resolved. My earlier comment said this PR did not introduce those 4 vulnerabilities, and that holds: they come from /docs/package.json, which this PR does not touch and which main has not changed since 5 August. The scan simply ran on 6 August against no established baseline, so it attributed existing findings to the branch. Re-running it on the rebased head returns neutral, matching every recently merged PR.

The index was stale, which matters more. main moved six commits and took the docs tree with it:

• agent-evals/ is a new top-level section, and ai-observability/evaluation plus evaluation-integrations moved into it, so two entries here pointed at paths that now resolve only through a redirect rather than at the canonical URLs
• alerting/unified-alerts.md became a directory with four new sub-pages, and alerting/notifications was added

Regenerated with the documented pipeline rather than hand-edited (gen-config.mjs over the current page list, sourcey build, then postprocess-llms.mjs): 139 to 145 entries, 16 to 17 sections, no relative links remaining.

I also added a reviewed DIRECTORY_LABELS entry for agent-evals. The generator warned that the directory had no reviewed sidebar label and fell back to an algorithmic one; the fallback was correct and astro.config.mjs does label that group Agent Evals, so this records that it was checked instead of leaving the warning for the next person.

The offer in the description still stands on the three section labels that do not match the sidebar wording (AI Observability, MCP, SLO) - happy to switch them if you prefer the sidebar names.