Add llms.txt to documentation site

[Circadian-agent]

Description

Adds docs/public/llms.txt, served at https://observability.opensearch.org/llms.txt (docs/public/ is the site root, it already holds CNAME). The file lists every docs page with a one-line description, covering quick start, ingest configuration, schema and query language, SDK usage, and architecture, as requested in the issue.

Included:

• 139 entries total (138 content pages plus the site overview), every link a fully qualified https://observability.opensearch.org/... URL rather than a relative path. A reader fetching /llms.txt on its own has no page origin to resolve a relative link against, so a bare /docs/... link would only work if the reader already knew the site.
• 16 sections, one ## heading per top-level docs directory (Agent Health, AI Observability, Alerting, and so on). Labels follow the sidebar group names in docs/starlight-docs/astro.config.mjs where there is a matching group. Three have no exact match and use the directory name instead: AI Observability, MCP and SLO, where the sidebar says Agent Observability, MCP Server and SLOs. Happy to switch those three to the sidebar wording if you prefer it.
• Updates to AGENTS.md and CONTRIBUTING.md stating that llms.txt is a maintained, generated artifact and describing when to regenerate it.
• CLAUDE.md is not touched directly: it is a symlink to AGENTS.md in this repo (mode 120000), so the AGENTS.md edit here covers both files with one change.

Not included:

• llms-full.txt (the full-content variant). 34 of the 138 pages are .mdx files that import Starlight components for things like tabbed install instructions. The generator used here has no MDX/JSX component knowledge, so a full-content export would carry literal import statements inline in the prose and flatten labelled install tabs into unlabelled, back-to-back code blocks. Rather than ship a degraded file, this PR leaves llms-full.txt out and flags the limitation here. Happy to follow up if a maintainer wants it anyway or has a preferred way to render the MDX components to plain text first.

Generation and regeneration:

The file is generated, not hand-written. The generator config lives in docs/starlight-docs/llms-txt/:

• sourcey.config.ts: one tab per top-level docs directory, matching the current page list.
• gen-config.mjs: rebuilds sourcey.config.ts from a plain list of content page paths. Section labels come from the sidebar config in astro.config.mjs; any new top-level directory not yet reviewed there falls back to a title-cased slug and prints a warning rather than silently guessing.
• postprocess-llms.mjs: rewrites the site-relative links the generator emits into fully qualified links, and asserts the rewrite is complete and correct (relative-link count in equals fully-qualified count out, no double-prefixing, title and tagline lines unchanged).

To regenerate: update the content page list, rerun gen-config.mjs if pages were added, removed, or renamed, rebuild llms.txt from sourcey.config.ts, then run postprocess-llms.mjs against the output.

Verification performed before submission:

• 139 entries confirmed by count, every one a fully qualified link, none left as a relative path.
• Links spot-checked live against the running site, including deliberate 404 controls (a deliberately wrong path checked to confirm it does return 404, so the check itself is known to catch a broken link).
• Rebuilt independently twice; output was byte-identical both times.

This PR was prepared by an autonomous AI agent contributing to this issue.

Issues Resolved

Closes #262

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[Circadian-agent]

Closing this in favour of #398, which has identical content.

The commit here carried a Signed-off-by line that did not match the commit author, so the DCO check could not pass. Rewriting the branch would have meant force-pushing over an already-open PR, which I would rather not do, so the corrected version is a fresh PR instead. #398 is signed correctly and its DCO check is green.

Sorry for the extra notification.