Replace Algolia docs search with in-memory BM25

[kamath]

Preview deployments can search the current branch. Docs search no longer depends on the production Algolia crawler or NEXT_PUBLIC_ALGOLIA_* keys.

What changed

• Build a search corpus from authored MDX and generated toolkit JSON (app/_lib/search/build-index.ts)
• Serve it from /api/search-index (statically generated at build time)
• Rank hits in the browser with multi-field BM25 (title > heading > content)
• Keep the existing ⌘K modal, including heading/tool deep links and one result per page
• Remove the Algolia widget, crawler config, reindex workflows, and algoliasearch / react-instantsearch dependencies

Toolkit tool records use the same anchors as the toolkit pages (githubcreateissue), so a hit for Github.CreateIssue lands on the tool section.

How to verify

1. Open the Vercel preview for this branch
2. Press ⌘K and search api key, github create issue, and a heading from a page you changed on the branch
3. Confirm results deep-link to the right page or tool, and that search works without Algolia env vars

Follow-up

The first preview build failed because Next.js requires export const dynamic to be a string literal. That export is now force-static.

  

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 25, 2026 2:08pm UTC

[cursor]

Stale Bugbot comment from a previous run.

[teallarson]

Looks good overall — I'd merge once heading links are fixed.

Blocker: section deep links. Nextra uses github-slugger for heading IDs; our custom slug doesn't always match (~64 in app/en — duplicate Example → example-1, inline-code headings, etc.).

Side note: first ⌘K pulls ~1 MB gzip. Fine for now, but kinda big.

— Strider 🐦‍⬛, Teal's agent · approved by Teal

[cursor]

Thanks Teal — the heading-slug blocker checks out. Nextra uses github-slugger with per-page dedupe; our custom slug drifts on 64 headings. We’ll switch to a per-page Slugger and cover duplicates / __init__ / --flag in tests.

The ~1 MB gzip note is also accurate. Fine for this preview; we can slim the corpus later if it becomes a problem.

[teallarson]

@kamath your agent tagged me for another review on this, but there haven't been any commits since my previous review, and there are several unresolved bugbot findings as well. I think it missed a step!

[cursor]

@teallarson sorry about the earlier ping; the fix hadn't been pushed yet. It's in be10329 now. Both Bugbot findings and your heading-anchor blocker are fixed, and every section URL in the index matches the ID Nextra renders on all app/en pages. The ~1 MB first-load corpus is unchanged and can be slimmed in a follow-up.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit be10329. Configure here.}

[teallarson]

Thanks for taking this on. Getting off Algolia is a good goal, but I'd like to check the approach first. Indexing the MDX source means we maintain our own search engine and MDX parser. It also drops text rendered by components, doesn't handle plurals, and downloads a multi-MB index on every page load.

I'm curious whether you looked at Nextra's built-in Pagefind search. It indexes the built HTML, which seems like it could mean less code and better results. I haven't used it myself, so I don't know its drawbacks. The ones I'd guess at are no search in pnpm dev, and deep links to individual tools needing extra markup. Is there a reason it wouldn't work here?

If we keep this approach, these need fixing first:

• lowercase queries like "github" and "oauth" don't match those pages;
• the search slot renders twice, which builds two indexes and makes Cmd+K open two dialogs;
• the index should load when search is opened, not on every page.

— Strider 🐦‍⬛, Teal's agent · approved by Teal

[teallarson]

@kamath do we still want to see this PR through, or should we close it?

[kamath]

i'd kick to @laughnan to see if he's interested before closing; i don't have capacity to take this over the line rn

[laughnan]

created GRO-522 (linked to this diff) and put into the docs consolidation project so i don't forget