perf(search): build a LanceDB vector index — dense recall is a full scan today (81ms → 2.2ms measured)

[gloryfromca]

Summary

EverOS builds no vector ANN index. ensure_business_indexes() only creates FTS indexes — the sole
create_index call in the codebase is BaseLanceTable.ensure_fts_indexes with config=FTS(...), and
lancedb.index is imported only for FTS. Dense retrieval is therefore an exhaustive scan on every recall
call.

Measured on a live index (atomic_fact, 117,167 rows × 1024 dims, 200 queries, exact cosine ground truth):

Configuration	best match ranked first	p50 latency
No index (current behaviour)	100%	81.1 ms
IVF_FLAT, nprobes=1	100%	2.2 ms
IVF_FLAT, nprobes=10	100%	2.8 ms
IVF_FLAT, nprobes=64	100%	7.4 ms
IVF_FLAT, nprobes=256	100%	14.7 ms

Index build took 5.6 s. Recall was unchanged at every nprobes setting on this corpus.

A single /search issues several dense recall calls (one per route, plus an extra atomic_fact recall in
hierarchical mode, multiplied by sub-queries in agentic mode), so the per-call cost compounds.

Why this is low-risk to adopt

• The index is derived data. create_index builds over existing rows in place; table schemas do not change,
  so verify_business_schemas is unaffected and no data migration is needed.
• ensure_fts_indexes already checks list_indices() and skips indexed columns, and
  ensure_business_indexes() runs at every startup (API lifespan and CLI _runtime). Extending that
  contract means existing deployments build the index on first startup after upgrade — no script, no
  downtime.
• Rows written after the build are already handled: optimize() folds the unindexed tail into the index, and
  until it does LanceDB flat-scans that tail and unions it with indexed hits, so query results stay correct.
  This is documented and verified in LanceRepoBase.optimize.

Known obstacles

• Small tables fail. create_index with num_partitions=256 on 10 rows raises
  RuntimeError: KMeans cannot train 256 centroids with 10 vectors. 500 rows succeeds. A fresh install has
  empty tables, and agent_case / agent_skill / knowledge_topic are tiny in practice — a row-count guard
  is required, and the partition count probably needs to scale with table size.
• The rebuild-cost assumption changes. DEFAULT_OPTIMIZE_REBUILD_INTERVAL_SECONDS = 12h is justified in
  its docstring by "rebuild cost is ~0.3s per 50k rows × indexed columns". A vector index costs ~2.4 s per
  50k rows at 1024 dims, roughly 8× that, growing with dimensionality. 12 h remains comfortable, but the
  justification needs updating rather than left stale.
• Index-file accumulation gains another category. rebuild_indexes() exists because optimize()
  accumulates one new active index UUID (vector) or part_N (FTS) per call, with no application-level way to
  collapse them on the embedded lance crate 4.0. Adding a vector index adds to that surface — the same one
  behind the index-bloat incident. When extending the index contract, both the drop and the rebuild side of
  rebuild_indexes() must cover the vector index.
• nprobes is never passed. Recall paths use the LanceDB default. nprobes=1 already reaches 100% on
  this corpus so the default is adequate, but it becomes a tuning surface and needs a decision on whether to
  expose it as configuration.

Before implementing

1. Repeat the measurement on the other tables — episode (688 rows), foresight (4,479),
   knowledge_topic (a handful) — to fix the row-count guard threshold and partition sizing.
2. Re-measure recall with queries that have no exact duplicate in the corpus. The 100%-at-nprobes=1 result
   was obtained on a corpus where every query had an exact match. Include an exact-search control in any
   recall measurement: if a brute-force engine does not score 1.0 on the chosen metric, the metric is wrong,
   not the engine.
3. Confirm the periodic rebuild loop actually collapses the vector index, given the accumulation issue above.

Reproduction script for the numbers above: recall_v4_lance_ann.py (kept with the internal notes; happy to
attach it to this issue if useful).

[gloryfromca]

Follow-up measurement: LanceDB's default nprobes.

The Python docstring only says "a default number of partitions will be used" — the value lives in Rust. Inferred from the latency curve on 117k × 1024 with num_partitions=256:

Setting	p50
unset (library default)	3.18 ms
nprobes=1	1.94 ms
nprobes=10	2.31 ms
nprobes=20	2.75 ms
nprobes=30	3.35 ms
nprobes=64	4.88 ms

The default sits between 20 and 30, consistent with the API exposing minimum_nprobes and maximum_nprobes as separate knobs — the default is an adaptive 20–30 band (probe 20 partitions, expand toward 30 when needed) rather than a fixed value.

So the implementation decision is whether to leave the adaptive default alone or pin nprobes explicitly.

Note ef is not applicable here: ef is the HNSW search knob and nprobes is the IVF one — the index measured above is IVF_FLAT.