fix(embedding): frequent-token subsampling matches its cited CBM source (#184)

[cdeust]

Closes #184.

What actually diverged

mcp_server/shared/algorithmic_embedding.py::embed_text cited CBM's
CBM_SEM_MAX_OCCUR = 512 (semantic.h:52, applied in semantic.c:874 cooccur_sparse_one_target) for frequent-token subsampling, but implemented
different semantics on both axes the issue names:

	CBM (cited source)	Cortex as shipped
Trigger	one token occurs >512 times	the document exceeds 512 tokens
Scope	that token's co-occurrence pass only	the whole loop — first-order terms too

Mikolov et al. 2013's P(discard) = 1 - sqrt(t/f(w)) was never the mechanism in
play: it needs a corpus-relative f(w) this stateless per-text seam does not
have (the same reason IDF is excluded). CBM's mechanism is stride-sampling of a
frequent token's occurrence positions — inspired by word2vec subsampling, not
implementing its formula. The shipped code borrowed the constant and the
citation but applied a document-length cost bound the source does not describe.

Which fix path the evidence chose: (a) match the source

Measured on LongMemEval-S (n=50 haystack, 2556 memories):

• token count per memory: median 1622, p90 2782, max 6008.
• 86.2% exceed the 512 cap → the shipped document-stride fired on the
  majority of stored vectors (the committed feat: zero-download semantic fallback — algorithmic embeddings + code-aware FTS tokenizer #169 numbers were produced with it).
• largest single-token within-document repetition is 609; only 2/2556
  memories would trigger the CBM per-token stride → inert on prose, exactly as
  semantic.h:52 claims.

Three-way retrieval comparison (benchmarks/results/subsample-fidelity-184/):

variant	MRR	Recall@10	vs off
off	0.3786	68.0%	—
current	0.3777	66.0%	ΔMRR −0.0009, ΔR@10 −2.0 pp
faithful_a	0.3786	68.0%	ΔMRR ±0.0000, ΔR@10 ±0.0 pp

The shipped document-striding does not merely fail to help — it hurts (−2.0
pp Recall@10), because it drops the first-order term at half the positions of the
~86% of memories over the cap. Chose (a): CBM-faithful. It restores §8 source
fidelity (code now matches the citation and the module docstring, which already
described the CBM semantics), scores best, and keeps a source-faithful cost guard
that genuinely fires (2/2556), which the acceptance criteria expect tested rather
than removed. Removal would score identically but delete a documented, firing
guard for no §8 benefit.

Benchmark deltas

Committed #169 sweep re-run on the fixed embedder (§8 — a changed embedder
invalidates recorded numbers; benchmarks/results/semantic-fallback-169/
updated, only the fallback row moves — no-vector/sentence-transformers
do not use this embedder):

mode	before (#169)	after (#184)
algorithmic fallback	MRR 0.378, R@10 66.0%	MRR 0.379, R@10 68.0%

Elapsed rose (38.6 → 62.9 s) because the co-occurrence pass is no longer strided
on long prose — an accepted trade for a non-hot-path encode whose vectors upgrade
to neural on the first consolidate.

Completion Ledger

Acceptance criterion (#184)	Status
1. Decide + state semantics at the use site	Done — option (a); # source: on _MAX_OCCUR states per-token trigger + co-occurrence scope
2. Docstring and code agree	Done — module docstring already described (a); code now matches; Mikolov relationship clarified (inspiration, not formula)
3. Tests cover the branch (striding + boundary at exactly _MAX_OCCUR)	Done — test_frequent_token_triggers_subsampling…, test_boundary_at_exactly_max_occur_does_not_stride, direction-preserved test
4. Negative assertion: normal memory → stride == 1	Done — test_normal_memory_takes_no_subsampling, test_first_order_terms_are_never_subsampled
5. Mutation run scoped to the module; stride branch the target	Done — 0 survivors in _cooccurrence_pass/_first_order_pass/_accumulate_window; remaining survivors documented-equivalent or pre-existing in unchanged functions
6. Since (a), re-run committed #169 sweep + update manifest	Done — semantic-fallback-169/ MANIFEST + JSON updated

Gates

• ruff check + ruff format --check: pass.
• Full pytest: 5364 passed, 9 skipped, 0 failed (43m) — test: test_rebuild_profiles depends on live ~/.claude data — 3 failures reproduce on main #174 zero-failure baseline holds. The 4 warnings are pre-existing asyncio-teardown ResourceWarnings, unrelated.

§15 No-Deviation

Scope held to the one module, its test, and the two benchmark result dirs. No
unrelated files touched; pyproject.toml mutmut scope restored to its committed
value after the per-change run.

🤖 Generated with Claude Code