perf(entity-resolver): optimize in-batch dedup via prefix filtering

[Sanderhoff-alt]

Summary

Replaces the O(N^2) pairwise loop in _find_intrabatch_similar_pairs with a prefix-filtering set-similarity join (AllPairs / PPJoin), so in-batch entity dedup verifies only the pairs that can clear the merge cutoff instead of all of them. Returns exactly the pairs the double loop returned — same Jaccard, same cutoff.

Also memoises the candidate trigram sets in _resolve_from_candidates, so a canonical name shared by several mentions in a batch is trigrammed once rather than once per mention.

Relates to #3107, #3211, #3751.

How the pruning is lossless

Sort each name's trigrams rarest-first and index only the leading |A| - ceil(t * |A|) + 1 of them. Two sets sharing none of those tokens cannot overlap enough to clear t, so a pair that misses every indexed token can be skipped without computing its similarity.

Each prefix is cut from its own size, which is what makes it safe: a qualifying pair needs an overlap of at least ceil(t * max(|A|, |B|)), and the shorter set's prefix — cut for its own smaller size — is longer than that bound demands, so the pair cannot slip past both prefixes. Ascending set-size order is not load-bearing for correctness (descending is equally correct); it is there so the |B| >= t * |A| size filter has something to reject.

Equivalence

tests/test_entity_intrabatch_clustering.py pins the join against the loop it replaced, over randomized batches at 14 cutoffs — including exactly-achievable Jaccard ratios like 7/13, where a float t * |A| landing a hair above a whole number would shorten the prefix by one and silently drop a pair sitting on the cutoff. Plus the ordering, no-trigram and all-names-alike cases. An off-by-one in the prefix length fails 14 of these tests; dropping the size filter fails 15.

Checked outside the suite as well:

• 60,000 randomized set-level trials at exactly-achievable thresholds (a/b for b <= 24) — 0 mismatches.
• 9,525 per-document name batches harvested from the LoCoMo and LongMemEval corpora — the shape retain actually builds — at six cutoffs. 0 mismatches.

Performance

./scripts/benchmarks/run-entity-resolver-bench.sh --repeats 10. Workloads are built from distinct names, because the caller passes rep_by_lower.values() — one entry per distinct new name in the batch.

Workload	N	baseline	prefix filtering	speedup
micro_batch_20	20	0.11 ms	0.17 ms	0.68x
small_batch_50	50	0.47 ms	0.40 ms	1.19x
cap_batch_250	250	9.54 ms	2.47 ms	3.87x
cap_batch_250_low_cutoff (t=0.2)	250	9.65 ms	3.92 ms	2.46x
adversarial_250_alike	250	24.65 ms	29.21 ms	0.84x
above_cap_500	500	37.78 ms	5.96 ms	6.34x
above_cap_500_alike	500	147.81 ms	156.07 ms	0.95x
bulk_import_1000	1000	149.12 ms	18.59 ms	8.02x

Peak memory rises by 151 KiB at the 250-name cap and 483 KiB at 1000 — the inverted index and the prefix lists, all local to the function.

What this looks like on real batches

The synthetic numbers above are the ceiling, not the expectation. Replaying the 9,525 real per-document batches (median 58 distinct names, p95 156, max 438):

batch size	count	baseline	prefix filtering	speedup
0–24	2,234	0.051 ms	0.093 ms	0.55x
25–49	2,135	0.216 ms	0.230 ms	0.94x
50–74	1,159	0.613 ms	0.378 ms	1.62x
100–124	1,289	1.551 ms	0.733 ms	2.12x
150–174	379	3.286 ms	0.929 ms	3.54x
225–249	11	6.586 ms	1.541 ms	4.27x

2.47x over the whole corpus (9,136 ms → 3,705 ms), with the win entirely in the tail: roughly 46% of real batches are under 50 names and get slightly slower, by ~40 µs each. That is well below the noise floor of a retain that spends 10–100 ms on DB and LLM, and a size threshold would buy those microseconds back at the cost of a second code path — so there isn't one.

Two shapes that are slower

1. Small batches. Under ~50 names the index costs more than it saves (0.55x at the smallest sizes). Tens of microseconds; see above.
2. Mutually similar names. 250 names shaped like Acme Corporation Subsidiary 0001 probe into every bucket, so the filter prunes nothing and the index is pure overhead: ~1.2x slower than the double loop. Bounded, and _INTRABATCH_MAX_NAMES = 250 keeps it around 29 ms. Both shapes are now workloads in the benchmark rather than footnotes.

On raising _INTRABATCH_MAX_NAMES

Left at 250. The distinct-name numbers make 500 look free, but the mutually-similar shape at 500 names costs ~156 ms of CPU with no await inside the join — worse than the ~80 ms at N=1000 that the cap was originally chosen against. Real batches also top out at 438 names and sit at 58 median, so the cap is rarely the binding constraint. Worth revisiting against a real bulk-import corpus (catalogs, SKUs), not on these numbers.

Also in this PR

• entity-resolver-bench microbenchmark + ./scripts/benchmarks/run-entity-resolver-bench.sh. Its conformance column is measured against _v_baseline_quadratic specifically, not against whichever variant runs first — otherwise it would compare the optimisation to its own output.
• _PrefixIndexed dataclass rather than a 5-tuple.
• The candidate trigram memo is filled lazily, so a candidate the scoring loop never reaches (a label row, or a mention that resolves before scoring) costs nothing, and no work happens ahead of the _SCORING_YIELD_EVERY yield points (resolver: synchronous candidate-scoring loop can block the worker event loop for minutes on large candidate sets #3211).

[Sanderhoff-alt]

Update on the Open Question: Safely Raising _INTRABATCH_MAX_NAMES

Regarding the open discussion in this PR on whether _INTRABATCH_MAX_NAMES (currently capped at 250) can be safely raised to 500+ for large document ingest:

1. The Original Bottleneck & Downstream Risk

When #3991 was authored, _INTRABATCH_MAX_NAMES = 250 was kept as a safety cap because, although Prefix Filtering solved Stage 1 ((N^2)$ intra-batch clustering, completing =500$ in 4.3ms), passing =500$ un-truncated entities downstream to Stage 2 (_resolve_from_candidates) would trigger up to \times 200 = 100,000$ candidate scoring comparisons. Under the pure-Python difflib.SequenceMatcher implementation, scoring 100k pairs consumed ~850ms of synchronous CPU, risking event-loop stalls (#3211).

2. Downstream Bottleneck Resolved in #3995

With PR #3995 replacing SequenceMatcher with C++ SIMD-accelerated rapidfuzz (Indel.normalized_similarity, ~9.3M pairs/s throughput):

• 100,000 pairs ($N=500$ candidate limit): CPU time drops from 850ms $\rightarrow$ 10.0ms (85x speedup).
• 200,000 pairs ($N=1000$ candidate limit): CPU time drops from 1,740ms $\rightarrow$ 20.2ms (86x speedup).

3. Combined Pipeline Latency Profile

With both #3991 (Stage 1) and #3995 (Stage 2) in place, the complete CPU footprint scales sub-linearly:

Batch Scale ($)	Stage 1: Intra-batch Clustering (#3991)	Stage 2: DB Candidate Scoring (#3995)	Total Combined CPU Latency
$N = 250$ (current cap)	1.60 ms	~5.0 ms (50k pairs max)	~6.6 ms
$N = 500$ (proposed)	4.34 ms	~10.0 ms (100k pairs max)	~14.3 ms
$N = 1000$	12.65 ms	~20.2 ms (200k pairs max)	~32.8 ms

Conclusion & Recommendation

Raising _INTRABATCH_MAX_NAMES to 500 (or exposing it via config) is now completely safe from an event-loop responsiveness perspective ($< 15\text{ms}$ total CPU), allowing long documents, dense financial reports, and bulk imports to achieve full intra-batch entity deduplication without artificial truncation.

We can land #3991 and #3995 first, and follow up with a minor tune to lift _INTRABATCH_MAX_NAMES.