From 2e7756f46a003ff568e21ef93a24a6c545b6ef70 Mon Sep 17 00:00:00 2001 From: Julien Cornebise Date: Tue, 28 Jul 2026 00:23:57 +0200 Subject: [PATCH] =?UTF-8?q?python-math=20#50:=20docs(delphi):=20condition?= =?UTF-8?q?=206=20FINAL=20=E2=80=94=20vectorized=20EC2=20comparison,=20ver?= =?UTF-8?q?dict=20serial-OK?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DONE condition 6 of `GOAL_CUTOVER_READY.md` (the EC2 performance measurement) finalized. Same r8g.4xlarge instance, conversation shape, and seed as the pre-vectorization run: cold tick 519.6s → 29.0s (~18x), warm tick 1856.0s → 26.6s (~70x). - `CUTOVER_RUNBOOK.md` risk item 3 now carries the final verdict — serial OK at every observed conversation shape; no blocklisting (none needed); item 9b (seeded sampled PCA for extreme shapes) is now optional — plus the historical measurement chain. - `GOAL_STATE.md` condition-6 line updated. - Final battery pair on this tree: the certification battery (20 recorded dataset entries replayed and compared against Clojure reference recordings) at 20/20 MATCH, twice. commit-id:9547faa9 --- delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md | 13 +++++++ delphi/docs/CUTOVER_RUNBOOK.md | 47 +++++++++++++------------ delphi/docs/GOAL_STATE.md | 9 ++--- 3 files changed, 43 insertions(+), 26 deletions(-) diff --git a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md index 680d0b4ee..1feae166b 100644 --- a/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md +++ b/delphi/docs/CLJ-PARITY-FIXES-JOURNAL.md @@ -4483,3 +4483,16 @@ prod starts services BY NAME via scripts/after_install.sh role dispatch tracks branch `stable` — so the shadow wiring is one edit to the math role's compose-up line, deliberately NOT made yet (Julien weighing shadow vs clean replace). + +### Condition 6 FINAL — EC2 comparison recorded, verdict flipped to serial-OK + +Vectorized run (i-03c84ff0b574cebfa, r8g.4xlarge, same shape/seed, +self-terminated + verified): cold 29.0s / warm 26.6s vs 519.6s / 1856.0s +non-vectorized — ~18x / ~70x. Runbook risk item 3 now carries the FINAL +verdict: serial OK at every observed shape; no blocklisting (none +needed); item 9b (seeded sampled PCA) optional. PRs #2679 (item 9a, +bit-identical) + #2680 (math-python rename) pushed; CI dispatched (run +30310377752); independent review in flight. Ops gotcha logged: SQS +completion messages need JSON parsing (tab-split receipt handles broke +delete → stale redelivery); the vectorized run's job label says +large-conv-tick (sed missed escaped quotes) — S3 key disambiguates. diff --git a/delphi/docs/CUTOVER_RUNBOOK.md b/delphi/docs/CUTOVER_RUNBOOK.md index 7aa209435..d73262cc3 100644 --- a/delphi/docs/CUTOVER_RUNBOOK.md +++ b/delphi/docs/CUTOVER_RUNBOOK.md @@ -36,28 +36,25 @@ equivalence evidence), CLOJURE_QUIRKS.md (Q1-Q19). (~30.9 min)** — the warm tick is ~3.6x the cold one (legacy kmeans lineage warm-start dominates). Local M-series cross-check: 430.8s / 2095.3s — same order, so this is algorithmic, not instance-bound. - **VERDICT: serial is NOT OK at the extreme shape** — the old 0.5-2 - min/tick estimate was an order of magnitude optimistic. A ~31-minute - tick would occupy a poller process/shard for its duration whenever one - of the 7 historical large convs receives votes. REQUIRED fix - (Julien ruling, s7: NO zid is ever blocklisted, and the warm start - STAYS — cluster-id stability across ticks is user-facing): item 9, - re-scoped as (a) VECTORIZE the warm-start k-means hot path — replace - the per-pair python _euclidean loop with per-center BLAS columns - (d2 = row_norms + |c|^2 - 2*(X@c), the same cancellation formula) in - cluster_step/most_distal/weighted_mean; bit-identity is the - acceptance bar (Q11 0.0-ties are load-bearing for cluster ids — the - vw every-vote step-57 tie test + the full battery gate it); plus - (b) a deterministic (seeded) sampled PCA for the extreme shapes. - SCOPE NOTE (Julien question, s7): Clojure's large-conv graph - overrides ONLY the :pca node (mini-batch PCA over an unseeded - 1500-row sample; conversation.clj:760-773 — large-conv-update-graph - merges small-conv-update-graph) — the k-means warm start is IDENTICAL - in both graphs, so there is no Clojure-side large-conv k-means - treatment to port; vectorz's JVM loops simply outran our per-cluster - Python port at 33k rows. The flip is NOT blocked: all 7 large convs - are historical and rarely active; if one ticks before item 9 lands it - is slow (~31 min) but correct and stable. + **VERDICT (FINAL, 2026-07-28 s7): serial is OK at every observed + shape.** Item 9a (vectorized warm-start k-means, PR #2679 — + bit-identical: exact-== pins vs the scalar reference, knife-edge Q11 + ties preserved, battery 20/20 x2) re-measured on the SAME r8g.4xlarge + / shape / seed: + cold tick 519.6s -> 29.0s (~18x) + warm tick 1856.0s -> 26.6s (~70x) + (local M-series cross-check: 430.8s -> 28.2s / 2095.3s -> 26.7s.) + A ~27s worst-case steady-state tick on the 7 historical giants is + compatible with the serial poller (~1.66 ticks/s on normal convs). + NO blocklisting (Julien ruling s7) — none needed. The deterministic + seeded sampled PCA (item 9b) is now OPTIONAL (further speedup / + Q10-class hygiene), not a throughput requirement. + Historical note: the pre-vectorization measurement (cold 519.6s, warm + 1856.0s, verdict then NOT serial-OK) drove item 9a; Clojure's own + large-conv path only ever special-cased :pca (conversation.clj: + 760-773), never k-means — the Python fix was vectorizing our port's + per-pair loops (~3.3M python calls/iteration -> batched-matmul BLAS + columns, bit-equal by construction and by 85-combo probe). Sharding (#2658) is the scale-out path, opt-in via POLL_SHARD_INDEX/ POLL_SHARD_COUNT — one shard = one process. Start UNSHARDED (defaults are a verified no-op); shard only if the shadow soak shows lag. @@ -90,6 +87,12 @@ from Clojure's env, rows invisible to the server (UNIQUE(zid, math_env)). ``` docker compose --profile math-python up -d math-python # env: MATH_PYTHON_ENV=python (engine has one path since the mode collapse) +# env: MATH_CONV_CACHE_CAP= — SET THIS FOR THE SOAK (s7): the conv cache +# never evicts by default; a long soak accumulates convs toward the 16g +# container limit and an OOM-kill restart loop. LRU eviction is cheap +# (reload = from_dict warm restore). Memory math: host 128 GiB; python +# capped 16g; clojure unchanged by shadow. Verify the clj container's +# actual -Xmx on the host before the soak (empirically fits today). # PROD NOTE (deploy-script reality, s7): prod instances start services BY # NAME from scripts/after_install.sh per-role dispatch (profiles are a # dev-only gate) — shadow on the math role = add `math-python` to its diff --git a/delphi/docs/GOAL_STATE.md b/delphi/docs/GOAL_STATE.md index 8a291e9d1..d8f4a2bcb 100644 --- a/delphi/docs/GOAL_STATE.md +++ b/delphi/docs/GOAL_STATE.md @@ -21,10 +21,11 @@ All seven DONE conditions hold, evidenced in-repo: 0 envelope-excused divergences. 6. EC2 measurement in CUTOVER_RUNBOOK risk item 3: r8g.4xlarge, 33,422×783/2.0M votes — cold 519.6s, WARM 1856.0s (~31 min). - VERDICT: NOT serial-OK at the extreme shape; fix = item 9 re-scoped - (vectorize warm-start k-means, bit-identity-gated + seeded sampled - PCA; NO blocklisting — Julien ruling s7); flip not blocked (slow but - correct+stable if a large conv ticks first). + VERDICT (FINAL): serial OK at every observed shape — item 9a + (vectorized warm-start k-means, PR #2679, bit-identical) re-measured + on the same r8g.4xlarge: cold 519.6s→29.0s, warm 1856.0s→26.6s + (~70x). No blocklisting (none needed); item 9b (seeded sampled PCA) + now optional. 7. This line 1 flip. ## For walkthrough (Julien)