compdemocracy · jucor · Jul 24, 2026
diff --git a/delphi/docs/MATH_POLLER_EQUIV_SPEC.md b/delphi/docs/MATH_POLLER_EQUIV_SPEC.md
@@ -0,0 +1,100 @@
+# Poller-equivalence harness — spec (goal condition 2)
+
+**Status:** spec — 2026-07-24 (session 5). Companion to `MATH_POLLER_DESIGN.md`
+(the py poller under test) and `GOAL_R1_PARITY.md` ("DONE means": *poller
+equivalence: identical math_main/bidToPid/ptptstats rows and tick/watermark
+semantics vs the Clojure math container on the same vote streams, including a
+restart-mid-schedule seam*).
+
+## 1. Shape
+
+One Postgres, two writers, one comparer:
+
+```
+scripts/poller_equiv.py  (CLI orchestrator)
+  ├─ seed:   create throwaway DB (polis_equiv) with the polis schema subset
+  │          (conversations, votes, comments, math_main, math_bidtopid,
+  │          math_ptptstats, math_ticks); insert conversation + comments;
+  │          votes are inserted in TIMED BATCHES by the driver loop below
+  ├─ clj:    the REAL container loop — clojure -M:run full (math/), env
+  │          DATABASE_URL=…/polis_equiv, MATH_ENV=clj-ref,
+  │          POLL_FROM_DAYS_AGO=10000 (historical vote timestamps)
+  ├─ py:     scripts/math_poller.py, same DB, MATH_ENV=py-shadow,
+  │          POLISMATH_ENGINE_MODE=clojure-legacy, same poll window
+  ├─ feed:   insert vote batch k → wait until BOTH math_envs' math_main
+  │          rows advance past batch k's votes (poll by caching_tick /
+  │          lastVoteTimestamp in the blob) → next batch. Batches mirror a
+  │          battery schedule's cuts (vw uniform8; pc-meta-02 uniform6-mod
+  │          for the moderation stream — comments.modified drives mod polls)
+  ├─ seam:   after batch R (mid-schedule), SIGKILL the py poller process,
+  │          restart it (load-or-init warm path — the from_dict restore
+  │          fixed 2026-07-24), continue feeding. Also restart the clj
+  │          container at the same seam for symmetry (its load-or-init).
+  └─ compare per batch k and per table:
+       math_main.data     → the SAME acceptance as certify (StepComparer:
+                            structural identity on memberships/ids/
+                            selections/priorities; declared float
+                            tolerances; subgroup-* excluded per Q7)
+       math_bidtopid.data → EXACT equality (bid→pids map)
+       math_ptptstats.data→ structural + tolerances
+       caching_tick       → per-env MAX+1 monotonicity (not cross-env equal —
+                            each env has its own sequence)
+       math_ticks         → equals the number of completed recomputes per env
+       watermark semantics→ each batch processed exactly once (no vote
+                            reprocessing: assert vote_counts in the blob
+                            match cumulative inserts at each cut)
+```
+
+## 2. Acceptance — the float bar
+
+The PRODUCTION clj container cannot be Q10/Q12-pinned (no Clojure source
+edits allowed): its cold-tick PCA start is unseeded-random, so even two clj
+container runs differ in float tails. The bar is therefore:
+
+1. **clj self-jitter envelope first**: run the clj side TWICE on the same
+   stream (fresh DB each). Per compared key, record the max cross-run
+   delta — the envelope. (H-A/H-B self-jitter pattern, journal 2026-07-18.)
+2. **py must sit within the envelope** (per key: |py − clj| ≤ envelope ×
+   safety factor 2, floor 1e-9) AND be STRUCTURALLY identical (memberships,
+   cluster ids, repness/consensus selections, priority ordering) to the clj
+   reference run.
+3. Report the envelope + verdict per (dataset, batch, table) in a compact
+   JSON verdict file (terse-output contract: ≤40 lines to stdout).
+
+Datasets: vw (knife-edge-free warm chain, certified 8/8) + pc-meta-02
+(mod/meta warm chain, certified 6/6). Both small → container runtime fine.
+
+## 3. Build plan (delegable, in order)
+
+- **A. schema + seeder** — smallest schema subset the clj container's
+  queries touch (poller.clj/postgres.clj: votes, comments, conversations,
+  math_main, math_bidtopid, math_ptptstats, math_ticks; check
+  db/load-conv's SELECT for exact columns). Seeder loads a replay dataset
+  (real_data loaders) and inserts conversation+comments; vote inserts
+  exposed as `insert_votes(conn, dataset, from_slot, to_slot)`.
+- **B. runners** — subprocess wrappers: clj container (env as §1; verify
+  `clojure -M:run full` works headless from math/ — bin/run wraps it),
+  py poller CLI. Health = row appears in math_main for the env.
+- **C. feeder + comparer** — batch loop, per-batch row snapshots, the
+  StepComparer adapter (math_main.data JSON ≈ the certify blob surface —
+  verify key overlap first; bidToPid exact; tick/watermark assertions).
+- **D. seam + envelope** — restart choreography, two clj runs, envelope
+  computation, verdict JSON.
+
+Integration gates at the top level after each stage; the harness lives in
+`polismath/replay/` + `scripts/` next to certify (same store/report
+conventions). Tests: unit-test the comparer adapter + watermark assertions
+with canned rows (no containers); the full harness is an opt-in script
+(RUN_POLLER_EQUIV=1), like the RUN_CLJ_INTEGRATION certify tests.
+
+## 4. Known hazards (from session-5 recon)
+
+- Postgres reachable at localhost:15432 via OrbStack pgproxy (socat →
+  polis-dev-postgres-1:5432); create polis_equiv there, NEVER touch
+  polis-dev / polis_prodclone.
+- Vote timestamps are historical → POLL_FROM_DAYS_AGO=10000 on BOTH sides.
+- The clj container writes ALL zids it sees in the window — the throwaway
+  DB isolates this.
+- 4h JVM self-reboot (bin/run timeout) — irrelevant at harness timescales.
+- polismath/poller/__init__.py "load-or-init finding" docstring is stale
+  (base_clusters DO restore since 2026-07-24) — refresh it in stage B.
diff --git a/delphi/polismath/conversation/conversation.py b/delphi/polismath/conversation/conversation.py
@@ -175,7 +175,7 @@ class Conversation:
     """
 
     def __init__(self, 
-                conversation_id: str, 
+                conversation_id: Union[str, int],
                 last_updated: Optional[int] = None,
                 votes: Optional[Dict[str, Any]] = None):
         """

diff --git a/delphi/polismath/database/postgres.py b/delphi/polismath/database/postgres.py
@@ -520,11 +520,30 @@ def poll_votes(
         # Execute query
         votes = self.query(sql, params)
 
-        # Format votes for processing, flipping sign at PostgreSQL boundary
+        # Format votes for processing, flipping sign at PostgreSQL boundary.
+        # pid AND tid are kept as the DB's native int (votes.pid/tid are both
+        # INTEGER) — NOT str()-wrapped. Found live (2026-07-24, poller-
+        # equivalence harness, session 2): the pid cast was the ONLY source
+        # of a Type-mismatch divergence in math_main.base-clusters.members
+        # against Clojure (which holds an int pid throughout) —
+        # Conversation.update_votes is deliberately type-agnostic at ingress
+        # ("Preserve original type", both pid AND tid) and
+        # raw_rating_mat/rating_mat are ALWAYS rebuilt fresh from these two
+        # methods on every load-or-init (never restored via from_dict — see
+        # polismath/poller/__init__.py's "load-or-init finding" docstring),
+        # so removing the cast is a one-point fix with no other code changes
+        # needed. Session 3 (same day): fixing pid alone left tid's OWN
+        # str() cast unmasked — a live vw full-run then showed the SAME
+        # Type-mismatch pattern on zid/tids[]/repness.*.tid, traced to this
+        # same cast. The certified/CSV replay driver never cast tid either,
+        # and matched clj int-for-int across 20 cross-validated entries —
+        # the evidence that authorized this fix. See also poll_moderation
+        # below, which needed the SAME fix for mod_out_tids/mod_in_tids/
+        # meta_tids/mod_out_ptpts to stay type-consistent with these two.
         return [
             {
-                "pid": str(v["pid"]),
-                "tid": str(v["tid"]),
+                "pid": v["pid"],
+                "tid": v["tid"],
                 "vote": postgres_vote_to_delphi(int(v["vote"])),
                 "created": v["created"],
             }
@@ -555,11 +574,14 @@ def poll_votes_since(self, since: int) -> List[Dict[str, Any]]:
             """,
             {"since": since},
         )
+        # pid AND tid kept as the DB's native int — see poll_votes's
+        # docstring/comment above for the full root-cause rationale
+        # (2026-07-24 live findings, sessions 2-3).
         return [
             {
                 "zid": int(v["zid"]),
-                "pid": str(v["pid"]),
-                "tid": str(v["tid"]),
+                "pid": v["pid"],
+                "tid": v["tid"],
                 "vote": postgres_vote_to_delphi(int(v["vote"])),
                 "created": v["created"],
             }
@@ -679,13 +701,22 @@ def poll_moderation(
         # Execute query
         mods = self.query(sql_mods, params)
 
-        # Format moderation data
+        # Format moderation data. tid is kept as the DB's native int — NOT
+        # str()-wrapped (2026-07-24 live finding, session 3): mod_out_tids
+        # feeds Conversation._apply_moderation's
+        # ``[c for c in self.mod_out_tids if c in self.rating_mat.columns]``
+        # intersection UNCONDITIONALLY (no engine-mode branch, unlike the
+        # participant-ban check below) — left str while poll_votes/
+        # poll_votes_since's tid became int, that intersection would ALWAYS
+        # be empty, silently disabling moderated-out comment zeroing in the
+        # live poller. poll_moderation_since (the OTHER, global-watermark
+        # variant) already used int(m["tid"]) and was never affected.
         mod_out_tids = []
         mod_in_tids = []
         meta_tids = []
 
         for m in mods:
-            tid = str(m["tid"])
+            tid = m["tid"]
 
             # Check moderation status with support for string values
             mod_value = m["mod"]
@@ -712,8 +743,15 @@ def poll_moderation(
         # Execute query
         mod_ptpts = self.query(sql_ptpts, params)
 
-        # Format moderated participants
-        mod_out_ptpts = [str(p["pid"]) for p in mod_ptpts]
+        # Format moderated participants. pid kept as the DB's native int —
+        # NOT str()-wrapped (2026-07-24 live finding, session 3): keeps this
+        # consistent with poll_votes/poll_votes_since's (also-int) pid, for
+        # Conversation._apply_moderation's ``p not in self.mod_out_ptpts``
+        # check ('improved' engine mode only — 'clojure-legacy' intentionally
+        # leaks bans and skips this check entirely, so this specific fix has
+        # no observable effect in the mode this harness runs in, but matters
+        # for 'improved' mode elsewhere).
+        mod_out_ptpts = [p["pid"] for p in mod_ptpts]
 
         return {
             "mod_out_tids": mod_out_tids,

diff --git a/delphi/polismath/poller/__init__.py b/delphi/polismath/poller/__init__.py
@@ -48,39 +48,79 @@
 (conversation.py:789) and ``_fold_base_clusters`` writes ``base-clusters.id`` /
 ``.members`` in that order (conversation.py:1643-1649), so
 ``[c['members'] for c in conv.base_clusters]`` is the exact alignment the server
-needs.  ``derive_bidtopid`` (math_writer.py) implements this.  Pids are strings
-Python-side (poll_votes casts ``str(pid)``) vs ints Clojure-side; the server
-parseInt()s them (participants.ts:53-55), so a parity comparer needs int/str
-tolerance on this one field.
-
-load-or-init finding (from_dict restoration is PARTIAL)
--------------------------------------------------------
-``Conversation.from_dict`` (conversation.py:2249-2303) restores from a dict with
-underscore/nested keys: ``last_updated, participant_count, comment_count,
-vote_stats, moderation{...}, pca{center,comps}, proj, group_clusters, repness,
-participant_info, comment_priorities``.  ``Conversation.to_dict`` (used as the
-math_main ``data`` blob) is a SUPERSET that carries those same underscore keys
-alongside the hyphenated Clojure keys, so ``from_dict(to_dict(conv))`` round-trips
-the listed fields — notably the PCA warm-start vectors and prior moderation.
-
-But ``from_dict`` does NOT restore: ``raw_rating_mat`` / ``rating_mat`` (the vote
-matrices), ``base_clusters``, ``subgroup_clusters``, ``group_clusterings`` /
-``group_k_smoother`` (warm smoother state), ``consensus`` or ``group_votes``.
+needs.  ``derive_bidtopid`` (math_writer.py) implements this.
+
+UPDATE 2026-07-24 (poller-equivalence harness live debugging, quirk finding):
+until this date, ``PostgresClient.poll_votes``/``poll_votes_since`` cast
+``str(pid)`` at ingress, while Clojure holds the DB's native int pid
+throughout — the server's ``parseInt()`` (participants.ts:53-55) papered over
+it, but it made ``bidToPid``/``base-clusters.members`` diverge bit-for-bit
+from a live clj container (confirmed: the CSV/certify replay driver never
+cast pid at all, and its blobs already matched clj int-for-int).
+``Conversation.update_votes`` is deliberately type-agnostic at ingress
+(``ptpt_id = vote.get('pid')``/``comment_id = vote.get('tid')  # Preserve
+original type``) and raw_rating_mat/rating_mat are ALWAYS rebuilt fresh from
+these two methods on load-or-init (never restored via ``from_dict`` — see
+this file's "load-or-init finding" section), so removing the ``str()`` cast
+was a one-point fix with no other code changes needed: pids are now native
+ints end-to-end, Python-side AND Clojure-side, and ``derive_bidtopid``'s
+``_normalize_bidtopid``-style int/str tolerance is now redundant
+defensive-coding for this field rather than a load-bearing requirement
+(kept — harmless, and guards a future regression).
+
+UPDATE 2026-07-24, same day (session 3): ``tid`` (and ``zid``) had the
+IDENTICAL bug, just masked by the sheer volume of pid divergences until
+session 2's fix above landed — a follow-up live vw full-run then showed
+``Type mismatch: golden=int, current=str`` on the top-level ``zid``, every
+``tids[i]``, and every ``repness.<gid>[i].tid``. Fixed the same way, same
+day: ``poll_votes``/``poll_votes_since`` no longer cast ``str(tid)`` either,
+``poll_moderation`` (the single-zid full-state variant — NOT
+``poll_moderation_since``, which already used int) no longer casts
+``str()`` on tid OR pid (needed for internal consistency once votes-side
+ids became int — see ``postgres.py``'s ``poll_moderation`` docstring for
+why a stale str-tid there would have silently DISABLED moderated-out
+comment zeroing), and ``polismath/poller/service.py``'s cold-start
+``Conversation(str(zid), ...)`` construction now passes the int through.
+The certified/CSV replay driver never cast tid (or zid) either, and matched
+clj int-for-int across 20 cross-validated entries — the evidence that
+authorized this follow-up fix. The scattered ``int(tid) if
+isinstance(tid, str) and tid.isdigit()`` idioms elsewhere in
+conversation.py are DEFENSIVE normalizers (no-ops on an already-int input),
+not evidence tid needed to stay a string.
+
+load-or-init finding (from_dict restoration is PARTIAL — updated 2026-07-24)
+-----------------------------------------------------------------------------
+``Conversation.from_dict`` (conversation.py:2818-2966) restores from a dict with
+underscore/nested keys: ``zid, last_updated, participant_count, comment_count,
+vote_stats, moderation{...}, pca{center,comps}, proj, group_clusters,
+base_clusters, group_votes, repness, participant_info, comment_priorities``.
+``Conversation.to_dict`` (used as the math_main ``data`` blob) is a SUPERSET
+that carries those same underscore keys alongside the hyphenated Clojure keys,
+so ``from_dict(to_dict(conv))`` round-trips the listed fields — notably the PCA
+warm-start vectors, prior moderation, base-cluster LINEAGE (id/members, unfolded
+exactly as Clojure's restructure-json-conv, conv_man.clj:171-186 ->
+clusters.clj unfold-clusters), and group-votes (needed by the recovery tick's
+comment-priorities calc, Q2, conversation.clj:658).
+
+As of 2026-07-24, ``base_clusters`` / ``zid`` / ``group_votes`` ARE restored
+(conversation.py:2905-2921 base_clusters, :2923-2952 group_votes) — this note
+previously said they were NOT; that was fixed to mirror Clojure's
+restructure-json-conv (conv_man.clj:173 keeps ``:base-clusters`` in the
+subset, :180 unfolds them) instead of re-deriving base-cluster lineage cold.
+
+``from_dict`` still does NOT restore: ``raw_rating_mat`` / ``rating_mat`` (the
+vote matrices — never touched anywhere in ``from_dict``) or
+``group_clusterings`` / ``group_k_smoother`` (warm smoother state), nor the
+dead ``subgroup_clusters`` / ``consensus`` paths (CLOJURE_QUIRKS.md Q7).
 Therefore load-or-init ALWAYS rebuilds the rating matrices from the full vote
 history (``poll_votes(zid)`` ordered by zid,tid,pid,created — parity with
-conv-poll offset 0) and recomputes base_clusters; the non-persisted smoother
-state cold-starts.  This is CLOSE TO — but not byte-identical with — a Clojure
-worker restart: on restart Clojure ``restructure-json-conv`` RESTORES
-``base-clusters`` (and the PCA) from the persisted blob before its ``:reboot``
-recompute (conv_man.clj:173 keeps ``:base-clusters`` in the subset, :180 unfolds
-them), whereas Python re-derives base_clusters cold
-from the vote matrices.  The rating-matrix rebuild itself matches
-(conv_man.clj:188-207 rebuilds ``raw-rating-mat`` the same way), and we
-opportunistically seed the warm PCA start from ``from_dict`` when a row exists
-(low-risk, literally what ``restructure-json-conv`` does).  The base-cluster
-lineage difference is a KNOWN divergence to trace against Clojure's ``:reboot``
-semantics before the parity gate; a full cold rebuild is otherwise correct —
-just without Clojure's restored-lineage warm start.
+conv-poll offset 0); the non-persisted smoother state cold-starts. The
+rating-matrix rebuild itself matches Clojure (conv_man.clj:188-207 rebuilds
+``raw-rating-mat`` the same way on restart). The remaining gap versus a true
+Clojure worker restart is narrower than before: only the non-persisted warm
+smoother state (group_clusterings/group_k_smoother) cold-starts — tracked as a
+KNOWN divergence to trace against Clojure's ``:reboot`` semantics before the
+parity gate.
 
 Config var mapping (config.py names PREFERRED, design aliases accepted)
 -----------------------------------------------------------------------