A CPU-only study of population and individual human preference modeling.
Main result: regularized known-user adaptation produced a small but reproducible prompt-disjoint improvement—a 0.879% relative reduction in negative log-likelihood—while wrong-user and shuffled-ID controls removed the advantage.
PMD asks whether a pooled human-preference model leaves stable person-specific signal, whether that signal improves prediction on unseen prompts, and how the predictive benefit changes with available user history. The answer is deliberately modest: person-specific preference structure is detectable and has real held-out predictive value, but the effect is small and develops gradually with additional user history. PMD is a reproduction-inspired diagnostic study, not a novel personalization algorithm, an LLM post-training project, or a demographic-personalization system.
The project follows one frozen evidence chain:
- Pooled modeling: construct a leakage-safe baseline and verify non-trivial held-out signal against shuffled labels.
- Heterogeneity: test whether pooled-model residual directions are more stable within the same anonymous annotator than across annotators.
- Personalization: fit a regularized user-specific deviation using train history only and evaluate it on prompt-disjoint test contexts.
- Sample efficiency: after the confirmatory gate, describe how much of the all-history benefit appears with 1–10 observed preferences.
Day 2B is the frozen confirmatory personalization test. Day 2C is explicitly exploratory and does not revise the Day-2B decision.
Population preference models compress judgments from many people into one shared function. That is useful only if the remaining disagreement is mostly noise. PMD separates three possibilities that are often conflated: stable identity-linked residual structure, out-of-sample predictive value for known users, and practical sample efficiency. A positive heterogeneity statistic alone is insufficient; personalization must improve held-out prediction and fail under wrong-identity controls.
PMD uses a deterministic 10,000-context subset of the English, pregenerated, first-turn portion of the first official Parquet shard of Meta's Community Alignment dataset v1.1. Each context contains one prompt, four candidate responses, one selected response, and an anonymous annotator ID. The bounded shard contains wave-1 data only.
Exact normalized prompts are split before any pair expansion, producing 7,135 train, 1,328 validation, and 1,537 test contexts. Day 1 expands each context into three winner-versus-loser pairs, yielding 30,000 dependent pairs. Demographic columns are neither loaded into the modeling frame nor used by any model. See the data audit for source revisions, filtering, schema, and license details.
Responses are encoded with train-fitted TF-IDF followed by 64-dimensional Truncated SVD. The Day-1 pairwise model estimates
P(left preferred) = sigmoid(wᵀ(h_left − h_right) + b).
The context-level model restores the original four-choice unit and estimates
P(y_i = j) = softmax_j(wᵀh_ij).
Personalization freezes the shared weights and representation, then fits only a user-specific ridge deviation from that user's prompt-distinct train history:
w_u = w + δ_u, with objective NLL_train(u) + λ||δ_u||².
One validation-selected global value, λ=0.1, is used for every user. No neural user embedding, low-rank repair, demographics, API model, or GPU training is involved.
The primary personalization estimand is equal-user mean ΔNLL(u) = NLL_personalized(u) − NLL_pooled(u) on prompt-disjoint test contexts. Negative values favor personalization. The frozen Day-2B population contains 94 known users with at least 10 distinct train prompts and at least two test contexts; median histories are 12 train and 3 test contexts.
Regularization is selected on validation users only. Eligibility, the lambda grid, primary estimand, controls, and calibration boundaries were frozen before personalized test evaluation. Day 2C reuses the exact population and test contexts with 50 deterministic nested subsamples at k ∈ {1,3,5,8,10}; its intervals and patterns are descriptive, not new discovery tests.
- Exact prompt groups are assigned to train, validation, or test before pair generation.
- TF-IDF and SVD are fit only on unique training-split response texts.
- Per-user deviations use only that user's prompt-distinct train contexts.
- Validation selects lambda; test metrics never enter tuning.
- Exact prompt, context, pair, and response-text overlap across splits is checked automatically.
- Frozen artifact hashes and deterministic reruns protect the Day 1–2C chain.
The pooled pairwise baseline reached 0.6174 accuracy, 0.6565 log loss, and 0.2318 Brier on held-out pairs; shuffled training labels reduced accuracy to 0.4784 and worsened log loss to 0.6957. In the heterogeneity stage, 179 users with at least 10 train prompts showed mean same-user residual cosine 0.0630 versus a 0.0072 permutation null, with observed-minus-null 0.0558 and user-bootstrap 95% interval [0.0354, 0.0782]. Randomized user IDs reduced the statistic to 0.0039.
The frozen personalization test produced the following canonical summary. Metrics use equal-user weighting over the 94-user primary population.
| Block | Metric | Value |
|---|---|---|
| Pooled | NLL | 1.339939 |
| Pooled | Multiclass Brier | 0.724521 |
| Pooled | Top-1 accuracy | 0.347036 |
| Pooled | ECE | 0.023753 |
| Personalized | NLL | 1.328159 |
| Personalized | Multiclass Brier | 0.718399 |
| Personalized | Top-1 accuracy | 0.359372 |
| Personalized | ECE | 0.004094 |
| Effect | Delta NLL | -0.011780 |
| Effect | Relative NLL improvement | 0.879% |
| Effect | Users improved | 67/94 |
| Control | Wrong-user NLL | 1.340911 |
| Control | Shuffled-ID improvement | -0.000583 |
The paired user-bootstrap interval for Delta NLL was [-0.017497, -0.006157]. This supports a real known-user predictive benefit under the frozen design, but its practical magnitude is small.
The post-gate exploratory curve is gradual and high-variance. Relative to the all-history gain, k=1 captured 7.7%, k=3 captured 23.0%, k=5 captured 35.4%, k=8 captured 56.4%, and k=10 captured 70.0%. Mean NLL improvement was monotonic, but accuracy and the number of improved users were not. At k=3, within-user variability across sampled histories was about 3.5 times the mean effect magnitude, showing that history informativeness matters alongside history quantity. Full details are in the Day-2C report.
The evidence chain includes shuffled labels for pooled modeling, cross-user permutation and randomized identities for residual stability, and wrong-user plus shuffled-history identities for personalization. Correct-user adaptation improved NLL; wrong-user NLL was 1.340911 and shuffled-ID improvement was -0.000583, both eliminating the gain. In the exploratory curve, wrong-user adaptation also worsened the pooled baseline at k=3 and k=10.
- The personalization effect is small: 0.011780 absolute and 0.879% relative NLL improvement.
- The predictive result covers known users with sufficient train history, not unseen or population-wide users.
- Test contexts are sparse per user; the median is three.
- The frozen lambda was the lower boundary of the preregistered grid.
- Day 2C is post-gate exploratory analysis on an already observed test set.
- The response representation is a fixed 64-dimensional linear text space.
- The dataset is a bounded wave-1 shard subset and is not documented as a random sample of all v1.1 data.
- Anonymous annotator identity may proxy stable unmeasured factors such as topic exposure, session, or interface behavior.
- The study supports no causal, demographic, deployment, or population-wide generalization claim.
Python 3.11 or 3.12 is recommended. The complete pipeline is CPU-only and requires no proprietary API or GPU.
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txtDay 1 — source audit, preparation, and pooled baseline:
python scripts\inspect_data.py
python scripts\prepare_day1.py
python scripts\run_day1_baseline.pyDay 2A — context-choice model and heterogeneity gate:
python scripts\run_day2a_choice_model.py
python scripts\run_day2a_heterogeneity.pyDay 2B — validation-only regularization freeze, one test evaluation, and controls:
python scripts\run_day2b_personalization.py --phase validation
python scripts\run_day2b_personalization.py --phase test
python scripts\run_day2b_controls.pyDay 2C — frozen exploratory sample-efficiency analysis:
python scripts\run_day2c_cold_start.pyFinal verification:
python -m pytest -q
python -m compileall -q src scripts testsWithout --source, the Day-1 inspection and preparation scripts download the pinned 368 MB official Parquet shard into ignored data/cache/. The raw shard is not committed. On the development CPU, source inspection and preparation each took under 10 seconds after download, the Day-1 modeling run took about 14 seconds, and later frozen analyses completed in minutes. Peak observed memory was below 600 MiB; allow approximately 1 GB RAM.
configs/ frozen pipeline configuration
docs/ data audit, stage reports, technical note, and CV evidence
figures/ three selected primary figures
results/ compact machine-readable metrics and control summaries
scripts/ one entry point per analysis stage
src/ data, representation, evaluation, and personalization code
tests/ leakage, identity, phase-order, and determinism checks
Start with the technical note for the complete narrative and CV evidence file for claim-safe summaries.
PMD supports the claim that, in this bounded known-user setting, pooled-model residuals contain stable anonymous-identity-linked structure and a frozen regularized adaptation extracts a small held-out predictive gain on unseen prompts. It also supports the descriptive conclusion that the gain develops gradually and is sensitive to which early histories are observed.
PMD does not establish a novel personalization algorithm, publication-level scientific contribution, LLM post-training method, demographic personalization effect, causal preference mechanism, unseen-user generalization, population-wide benefit, or deployment readiness.
The derived results use the Community Alignment dataset by Zhang et al., released under CC BY 4.0. This repository filters and transforms that material; redistribution should retain attribution, the license link, and an indication that modifications were made. When using the dataset in a publication, cite Zhang et al., Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Dataset (arXiv:2507.09650). The source revision, Parquet conversion revision, object hash, and filtering decisions are recorded in the data audit.
Original PMD code and documentation are released under the MIT License. That license does not replace or override the Community Alignment dataset's CC BY 4.0 terms. Dataset-derived results and figures retain the source attribution and modification notice above.