feat(autodata): v2.2 — Evalchemy-style val-set objective for the meta-loop#50
Merged
Conversation
…-loop
The honest v1/v2.1 finding: live yield was noisy/low (best ~25%) and the root
cause is the EVAL SIGNAL, not the optimizer. The v2.1 reward — the per-batch Δ=1
keep-rate — re-samples fresh tasks every round (so it wobbles for reasons
unrelated to P) and rides exact-match arithmetic's thin ZPD. Optimizing that
proxy chases noise.
v2.2 swaps the proxy for a held-out, GT-graded LIFT objective (the Open-Thoughts/
Evalchemy move):
lift(P) = acc(weak | few-shot of D(P)) − acc(weak alone) on a FIXED val set
- valset.py: evaluate() + valset_lift() — score the weak solver on a fixed,
ground-truth-labeled val set, cold (baseline) and primed with a few-shot sample
of the round's kept teacher traces. In-context learning as a torch-free proxy
for fine-tuning lift; graded by the same trusted verifier the Δ-filter uses
(core.judge_correct → verify.py). Stable + comparable round-over-round.
- meta.py: meta_optimize(objective="valset") tracks best_lift/best_score, feeds
the lift trajectory to the prompt-optimizer, stops at val_target. The v2.1
objective="yield" path is the untouched default — nothing regresses.
- core.AutoDataConfig: objective / val_set / val_shots / val_target config keys.
- run.py: --objective valset CLI flag.
- Pure HTTP + stdlib only; fully offline-testable via injected fake clients
(valset.demo, meta.demo_valset, + pytest cases). Live endpoints are the
operator's to run.
- V2_DESIGN.md: v2.2 marked implemented, objective documented, RESULTS placeholder.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017NziXzqT1L9kj2T1byA3Ak
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the noisy v2.1 reward (per-batch Δ=1 keep-rate) with a held-out, ground-truth-graded lift objective:
lift(P) = acc(weak | few-shot of kept traces) − acc(weak alone)on a fixed labeled val set, graded by the same trusted verifier as the Δ-filter (the Evalchemy verifier role). In-context-learning lift as a torch-free proxy for fine-tuning lift — stable and comparable round-over-round, which raw keep-rate wasn't (the root cause of v2.1's noisy ~25%-best yields).valset.py(new):evaluate()+valset_lift()+ self-check demo.meta.py:objective="yield"|"valset"; early-stop acceptance gated on an exact one-sided McNemar test (stdlibmath.comb, paired on the same probes) +val_min_nguard (default 12, α=0.05) — review found the raw-proportion stop declared lift on noise with small n; reproduced live and fixed with regression tests.--objective valset,cfg.objective).Live fleet run + RESULTS recording follows post-merge (A7 acceptance).
🤖 Generated with Claude Code
https://claude.ai/code/session_017NziXzqT1L9kj2T1byA3Ak