Make walk-forward eval honest vs random walk (#387) - #391
Open
tsj2003 wants to merge 1 commit into
Open
Conversation
…yu-coder#387. Zero-shot Kronos-small was being judged as a single-name directional signal, while predict() averaged sample paths and hid the real forecast cone. Keep the mean path for compatibility, return quantiles from forecast(), and compare MAE/hit-rate/coverage against a last-price random walk. Co-authored-by: Cursor <cursoragent@cursor.com>
tsj2003
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Looked at #387. The backtests there (5m / 15m / 1h, T=1.0 and 1.5) showing no directional edge vs a random walk are pretty easy to reproduce, but I don't think that's quite the same thing the paper is measuring.
Two concrete problems on our side:
predict()averages all sample paths, then people plot high/low of that one mean path and call it a cone. That's why the cones look tiny. The spread has to come from the samples, not from the averaged bar.eval(). That's a silent footgun.What I changed:
predict()is unchanged for callers (still returns the mean DataFrame).forecast()keeps the sample paths and 10/50/90% quantiles.examples/walk_forward_eval.pywalks forward on a CSV and prints MAE / hit rate / coverage against a last-price random walk, plus a short interpretation. If there's no edge, it says so.I ran
tests/test_walk_forward_eval.pyand the existingtests/test_kronos_regression.py— both green. Didn't include anything unrelated.If zero-shot Kronos-small still has no edge on a given futures contract after this, that's probably a finetune / cross-sectional ranking problem, not something you fix by turning T up to 1.5.