Time-series forecasting audit: inference-path gaps and example-correctness defects (follow-up to #1229, #1225, #1161, #1172–#1175)

[immu4989]

Background

The ts_forecast task has a cluster of stale open issues from 2023 (#1229, #1225, #1161, #1172, #1173, #1174, #1175). Before proposing work, I re-verified each one against current main (through 38457fba) with runnable reproductions, in the same spirit as the estimator reproducibility audit (#1540). Results below — several are already fixed and can be closed, several are confirmed live.

Track A — code

A1. Timestamp-in-index fails at fit with a misleading error (live, reproducible)

FLAML TS expects the time to be a column (time_col, defaulting to the first column). When users instead supply the timestamp as the DataFrame index — the idiomatic pandas shape, and exactly what the reporter of #1229 did — fit() silently takes the label column as the time column and raises:

ValueError: For '(...ts_forecast...)' task, time column y must contain timestamp values.

Repro on current main:

import numpy as np, pandas as pd
from flaml import AutoML

n, horizon = 120, 12
train_df = pd.DataFrame(
    {"y": np.sin(np.arange(n) / 6) * 10 + 50},
    index=pd.date_range("2018-01-01", periods=n, freq="MS"),
)
automl = AutoML()
automl.fit(dataframe=train_df, label="y", period=horizon, task="ts_forecast")
# ValueError: time column y must contain timestamp values

The message points the user at the wrong column and doesn't mention the index. The 2024 comment thread on #1229 confirms real users hit this and had to discover reset_index() by trial and error.

Proposed fix: in the ts_forecast data-validation path, detect a DatetimeIndex (or a RangeIndex alongside a missing/failed time column) and either (a) automatically reset_index() it into the time column with an informative log line, or (b) raise a targeted error: "time appears to be in the DataFrame index; move it into a column (df.reset_index()) and pass time_col=...". I lean (a) for the DatetimeIndex case and (b) for anything ambiguous.

A2. #1229 / #1225 predict-path failures — apparently fixed; propose regression tests + closure

The originally-reported failures (KeyError/ValueError at predict() with Fourier-feature names like index_sin1, index_hour_cos on prophet/arima/sarimax; docs-tutorial KeyError on 2.1.0) do not reproduce on current main. I ran: univariate monthly (lgbm/arima/sarimax), multivariate + exogenous (lgbm/arima/sarimax), and sub-daily hourly (lgbm/sarimax) — all fit+predict cleanly. The intervening TS refactor appears to have fixed the feature-replay-at-predict problem.

Proposed: add the three reproductions above as regression tests under test/, then close #1229 and #1225 referencing the tests (prophet variant to be covered in CI where prophet is installed).

A3. #1161 — TFT/panel path breaks on modern pytorch-lightning (live, environment)

Trainer.__init__() got an unexpected keyword argument 'gpus' on modern pytorch-lightning; a maintainer (int-chaos) acknowledged in 2024 that the pinned combination is pytorch-forecasting==0.9.0 + pytorch-lightning==1.5.10. Options: pin the [ts_forecast] extra accordingly, or update the TFT wrapper to the current lightning API. The former is a small pyproject.toml change; the latter is real work I'd scope separately after maintainer input.

Track B — documentation / examples (all four confirmed in current files)

#	Defect	Location	Verified
#1172	Data leakage in panel-forecasting example: avg_volume_by_sku is a groupby-transform over the full dataset before the train/test split	website/docs/Examples/AutoML-Time series forecast.md (lines ~730–735)	✅ present
#1175	Data leakage in Forecasting Discrete Values: above_mean_sales thresholded on the full-data mean before the split	notebook/automl_time_series_forecast.ipynb	✅ present
#1173	NameError: multivariate example builds multi_df but calls automl.fit(dataframe=df, ...)	same doc, line ~534	✅ present
#1174	Comment mismatch: del multi_df["month"] # remove temperature column…	same doc, line ~505	✅ present

The two leakage items matter beyond cosmetics — official examples teach the evaluation pattern users copy into production. The fix is to compute derived features within-fold / on train only (or state the simplification explicitly).

Proposed plan

Following the per-item-PR pattern from #1540, in this order:

1. PR 1 (Track B): fix all four doc/notebook defects in one docs PR (they're one file each, same subject) — closes seems to be you have forecasting-with-panel-datasets bug: data leaking #1172, error in example for multivariate-time-series-forecasting-with-exogenous-variables #1173, comment remove temperature column to reduce redundancy is not matching column name month #1174, leaking data in 4. Forecasting Discrete Values in automl_time_series_forecast #1175.
2. PR 2 (A2): regression tests for the fixed predict paths — closes ts_forecast - KeyError/ValueError in predict() #1229, Time-series bug when running tutorials on FLAML==2.1.0 #1225.
3. PR 3 (A1): DatetimeIndex handling + targeted error message, with tests.
4. A3: decide pin-vs-migrate after your input; separate PR either way.

Questions for maintainers

1. Any objection to closing ts_forecast - KeyError/ValueError in predict() #1229/Time-series bug when running tutorials on FLAML==2.1.0 #1225 via regression tests given they no longer reproduce?
2. For A1, do you prefer auto-reset_index() on DatetimeIndex (convenient, slightly magical) or a targeted error (explicit)?
3. For A3, is pinning the [ts_forecast] extra acceptable short-term, or is a lightning-API migration the only acceptable fix?

Happy to adjust scope; will start with PR 1 once this direction is confirmed.