Data-leakage in ML examples

[raivo-otus]

issue forwarded from outside source

In the OMA chapter 21. on Machine Learning there are examples on machine learning workflows.

Examples show how to preprocess the data for a regression or classification task and the subsequent training and fitting of the ML model.

Data-leakage is nicely explained as in a callout block, but in the code examples data-leakage occurs!

The conceptual issue in the code:

1. Data preprocessing is done first with mikropml::preprocess_data() which by default does centering and scaling
2. Train/Test split and inner CV are done in the mikropml::run_ml() -function call

This results in data leakage as the the full dataset has been centered and scaled together before splits.

This same issue is in the mikropml documentation where preprocessing is done to the full dataset before model. The mikropml pattern of preprocess_data() followed by run_ml() appears to cause data-leakage problems.

Actionable steps

• Adjust OMA chapter code to avoid data-leakage by setting options in preprocess_data() to FALSE that are problematic (center, scale, nzv removal)

• Migrate to another ML package ecosystem that allows more explicit control, e.g. Tidymodels ?

• Raise this issue with mikropml directly as data-leakage appears structural to how the package and suggested workflow functions