Row anomaly detection: a second detector, and the end of segment_by - #1489
Open
vb-dbrks wants to merge 102 commits into
Open
Row anomaly detection: a second detector, and the end of segment_by#1489vb-dbrks wants to merge 102 commits into
segment_by#1489vb-dbrks wants to merge 102 commits into
Conversation
`_save_training_record` hand-rolled the `features.feature_metadata` JSON with a literal five-key dict, so `SparkFeatureMetadata.to_json()` was dead on the training path. Any field added to the dataclass would have been dropped silently at persistence, and the model would then score with a different feature set than it trained on — a silent train/score skew rather than a loud failure. Three changes, no behaviour change: - `_save_training_record` now calls `artifacts.feature_metadata.to_json()`, and `to_json` iterates the dataclass fields instead of naming them, so it stays complete by construction. Field order is declaration order, so the emitted JSON is byte-identical for existing models. - `from_json` ignores unknown keys instead of raising. It did `cls(**data)`, so a record written by a newer DQX version made an older reader fail outright; it now falls back to defaults for fields it does not recognise. - New `apply_feature_engineering_from_metadata` collapses the five call sites that each hand-passed `frequency_maps` + `onehot_categories` + `categorical_cardinality_threshold` back out of a `SparkFeatureMetadata` (`core.py` x3, `training_service.py`, `feature_prep.py`). Future metadata fields thread through one function rather than five. Groundwork for #1484.
Four documentation defects found while reading the anomaly module for #1484. Each was verified against the code; no behaviour changes here. - `drift_threshold`'s docstring claimed "default 3.0, None to disable". The signature default is `None`, and `None` disables drift detection outright (`drift.py:227` gates on `is not None`), so the stated default was the exact opposite of the real one. The reference docs already had this right. - Drift was described as watching the "score distribution". It compares the *input feature* distributions against the per-column baseline statistics recorded at training and reports which columns drifted (`compute_drift_score`); it never looks at anomaly scores. Corrected in the check table, the parameter reference, and the guide. Also documented the 1,000-row floor below which drift is skipped as too noisy to judge. - `ensemble_size` was documented as if it always applied. Segmented training passes `allow_ensemble=False`, so it trains exactly one model per segment regardless, which also means `confidence_std` is unavailable for segmented models. Said so on the parameter, in the reference table, and in the guide. - The guide advised "100+ rows per segment" without mentioning that segments below **10** rows are skipped entirely, leaving their rows unscored with no error. Documented the hard threshold alongside the guidance. Groundwork for #1484.
The numbers governing per-segment training were literals spread across the training service. Collecting them in `anomaly/group_config.py` gives each one a name and a reason, and makes the ceiling below reviewable as policy rather than as a magic number. MAX_SEGMENT_MODELS is the cost fuse: a 90-segment run was measured at 70 minutes without completing. MIN_ROWS_TO_TRAIN_SEGMENT and MIN_ROWS_PER_SEGMENT separate "too small to fit at all" from "small enough to warn about", which were previously the same literal used for two different questions.
Five checks, each guarding a failure that is silent rather than loud if it gets through:
the columns must exist; they must not also be feature columns; their types must render
identically in Python and Spark; the list must not be empty when declared; and it must not
duplicate an entry.
The type restriction is the one with teeth. Floating-point and decimal columns are rejected
because Python's `str()` and Spark's `cast("string")` disagree on them, which would produce
one key at training and a different key at scoring — and that mismatch does not raise, it
just misses every baseline lookup and silently falls back to the global baseline.
…ment
Training persists a baseline under a key built in Python; scoring looks it up with a key
built in Spark. A disagreement between the two does not raise — every lookup simply
misses, falls back to the global baseline, and produces a model that appears to work
while conditioning on nothing. That makes this the most expensive invariant in the
feature to get wrong, so both halves and the test that pins them land together.
`_as_spark_string` exists because `str()` is not a drop-in for Spark's `cast("string")`.
Booleans are the case that bites: Spark renders `true`/`false`, Python renders
`True`/`False`. The first real Spark run of this code asserted
`'true\x1f42' == 'True\x1f42'` and failed, which is precisely the class of bug this
contract exists to catch — and a stale unit test had been pinning the wrong behaviour.
Floating-point and decimal baseline columns are rejected rather than shimmed: their two
renderings diverge in ways no small helper can reconcile.
The separator is ASCII unit separator, a control character, so ordinary categorical data
cannot contain it. It is a separator and not an escaping scheme, and the docstring says
so: data that genuinely contains \x1f can still collide. Deliberately not solved by
length-prefixing, which would buy protection against pathological data by widening the
one invariant most expensive to get wrong. A test pins the documented limitation so
nobody mistakes it for collision-proofing.
Closes #1484. A value can be perfectly ordinary for the table and badly wrong for its own group. Row anomaly detection had no scalable way to express that: the only grouping mechanism was `segment_by`, which trains one model per group, so conditioning cost one model per group and a discovered 90-way grouping became an hours-long run. `baseline_by` adds, for each numeric metric, its deviation from that group's baseline: rel = signed_log(value) - signed_log(group_median(value)) on a **single** model, whatever the group count. Baselines are computed once at training, persisted in the feature metadata, and broadcast-joined at scoring. The log-ratio form is stable when a baseline is near zero and symmetric for halving versus doubling. The raw metric is kept alongside, so a globally absurd value stays detectable even where it is ordinary for its group. Severity is calibrated per baseline group as well, since a score that is extreme for one group is unremarkable for another, with the global calibration as the fallback for groups that have none. A row whose group was never seen at training is reported rather than scored. The alternatives are both silent and both wrong: a high-cardinality frequency map returns 0.0 for an unseen key, which reads as "perfectly normal", and a one-hot encoding leaves every indicator at zero, which reads as extreme. `is_new_baseline` and `new_baseline_key` name the case instead. This widens the `_dq_info` struct, so appending to an existing results table needs `mergeSchema`. Three properties worth calling out, because they are what makes this safe to enable: * **Baseline columns are never features.** They are the basis of comparison, not a metric being compared, so they are excluded before the sklearn pipeline and the inferred MLflow signature see anything. `prepare_training_features` projects to the feature list explicitly; the key column rides through as a passthrough for the calibration stage, which runs downstream of where the raw group columns are dropped. * **`segment_by` keeps its exact behaviour.** `_resolve_grouping` clears `baseline_by` on that path. Leaving both set would compute relative features *inside* each segment, where the key is constant because the frame is already filtered, while `compute_config_hash` is built from `segment_by` alone and would not change — an identical hash with a different feature list, which is a silent train/score hazard. * **Models trained before this change score identically.** Every new metadata field defaults to empty, so the relative transform returns immediately and `engineered_feature_names` is byte-identical to what it was. There is deliberately no strategy selector and no heterogeneity gate. `baseline_by` declared means relative features, always. The evidence for that choice, including the measurement that removing the gate costs nothing, is in `benchmarks/anomaly_conditioning/`. Also folded in here, because `config.py` and `training_service.py` carry it in the same hunks: **the segment model ceiling now errors rather than warns.** Above the ceiling the previous behaviour was to log a warning and continue, which meant a 90-segment run started training 90 models and was measured at 70 minutes without completing. A warning that precedes an hour of unusable work is not a warning. It now raises, and the message names the two ways forward — segment more coarsely, or raise `max_segment_models` explicitly — because an error that only says no is a worse experience than the warning it replaces. `max_group_models` is renamed `max_segment_models`. Its only consumer is the legacy `segment_by` path, the only path that trains N models, so it is a cost fuse on that path rather than a general tuning knob. The name now says which mechanism it bounds. `AnomalyParams` cannot import from the anomaly package — `anomaly/__init__.py` raises without the anomaly extra — so the default is a literal, kept in step with `MAX_SEGMENT_MODELS` by a unit test rather than by hope.
Predates this feature — `git log -- tests/perf/test_anomaly_benchmark.py` last touches it in #1129/#990 — and is separable from it. It lands here rather than as its own PR because the anomaly benchmark is what this branch changes the behaviour of. `generate_md_report.py` has always contained a fully written "Anomaly Benchmarks" section with ROC-AUC, precision, recall, F1 and precision@N columns. It has never rendered. The cause was one missing line: neither benchmark carried `@pytest.mark.benchmark(group="anomaly_synthetic")`, so pytest-benchmark recorded `group: null` and the report, which filters on exact equality with `anomaly_synthetic`, found nothing. `baseline.json` held 213 benchmarks and zero anomaly entries. Also fixed, all found while making the section actually appear: * The two benchmarks would have listed **twice** once the marker worked — once in the main results table, which is built from every benchmark, and once in their own section. * Five rounds of a Databricks-backed train-and-register cycle ran per nightly, twice over since the job invokes pytest again for the comparison step. Converted to `benchmark.pedantic(rounds=1)`, which takes its round count explicitly; verified against `--benchmark-min-rounds=5` and confirmed to record one round. * A module-global `_TRAINED_MODEL` with `needs_training` fallbacks made the score benchmark order-dependent and able to train up to three times. Each test now trains its own model as unmeasured setup. * The score benchmark read `_dq_info`, which does not exist on the frame `has_no_row_anomalies` returns — that column is assembled a layer up by `DQEngine`. So the test failed and recorded none of the quality numbers this section exists to publish. It now reads the struct column the check actually returns. Quality is published as *indicative and first-observed only*, and labelled as such. The nightly's baseline merge keeps the existing entry on conflict, so once a benchmark exists its `extra_info` is frozen at first observation and never refreshed — publishing quality that way publishes a fossil. Quality regressions are gated by assertions in `tests/integration_anomaly/` instead. The anomaly module is marked `pytest.mark.anomaly` and excluded from the timing gate. `--benchmark-compare-fail=mean:25%` is global and cannot be scoped per test, and these benchmarks are dominated by MLflow and Unity Catalog control-plane latency, so they would trip it on variance alone. They still produce a baseline; only the comparison skips them.
Reimplements the transform in numpy and fits sklearn's IsolationForest on raw columns and on raw-plus-relative columns, so the claim behind #1484 is checked in the unit suite in under five seconds with no Spark session, no workspace and no MLflow. Measured: PR-AUC 0.0028 to 0.6962 on a contextual collapse, against a 0.0026 base rate -- so the pooled model is at chance, not merely worse. The reverse case is asserted not to regress: an anomaly already extreme against every group stays at 1.0000 either way. This guards the *mechanism*, not the pipeline, and the docstring says so. The pipeline is covered by tests/integration_anomaly/, which needs a real session.
No Spark path in this feature had ever been executed before these tests existed, which was by far its largest outstanding risk: the baseline-key agreement, the relative transform, the broadcast joins, per-baseline severity and unseen-baseline marking were verified only by inspection and by unit tests that never start a session. Running them found real defects that inspection had not: * Training any grouped model raised `UNRESOLVED_COLUMN`. Feature engineering drops the raw group columns so they cannot reach the sklearn pipeline or the inferred MLflow signature, but training-time severity calibration runs downstream of that, on the scored engineered frame, and rebuilt the key from columns that were no longer there. Fixed by computing the key once and reading it thereafter, via `with_baseline_key`. * The Python and Spark halves of the baseline key disagreed on booleans. * A discovered grouping trained one conditioned model where the test still expected one model per segment — the intended behaviour change, asserted the old way. `test_anomaly_quality.py` measures detection quality rather than shape: it trains a conditioned and an unconditioned model on identical data and asserts conditioning wins by a margin, beats a random and a max-abs-z baseline, and does not concentrate false positives in one group. Metric discipline lives in `quality_metrics.py`, which computes no point-adjusted F1 and says why. Two measurement flaws in those tests, both fixed here and both worth recording because they would have made the assertions meaningless rather than wrong: * Average precision over a single planted positive is a coin flip, so the fixture now plants an incident across several groups and days. * A per-group false-positive rate over four rows can only be 0, 0.25, 0.5, 0.75 or 1, so the maximum across ninety groups reached 1.0 by chance and measured the fixture rather than the model. The fixture now emits enough control days for the rate to exist, the metric ignores groups too small to express one, and the test asserts the statistic was actually measured instead of silently NaN. The severity assertion also matched on country alone, where one country spans fifteen groups and only one collapsed, so `first()` returned an arbitrary sibling — passing at 95.5 on one run and failing at 82.3 on the next with identical data. It now matches the whole key, built with the production function so the assertion depends on Python/Spark agreement too.
`benchmarks/anomaly_conditioning/` compares three ways of relating a model to a group — pooled, baseline-relative, and one model per group — across 1,395 configurations: a synthetic two-factor sweep plus the Server Machine Dataset and NSL-KDD, 15 seeds each, paired by seed and tested with Wilcoxon signed-rank. Run manually rather than nightly: it downloads third-party data and yields a correlation rather than a pass/fail. Datasets are fetched at run time and cached, never vendored, which is what keeps the licensing position simple. SMAP and MSL are excluded — "(c) Original Authors", no permissive licence. ADBench is unusable here despite being the obvious choice: it ships pre-processed numeric matrices, so the categorical identity this experiment measures is already gone. The results, in `docs/dqx/docs/reference/anomaly_detection_quality.mdx`: * Contextual anomalies: baseline-relative beats pooled by a median **+0.0734** PR-AUC. * Globally extreme anomalies: **+0.0000**, and the test does not reject (p = 0.20). Not "costs little" — costs nothing. At low heterogeneity every group median approaches the global median, so the relative feature degenerates into a monotone transform of the raw metric: a near-duplicate of an informative column rather than noise. * Real datasets: **-0.0010** at p = 0.003. Negligible in size, real in sign, and reported rather than buried — SMD and NSL-KDD anomalies are largely globally extreme or sequence-dependent, so the extra feature dilutes slightly without adding signal. * Baseline-relative beats one-model-per-group in **every** mechanism, on real data included, while training one model instead of twelve. That asymmetry — about +0.07 where conditioning applies, about -0.001 where it does not — is the actual argument for auto-discovery enabling it rather than requiring opt-in. **On the heterogeneity gate.** The decision rules were fixed before the first run: no gate if the worst delta below eta-squared 0.10 stayed above -0.01, gate needed if any such cell reached -0.02. The pre-registered rule fired. It is a minimum over single cells, which makes it maximally sensitive to estimator variance, and it fired on one seed of `nslkdd/protocol_type` whose other seeds were +0.1003 and +0.0989. Rather than rewrite the criterion to get the preferred answer, the rule is left in place and reported, with the per-grouping median published beside it and the disagreement explained in the code. Raising the seed count fivefold moved no grouping's median below zero. No grouping is systematically harmed at low heterogeneity, so the gate had nothing to gate on. Eta-squared is not useless: it correlates with the size of the benefit (Spearman rho +0.597, bootstrap CI +0.523 to +0.668, still +0.276 after controlling for anomaly mechanism). But it predicts how much you gain, never whether you lose, and a gate needs to identify harm. It cost a full Spark aggregation per training run to answer a question that never changed the decision. `scipy` joins the existing mypy `ignore_missing_imports` list, alongside pandas, sklearn, shap and mlflow — the harness needs Wilcoxon and Spearman, and hand-rolling statistical tests invites subtler errors than the missing stubs do.
The guide gains a "Baseline conditioning" section built around the one mechanism there is: what `baseline_by` compares against, that it costs one model however many groups you have, that baseline columns are the basis of comparison rather than metrics to be compared, and what happens to a group that was never trained on. The measured numbers are quoted with a link to `anomaly_detection_quality.mdx` for the full sweep and, more importantly, for what they do not mean — DQX scores rows independently, so its figures on time-series benchmarks are a different task, not a worse implementation. `segment_by` is documented as legacy with the reason rather than a bare recommendation. On the Server Machine Dataset per-entity models were the worst of three configurations, and one entity produced 15,963 false positives across 28,392 normal rows. Each per-group model calibrates its own contamination on its own rows, so a group containing nothing unusual still has its most-unusual few percent scored as extreme. Four breaking changes, of which the second is the one existing users will notice: * Above the segment ceiling, training now raises instead of warning. * **An auto-discovered grouping now trains one conditioned model instead of N per-group models**, so those runs produce different scores. * A row whose group was never seen at training returns a null score and is reported via `is_new_baseline` rather than being scored 0.0. * The `_dq_info` struct is wider, so appending to an existing results table needs `mergeSchema`.
1,395 cells: the synthetic two-factor sweep plus the Server Machine Dataset and NSL-KDD, 15 seeds each, stamped with the commit they were produced from so the numbers are reproducible rather than merely asserted. Committed as its own change because the results are an observation, not code. Re-running the harness at the same seeds reproduces this file, which is how the determinism claim in the README is checked; the figures quoted in docs/dqx/docs/reference/anomaly_detection_quality.mdx are read directly from here.
Ten ADBench datasets (BSD-2-Clause, redistributing the ODDS / UCI / Kaggle collections), chosen to span the axes that actually change a detector's behaviour: 1.8k to 285k rows, 9 to 100 features, and base rates from 0.17% (the Kaggle credit-card set) to 40%. These answer a different question from the rest of the harness. SMD and NSL-KDD exist to test whether conditioning on a group helps; these have no grouping at all, so they characterise the ungrouped path -- which this branch does not change -- and serve as its regression baseline. Reported per dataset against its own random floor and its own max-abs-z baseline, never pooled into a headline: PR-AUC moves with the base rate, so 0.19 at a 0.17% base rate (a 99x lift) and 0.48 at 40% are not comparable numbers. An earlier draft of this harness rejected ADBench outright because its preprocessing discards categorical column identity. That is disqualifying for the grouping question but not for plain tabular data, so the exclusion was too broad.
…ted one Auto-discovery routed a discovered grouping to `baseline_by` but kept the selection policy that only ever made sense for `segment_by`: take a single column, the lowest-cardinality candidate, requiring 100 rows per group. Both rules exist because a per-segment model has to train a forest on each group. Under baseline conditioning there is one model however many groups there are, and a group only has to yield a median. Measured on a dataset grouped by country x event_type x product -- 90 groups available: discovered baseline_by [product] -> [product, event_type, country] groups 3 -> 90 training.columns country, event_count, event_type -> event_count engineered features 13 (11 one-hot dummies) -> 2 PR-AUC, contextual 0.1302 -> 0.5703 PR-AUC, global 0.2702 -> 1.0000 The feature-list change falls out of the grouping fix rather than being a separate decision: the profiler already drops the chosen grouping from the feature columns, so recognising country and event_type as dimensions stops them being one-hot encoded as metrics. Eleven dummies alongside a single real metric were diluting the isolation signal, which is why the global scenario was scoring 0.2702 where explicit columns scored 0.7200. `select_baseline_columns` is public so its policy can be unit-tested directly rather than through a Spark session; `MIN_ROWS_PER_BASELINE_GROUP`, `MAX_BASELINE_GROUPS` and `MAX_BASELINE_COLUMN_CARDINALITY` sit beside the segmented thresholds with the reason they differ. The legacy segmented policy is untouched. Also closes a latent hole on the explicit-columns path: a discovered baseline column that the caller had named as a feature is now excluded, rather than producing the feature-and-baseline overlap that `validate_baseline_columns` rejects.
The page carried only `AvailableSinceVersion`, and the lifecycle reference is explicit that "a feature with no status badge is generally available" — so for three releases the docs promised semantic versioning and backward-compatibility guarantees for a feature that was Experimental and is now Beta. DQX Studio is tagged experimental, actions and the MCP server beta; this page was simply missed. The distinction is not cosmetic. It is what makes the breaking changes in the following commits legitimate: Beta states that "the API may still change in backward-incompatible ways between releases", where GA would not.
The hash was f(columns, segment_by). Retraining under the same model_name with a different baseline_by therefore produced an identical hash, so the single thing this hash exists to catch -- same name, different configuration -- was blind to the grouping, even though the grouping changes the feature list and the persisted baselines. baseline_by now joins the inputs, which changes the hash of every configuration including ungrouped ones, since the key is present either way. A model registered before this fails the configuration check in score_global_model and must be retrained. That break is the point rather than a side effect. Silently scoring a model whose persisted hash no longer describes its configuration is worse than refusing to, and the error now names retraining as the remedy and says which version moved the goalposts. Row anomaly detection was Experimental through 0.16.0 -- "API, behavior, and on-disk or table formats may change ... without notice or a migration path" -- so no compatibility was owed here. At scoring, baseline_by is read back from the persisted feature metadata rather than taken from the caller: it is a property of the trained model, not a scoring argument. That makes the recomputed hash match for anything this version trained, and mismatch for anything older, which is exactly the intended boundary.
RobustScaler sat in front of the IsolationForest and could not have changed a single prediction. It is an affine per-feature transform, and Isolation Forest splits on per-feature thresholds drawn uniformly between each feature's min and max, so the induced partitions are identical with or without it. Measured rather than argued, across five ADBench datasets at five seeds each -- PR-AUC with the scaler and without it agreed to four decimal places every time: covertype 0.0572 / 0.0572 shuttle 0.9789 / 0.9789 mnist 0.2740 / 0.2740 fraud 0.1926 / 0.1926 cardio 0.5766 / 0.5766 What it did cost: a fit and a transform on every training run, a transform on every scoring pass, and bulk inside every pickled artifact. On every path, grouped or not. The single-step Pipeline stays. named_steps["model"] is how the SHAP explainer reaches the tree model, keeping it avoids churning eight -> Pipeline annotations across core.py and ensemble_training.py, and it leaves a slot for a transform that genuinely does something. One real hazard fixed alongside: the second SHAP call site indexed named_steps["scaler"] directly, so it would have raised KeyError on any model trained by this version. It now looks the step up, which also keeps working for older models that do carry a scaler.
baseline_by lived only inside the features.feature_metadata JSON blob. That is where scoring reads it, and functionally it was enough -- but it made the first question worth asking when detection looks wrong unanswerable in SQL. That cost real time this cycle. Diagnosing why zero-config conditioning was not engaging needed a script to pull a model record, parse the JSON, and print baseline_by and the size of baseline_medians, just to tell "conditioning never engaged" apart from "conditioning engaged on a 3-group basis where 90 were available". Both look identical from the outside: one model, is_global_model true. It now sits in the segmentation struct beside training.columns, so: SELECT identity.model_name, training.columns, segmentation.baseline_by FROM <registry> WHERE identity.status = 'active' answers it. The metadata blob stays the source of truth for scoring; this is a projection for humans, and the duplication is noted where the field is declared. The registry write gains mergeSchema. Without it a table created by an earlier DQX would reject the wider struct, which matters more than usual right now: the configuration-hash change tells users to retrain, retraining writes to their existing registry table, and a remedy that fails is not a remedy.
The harness fitted IsolationForest(n_estimators=100, contamination="auto") while DQX ships IsolationForestConfig(num_trees=200) with contamination taken from expected_anomaly_rate (0.02 by default). So every absolute figure it published described a lighter model than the product. contamination is the harmless half: it only shifts the predict/offset_ threshold and cannot reorder score_samples, so PR-AUC was unaffected by that difference. Tree count is not neutral -- a held-out measurement at 200 trees put fraud at 0.2607 where the 100-tree fit-and-score run had reported 0.1926. Paired comparisons were never affected, since the configuration was held constant on both sides of each comparison. It is the absolute per-dataset column that was understated. Found while mapping the end-to-end heuristic flow, which is the sort of thing that exercise is for.
Isolation Forest loses to a four-line z-score on covertype and mnist under every hyperparameter tried, so the gap is inductive bias rather than tuning. The obvious remedy is to put a cheap complementary scorer in the ensemble. This commits the measurement that says not to, so the idea is not re-proposed from scratch and re-derived by whoever notices the same gap next. Held-out splits (70/30), 5 seeds, the shipped forest configuration, each member calibrated to percentiles against its own training distribution: mean-of-percentiles wins 5/10 median -0.0028 worst -0.1409 (thyroid) max-of-percentiles wins 3/10 median -0.0253 worst -0.0690 It helps exactly where predicted (covertype +0.0295, mnist +0.0724) and hurts where the forest is genuinely stronger (thyroid, fraud, shuttle, spambase, satellite). A coin flip on average with a 0.14 worst case is not a default. Two things the script records that are easy to miss when re-litigating this: * Naive blending is not even implementable as-is. DQX averages ensemble member scores raw and the scales are incomparable -- the forest sits around 0.4-0.7, max-abs-z is unbounded -- so a real version has to persist each member's training-score distribution. Ranking inside the scoring UDF would be wrong, since the UDF sees a pandas batch and ranks would depend on partitioning. * Choosing the better detector per dataset requires labels, and DQX is unsupervised. So the honest options are to expose the choice or document the limitation; guessing is not one.
Two dev pages, both written because this cycle found its defects by accident and that is not a repeatable way to work. **anomaly_heuristic_map** — every decision made between train(df) and a flagged row, as a mermaid flowchart, with the governing default at each stage and the three questions that isolate most failures: what was chosen (stages 1-2, which decide silently and held both defects found while drawing this), did conditioning actually engage and on what, and is the comparison even valid given that a 30% sample drawn with .sample is partition-dependent under Spark Connect. It also records the measured weak points rather than only the design. Drawing it immediately surfaced that the harness modelled a 100-tree forest where DQX ships 200, which is the kind of thing the exercise is for. **anomaly_compatibility** — every affordance that exists only to keep old behaviour alive, marked so `git grep "COMPAT(anomaly-v1)"` finds them, with the registry-schema item called out because it touches persisted data rather than code. It deliberately separates that debt from permanent runtime fallbacks -- unseen-group and missing-quantile handling look identical in a diff and must not be deleted alongside.
The sweep in this page measured mechanisms against each other; it never measured this build against the one users are running. That is the question that actually matters, and the answer now leads the page: v0.16.0 and this build, through the real Spark/MLflow/Unity Catalog path, over the same Delta tables. contextual, zero config 0.1885 -> 0.5703 contextual, explicit columns 0.0376 -> 0.5703 global, zero config 0.3743 -> 1.0000 global, explicit columns 1.0000 -> 1.0000 Better on three, identical on the fourth, worse on none. The 0.0376 -> 0.5703 row is the cleanest statement of what conditioning buys, since v0.16.0 has no conditioning mechanism at all; the two exact 1.0000s are the regression check, on the path this work does not touch. The page also records how the measurement went wrong the first time, because it will catch anyone who repeats it: training samples 30% of rows by default via .sample, which under Spark Connect depends on partition ordering, so an uncontrolled run had the two builds training on different rows and showed the new build losing three of four cells. With sample_fraction=1.0 every cell reproduces to four decimals. A single-point comparison of this pipeline is not evidence.
Regenerating the sweep with the corrected 200-tree harness reversed a published claim, so it is
corrected here rather than left standing.
baseline-relative minus pooled, real datasets
was (100 trees): -0.0010 p = 3.05e-03 "marginally worse"
now (200 trees): +0.0025 p = 1.75e-02 marginally better
The page keeps the old number and says why it moved: the harness fitted a lighter forest than DQX
ships, which understated it on exactly the datasets where the forest does the work. The contextual
(+0.0742) and global (+0.0000) rows barely moved, because a paired comparison holds configuration
constant on both sides — it was the real-data row, where the two mechanisms are closest, that a
fidelity gap could flip. The guide and changelog carried the same figure and are updated too.
Also adds the plain-tabular section the sweep has been producing but the page never showed: ten
ADBench benchmarks spanning 1.8k to 30k rows, 6 to 100 features, and base rates from 0.17% to 40%.
DQX beats the random floor on all ten and max-abs-z on eight, with fraud at 115x the floor.
The two it loses get their own subsection rather than a footnote. covertype (0.0534 vs 0.1122) and
mnist (0.2766 vs 0.3367) are Isolation Forest's inductive bias: it splits one random feature at a
time, so a single extreme value among few dimensions is diluted and in 100 dimensions the split
rarely lands on the informative axis. Tuning does not recover it and the obvious knob trades shuttle
and cardio away for fraud. Publishing the losses with the diagnosis is more use to a reader than a
table of wins, and anyone with ADBench reproduces covertype in three lines regardless.
Supersedes the two partial results files from earlier runs.
The page had accumulated material that existed to convince us the work was an improvement, not to help anyone use the feature: a build-versus-build comparison against v0.16.0, the story of getting that measurement wrong the first time, a note about which harness revision reported which number, and the design rationale for a heterogeneity gate users never saw. That belongs in the changelog, the dev docs and the PR, and it is already in those places. 265 lines down to 151. What a reader actually needs is now the shape of the page: whether anomaly detection suits their data, whether to give it a grouping, and what it is bad at. The conditioning tables are reframed from "mechanism A versus mechanism B" to "your anomalies are like this, so do that", and the underperformance section ends with what to do instead -- a range check or an outlier rule serves single-column extremes better, and both can run together. Kept, because they are the reader's protection rather than our defence: rows are scored independently so time-series comparisons are a category error, PR-AUC moves with the base rate so the columns are not comparable, no point-adjusted F1, and every figure measures separability rather than generalisation.
The detection-quality numbers sat in a page nobody regenerated, so they would go stale the first
time anyone changed the model. They now refresh with the timing benchmarks and ride the same PR.
Not routed through pytest-benchmark, for a specific reason. The nightly merges baseline.json with
if ($old[.] != null) then $old[.] else $new[.] end
so an entry that already exists keeps its previous value entirely -- which is why no existing
benchmark mean has changed across the last four baseline PRs. Quality published through
extra_info would be frozen at first observation and republished as fresh for ever. emit_docs.py
writes the numbers into the page directly instead.
Generated tables sit between HTML comment markers, so regeneration replaces the numbers and
leaves the prose. That is what makes this page different from benchmarks.mdx, which
generate_md_report.py rewrites wholesale and where hand-written guidance therefore cannot
survive -- the reason the two are separate pages rather than one.
Scoped for CI at 5 seeds rather than 15, and marked continue-on-error: the real datasets are
~250MB fetched from GitHub on a cold cache, and a third-party download failing should not fail
the timing benchmarks that already ran. The trade to be aware of is that the nightly now depends
on an external host it did not before.
emit_docs warns rather than stays quiet when handed a results file with no tabular data, since a
stale table that looks current is worse than an obvious gap.
SHAP contributions are keyed by *engineered* feature name, and redaction filters those keys by exact match against the caller's redact_columns. Baseline conditioning adds a derived feature per metric, so redacting `amount` did not stop `amount_rel_baseline` -- a signed log-ratio of the same column -- from being embedded in a prompt and sent to an external serving endpoint. Naming a column sensitive has to mean everything computed from it is sensitive, so the redaction set now expands to cover the derived feature. Expansion is by exact suffix rather than prefix match, because prefix matching would also swallow `amount_paid` -- a different column the caller never named -- and quietly narrow what an explanation can discuss. The suffix is now a named constant in transformers.py, where the feature is created, rather than an inline f-string, so the two cannot drift apart. Known gap, documented at the function and not closed here: one-hot and frequency-encoded features are still not covered, because their names cannot be reconstructed from the source column alone -- `country` becomes `country_C3`, `country_DE`, one per observed value. Closing that needs the feature metadata at prompt-construction time, which the explainer does not currently receive. That hole predates this branch; this commit closes the one this branch opened. redaction_set is public so its policy can be unit-tested directly rather than through a private.
…line_by grouping Row anomaly detection was Experimental through 0.16.0 and owed no compatibility. This removes the per-group model path (segment_by / max_segment_models) entirely, leaving a single pooled baseline_by model that measured better on detection, reliability and cost across all four benchmark regimes. Source removals: - config.AnomalyParams.segment_by / max_segment_models - training_service: segmented dispatch, segment resolution and validation - scoring_run: score_segmented, load_segment_models, score_single_segment, and the now-orphaned _split_max_groups_budget / _warn_if_max_groups_below_segments helpers - scoring_orchestrator segmented fallback; scoring_strategies.score_segmented - model_discovery segment-record selection; drift.check_segment_drift - SegmentationConfig -> GroupingConfig; persisted `segmentation` struct -> `grouping` - _dq_info.anomaly.segment field (permanently null once segmentation is gone) - group_config segmented thresholds (MAX_SEGMENT_MODELS, MIN_ROWS_PER_SEGMENT, ...) - segment_utils: canonicalize_segment_values / build_segment_name / build_segment_filter Models trained on a prior build must be retrained: the config hash changed and the registry struct was renamed. mergeSchema is retained on the registry write so the retrain writes cleanly to an existing registry table. Tests: delete test_anomaly_segments, test_anomaly_segment_naming, test_anomaly_scoring_run; update autodiscovery, registry, drift and apply-checks tests to the baseline_by policy (baseline columns are grouping, not features). Docs: CHANGELOG BREAKING entries; anomaly_compatibility.mdx converted to a migration guide; anomaly_heuristic_map.mdx segmented branch removed; user guide reworked around group-aware detection with a breaking-change/migration section; Experimental -> Beta. The LLM explainer's segmentation vocabulary is intentionally left for the next commit, which rewrites it with the baseline_by replacement in hand rather than blanking it. Co-authored-by: Isaac <no-reply@databricks.com>
Feature engineering expands each source column into engineered features by a fixed set of naming conventions (one-hot, frequency, null indicator, boolean, datetime cyclicals, numeric identity, baseline-relative). Nothing inverted those conventions, so two things were impossible: redacting every feature derived from a redacted column, and showing a reader a raw key like `event_count_rel_baseline` as a human phrase. New anomaly/feature_naming.py, pure functions over SparkFeatureMetadata: - source_column(name, metadata) -> str | None - human_label(name, metadata) -> str (falls back to the name; never raises, never hides a driver) - engineered_from(source, metadata) -> frozenset[str] (defined via source_column, so forward and reverse can never disagree) Resolution order is deliberate: one-hot first (matched against onehot_categories, since a value may contain an underscore or collide with a fixed suffix), then fixed suffixes accepted only when the remainder is a real source column, then numeric identity. Unit-tested across every convention, including a category value that collides with a suffix and a numeric column named like one. Unused until the next two commits wire it into redaction and prompt rendering. Co-authored-by: Isaac <no-reply@databricks.com>
…column Redaction filters LLM-prompt contribution keys by exact match, and those keys are engineered feature names. A prior fix covered the baseline-relative feature (`amount` -> `amount_rel_baseline`) by reconstructing that one name from the column, but one-hot and frequency features could not be reconstructed from the column alone (`country` -> `country_US`, `country_DE`, `country_freq`, `country_is_null`), so they still reached an external serving endpoint after the user redacted `country`. Thread the model's feature metadata to the point of redaction and enumerate the derived features exactly: - ExplanationContext gains an optional `feature_metadata` field; `from_scoring_config` takes it and `scoring_run` passes the `parsed_metadata` already in scope at the call site. - `redaction_set(redact_columns, metadata)` now unions `engineered_from(column, metadata)` per redacted column, covering one-hot, frequency, null-indicator, baseline-relative and identity. This replaces the suffix-only expansion. With no metadata (a caller who built the context directly), it falls back to the baseline-relative feature only -- best effort, unchanged. Redaction still matches engineered keys, upstream of any human rendering, so the hole stays shut when 2c adds human labels. Tests: metadata-driven redaction over one-hot + frequency + null-indicator + baseline-relative; the existing metadata-less tests are unchanged and still pass. Co-authored-by: Isaac <no-reply@databricks.com>
…ines Contributions were shown to users and to the LLM as raw engineered names — a reader saw `event_count_rel_baseline (74%)` in `_dq_info` and in the prompt — and the prompt still spoke the removed language of segments. - feature_contributions in the prompt now renders human labels via the reverse map, after the redaction filter: `amount vs its group baseline (82%), quantity (11%)`. A driver whose label cannot be resolved falls back to its raw name, so nothing is ever hidden. The label lookup is a SQL map literal with escaped keys/values (column names are user-derived). - The ai_explanation struct gains a `top_drivers` string (the same human-labelled sentence) for display. `top_features` stays engineered names for grouping/tooling, and the top-level `contributions` map keeps engineered keys so redaction and downstream tooling still match. - Prompt vocabulary moves from segments to baselines: the dead `segment` field (always empty since the segment path was removed) becomes `baseline_grouping`, the columns forming each row's baseline group — the material fact for a contextual anomaly, and a per-run constant with no PII. Few-shot examples and instructions reframed accordingly. The dead `_format_segment` helper and its tests are removed; the committed prompt snapshot is regenerated. Billable ai_query surface: the fixed prompt header grows ~322 chars (~+80 est. tokens) per call from the richer baseline vocabulary and label guidance, plus a few chars per group for longer human labels. Tests: unit coverage for the baseline-grouping string and the human-label map (identity omitted, derived features labelled, empty without metadata); the AI-explanation integration test asserts top_drivers is populated and carries the top features with weights. Co-authored-by: Isaac <no-reply@databricks.com>
…sions Every claim in the review was reproduced against the code before being fixed. All eight held, and so did the suppression count. Two of my own were wrong and are corrected below. BLOCKERS A pre-release registry could not be retrained, which is the remedy both the migration guide and the stale-hash error prescribe. get_active_model read values["grouping"] unconditionally while a pre-upgrade table carries "segmentation", so it raised KeyError, and training reads the registry, so it failed before save_model could write anything. Now migrated properly rather than papered over: the table is rewritten into the current schema and "segmentation" is gone, instead of sitting beside "grouping" forever with each null on half the rows. The mergeSchema that produced that outcome is removed, so a future schema change fails loudly and demands its own migration. Permission failures translate into an error naming the grant needed, and state plainly that models in an unmigrated table cannot be scored either until retrained -- my first draft claimed scoring was unaffected, which is false, because the hash formula changed. Grouped single-model training failed at MLflow signature inference. Feature engineering preserves the group key on purpose, so the engineered frame is wider than the feature list, and the whole frame reached model.predict carrying a string column the estimator was never fitted on. It affected profile="timeseries" always and the tabular profile at ensemble_size=1; the default three-model ensemble registers by URI and never reaches that code, which is why nothing caught it. One null timestamp ended temporal training with a TypeError: the null bucket's min(seconds) reached float() unguarded while the metric values beside it were guarded. Nulls are filtered from the fit; the all-null case still raises deliberately. The headline SMD evidence measured a 4,000-row prefix of each entity, after which the ten entities with no positives inside that prefix were dropped as unusable. Recomputed from the raw labels: the full series holds 29,444 anomalous rows in 327 incidents across all 28 entities, against 3,732 in 39 across 18. machine-1-1 alone has 2,694 anomalous rows and none in its first 4,000. The truncation is removed and the full series is now the default. The published "79% against 33%" at a 1% budget becomes 67.6% against 57.3%, and the justification moves to average precision, 0.425 against 0.278, because the per-entity coverage gain is +7.9% with a standard deviation of 38.2% and a 16/7 record -- inside its own spread. The guide now states the entity count, the full-series scope and that variance. ISSUES Spark and numpy disagreed on a degenerate severity tail: with p95 equal to p99, Spark pinned everything above the anchor to 95 while numpy interpolated to 100, so thresholds above 95 became unreachable and SHAP gating decoupled from the authoritative severity. Spark now matches numpy, and the linear chain is extracted so the head and the fallback share one implementation. An explicit baseline_by bypassed MAX_BASELINE_GROUPS, which auto-discovery has always respected. The failure is a driver out-of-memory rather than a slow run, because per-group medians and quantiles are each collected and persisted into model metadata. Now refused with the group count and the remedy. The rows-per-group floor was checked before sampling, so a guarantee of 30 rows became about 7 at the fit. The gate uses the effective figure and honours the caller's own sample_fraction. Two existing tests encoded the old optimism, one asserting 120 rows per group was "comfortably above the floor" when it is 28.8 by the time anything is fitted; both were rescaled to test the same boundary. A null group value collided with a literal "MISSING", silently merging two populations into one baseline. The sentinel is now a control character. Done now because it changes the persisted key format and every model must already be retrained this release. Narrower than the review stated: the imputation route is unreachable, because baseline_by may not overlap the feature list. SUPPRESSIONS Four, removed rather than justified. Three type: ignore gave way to the plain if/else that narrows, matching the block above them. One noqa was inert -- the rule it names is not enabled -- which is why nothing flagged it, and why citing "pylint 10.00/10" was checking the wrong signal. Verify with: git diff origin/main...HEAD -- src/ | grep -E "noqa|type: ignore|pylint: disable" Left for the maintainers: this change is dated 0.16.0 in three places and 0.17.0 in three others against a __version__.py of 0.16.0. Which is right depends on the release, and version props are historical facts rather than something to infer, so nothing was changed except the runtime error, which no longer names a release. Co-authored-by: Isaac <no-reply@databricks.com>
The one failure in the anomaly gate's first real run, 233 of 234 passing on both runners. The fixture gave `category` 5 distinct values over 200 rows, so 40 rows per group, which the fit sees as 9.6 once sampling and the train split have taken their share -- below the 30 the constant promises. So it is correctly no longer a grouping candidate, and the fixture was sized against a threshold nothing enforced. 1,000 rows gives `category` 200 rows per group and ~48 at the fit, so it is selected as before, while adding `user_code` as a second dimension would give 4 rows per group and is still correctly rejected. Same boundary, measured against the threshold that applies. Third fixture in this PR to have encoded the pre-sampling optimism. Co-authored-by: Isaac <no-reply@databricks.com>
`is_permission_error` was private and the tests that pin it in both directions imported it anyway, which pylint's C2701 caught. AGENTS.md rules out silencing protected-access to reach past a boundary and prescribes exactly this remedy: if a member needs outside access, make it public. Both directions are worth pinning, and the second more than the first. Mistaking a genuine failure for a missing grant sends someone to their platform team over a defect in DQX, with nothing in the message to suggest otherwise. Co-authored-by: Isaac <no-reply@databricks.com>
The integration fixture that writes a dummy Databricks config profile exists because of mlflow/mlflow#20599, "Fix Databricks unified auth support when MLFLOW_ENABLE_DB_SDK=true", merged 2026-04-30. That fix is in no 2.x release -- the last is 2.22.4 -- so pinning past it crosses the 2->3 boundary, which is why the floor moves this far. Capped below 4.0 rather than pinned, so the resolver takes the latest 3.x. It resolves to 3.15.2 rather than today's 3.16.0 because `make lock-dependencies` passes `--exclude-newer "7 days"`; the project's own guard rail refuses a same-day release, which is the behaviour I would have asked for. One quiet consequence: `log_sklearn_model_compatible` prefers `name=` when the signature offers it, and 3.15.2 accepts both `name=` and `artifact_path=`, so it now takes the `name=` branch where 2.x took the other. Same call, different keyword, and the `artifact_path=` fallback is dead on any supported version. Tracked separately rather than removed here. Unit tests pass, but they exercise no MLflow tracking or registry call at all -- every real one lives in tests/integration_anomaly. The anomaly gate is the first real signal on this bump. Co-authored-by: Isaac <no-reply@databricks.com>
Feature engineering builds derived columns by appending a suffix and writing the result with
`withColumn`, which replaces any column already carrying that name. A table with both `amount` and
`amount_rel_baseline` therefore lost the second one: the derived value overwrote it, the name was
appended to the positional feature list a second time, so the frame carried two identical columns,
and the model was fitted on two copies of the derived value with the user's real metric gone.
Nothing downstream can detect that, and per-feature attribution then names the wrong source column.
Every suffixed transform has the same hole -- `_rel_time`, `_is_null`, `_bool`, `_freq`, the seven
calendar names, and one-hot `{col}_{value}`.
Refused rather than worked around. Renaming the derived feature would leave the persisted feature
list disagreeing with the suffix that redaction and attribution both key on, and dropping either
column silently discards data the caller asked to be checked.
Two checks, because one is knowable from a schema and one is not:
- the predictable names, in `validate_generated_feature_names`, from the schema alone with no Spark
action. Conservative on the two things a schema cannot answer: a null indicator's name is claimed
even when this frame has no nulls, because a collision there is a bug waiting for the first null,
and a categorical column claims its frequency name whatever its cardinality
- the one-hot names, where they are built, because `{col}_{value}` depends on the values
Also catches two columns generating one name, which is why the collector returns a list rather than
a name-keyed dict -- a dict kept one entry and the clash could never be seen.
Suffixes become module constants used by both the transforms and the validator: one drifting apart
from the other would reopen the hole with every test still passing. The calendar block now pairs its
seven expressions with that constant tuple, so the emitted order is the checked order by
construction, and `_classify_column`'s isinstance chain moves into `feature_category_for_type`,
which is what lets validation answer "which transforms will run" from a schema.
The integration test is the one that matters: the unit suite pins the predicate, and only that pins
that `train` consults it.
Co-authored-by: Isaac <no-reply@databricks.com>
… from every metric Two feature-engineering defects, both of which made a fitted baseline depend on something it should not have. Nulls were imputed before the baselines were fitted. `_process_numeric_columns` did `coalesce(col.cast(double), 0.0)` in place on the raw metric, and both relative blocks run after it, so a missing observation contributed a fabricated zero to its own group's median and to its time bucket's median, then received a fabricated deviation from the baseline it had just moved -- `0 - expected(t)` -- on top of its null indicator. Measured on Spark: a group of three 100s and four nulls had a baseline of 0.0 where the median of the values that exist is 100.0. The cast stays where it was and only the imputation moves to the end, after both bases are fitted. `percentile_approx` skips nulls by construction, so the medians and the bucket aggregates now see what is actually there, and the derived features are imputed to a neutral zero -- which for a deviation genuinely means "no deviation", with the `_is_null` indicator carrying the missingness as it was always supposed to. Appends no feature, so the positional contract is untouched. The temporal basis was chosen by whichever metric came first. `select_basis` scored changepoint counts on `representative`, the first metric with more than two usable buckets, and took *its* masked time axis for the period search too, so reordering columns changed every fitted residual for every metric. Reproduced: 0 changepoints with a straight metric first, 3 with a bent one. It now reads every metric. Candidates are compared on the mean across metrics of the ratio between a candidate's holdout residual scale and the same metric's scale under the simplest basis, and the period search reads the full bucket axis. The ratio is what stops a large-magnitude column outvoting the rest, since a residual scale carries the metric's own units. Standardising each metric first was not in the plan and is not cosmetic: my own test asserting the ratio was unitless failed. The Huber fit's `alpha` penalty is absolute, so the same shape expressed in millions is effectively fitted with less regularisation, and rescaling one metric by 1e6 moved the selection from three changepoints to six. Each metric is now centred and scaled by its own MAD before being scored -- for the basis *choice* only, `fit_temporal` still fits raw values, so no coefficient moves. Verified against fieldeng rather than only in CI: the null-baseline, temporal-null and imputation tests pass on real Spark, and so do the group-relative, transformers, nulls and temporal suites (44 + 21 + 16 passed). Co-authored-by: Isaac <no-reply@databricks.com>
…in the width warning One-hot encoding dropped one category of a binary pair. That is the textbook way to avoid the dummy-variable trap, but here it cost a blind spot rather than collinearity: at scoring, the omitted reference category and any value never seen in training both encode as all-zeros, so a brand new value in a binary column produced no signal at all. Every category is now retained, so a known value sets exactly one indicator and anything unseen sets none. The collinearity is absorbed by the detector's ridge, and it is only exact when the column has no nulls. Categories are also sorted now. `distinct().collect()` has no ordering, so the same table trained twice produced the same features in different positions -- and for a binary column, a different surviving category each time. Harmless inside one model, since the list is persisted with it, but it made a model irreproducible from its own inputs under a positional contract. The feature-width warning counted one feature per numeric column and was never told about the comparison bases, each of which adds one more. Twenty numeric columns with both bases build sixty features; the estimate said twenty and stayed silent under the recommended maximum of fifty, on exactly the configuration it exists to flag. The bases are now threaded through validate_columns into the estimate, and the breakdown names them on their own lines, because dropping a basis is a different decision from dropping a metric. Two smaller corrections found while counting. One-hot contributed `cardinality + 1` "for MISSING", but no MISSING indicator is ever built -- the distinct values are collected from the pre-imputation frame with nulls filtered out -- so the estimate was one too high per one-hot column; a null is carried by the `_is_null` indicator instead. And the datetime count is now `len(CALENDAR_FEATURE_ SUFFIXES)` rather than a literal 7, so it cannot drift from the transform. My own test for the width warning failed first time for the right reason: it filtered on "recommended max", which the max_input_columns warning also contains, and twenty columns trips that one whatever the bases are. The filter now matches the width warning's own opening words. Verified on fieldeng: the one-hot, determinism and width tests pass on real Spark, along with the rest of the transformers and column-naming suites. Co-authored-by: Isaac <no-reply@databricks.com>
…ctually do Three claims in the documentation that the code does not support. No behaviour changes here. `expected_anomaly_rate` was described as controlling how many rows the model flags. It does not. It supplies `contamination`, which for both detectors places only `offset_` and therefore only `predict` / `decision_function`; every DQX scoring path reads `-score_samples` and ranks it against the training score quantiles, so what gets flagged is decided by the check's `threshold`. It also does not mitigate anomalies present in the training sample, which the engine docstring implied by calling it a defence against contaminated training data. Two unit tests now pin both halves: scores identical across contamination 0.01 and 0.20, `offset_` and `predict` different. The parameter stays, because it is real for anyone who loads the registered model and calls `predict` -- but the docs no longer sell it as a detection knob. `_rel_baseline` was documented as a log ratio that is "stable when a baseline is near zero and symmetric for halving versus doubling", and my own review brief called it unitless and therefore invariant. It is a signed-log1p *difference*, and only the first half is true. Doubling from 1 gives log3 - log2 = 0.405, the same ratio at 100x scale gives 0.688, and halving gives -0.288 rather than -0.405. So deviations are comparable within a group but not across groups of very different magnitude. What the form does buy is stated instead: finite and monotone where a true log ratio is undefined. A median/MAD standardisation would be genuinely dimensionless and is noted as the alternative, with the cost that argues for measuring it first -- a per-group MAD persisted beside every median. The severity tail above roughly p99 assumes an exponentially decaying score distribution. That held on four synthetic distributions and on real telemetry, but a heavier tail overshoots: on lognormal scores `threshold=99.9` flagged 0.21% of rows against 0.10% requested. The guide now says to read 99.5 and above as "much stricter than 99" rather than as a budget. Co-authored-by: Isaac <no-reply@databricks.com>
The headline SMD evidence combined three runs. The coverage table came from a script that fitted a single Isolation Forest on SMD's clean train split and wrote no JSON; the spread and win-loss record came from the ensemble panel, whose coverage is 57.5%/65.3% rather than 57.3%/67.6%; and the average precision pair matched no archived file at all -- 0.425 against an archived 0.4231, which does not round. The guide then said both detectors were "trained on data that still contained the anomalies", which was true of an even earlier protocol and false of the one actually run: the loader fits SMD's unlabelled train split. Replaced by one run measuring both shipping profiles under two protocols, archived as model_quality/results/smd-protocol.json: - dqx: sample 30% of the rows to be scored, fit 80% of that sample, score everything. What DQX does to a live table, contaminated training included. - clean: fit SMD's own train split, score its test split. The conventional protocol, and the one every archived number so far was really produced under. Under DQX's own protocol the second detector surfaces 96.8% of incidents inside a 1% alert budget against 65.4%, +31.4 points per entity with sd 29.5 and 22 wins to 1 loss. That inverts the published framing, which told readers to look at average precision because coverage was inside its spread: under this protocol average precision is the wash (+0.006, 14W/14L) and coverage is the consistent win. The clean protocol is worse for *both* detectors, because SMD's training period drifts from its test period and a model fitted on a sample of the rows it will score does not inherit that drift. That is not leakage, which was the obvious objection, so it is measured rather than argued: ranking metrics restricted to the rows the model never saw give ROC-AUC 0.826 against 0.845 and average precision 0.380 against 0.382, the same to three decimals. Event coverage is deliberately not reported there, because dropping a quarter of the rows fragments incidents into a different set of events. What this still is not is DQX's pipeline -- no group-relative, time-relative or calendar features, because those need Spark and a workspace. The guide now says so rather than implying otherwise. Benchmark figures come out of the docstrings entirely and are cited from the guide instead. Copying one number into five files is how they came to disagree, and two of those five were generated API pages that no one could have edited anyway. Also corrects the PCA rejection, which claimed truncation "lands on the correlation-aware detector's own number". The archive says otherwise: PCA at 95% variance before that detector reaches 74.5% coverage against its 65.3%, 15 wins to 5. The rejection stands on grounds that survive the number -- k is unchoosable, it collapses on the very anomaly class the detector exists for (recall 81.6% to 0.8%), and it destroys per-column attribution -- and now says that instead. Co-authored-by: Isaac <no-reply@databricks.com>
…ng it Follow-up to scaling this fixture from 200 rows to 1,000 so `category` clears the effective rows-per-group floor. The row count moved; an assertion expecting "200 distinct values" in the high-cardinality warning did not, and `transaction_ref` is unique per row, so it now reports 1,000. Third time a hardcoded fixture number in this file has had to be chased after a rescale, so it reads off `row_count` now. Also pins the collinearity that Stage 4 introduced by retaining every one-hot category. Retained categories sum to 1 on every trained row, so the correlation-aware detector sees a zero-variance direction. Measured both halves rather than assuming: rows satisfying the sum score identically to the one-dummy encoding, so the redundant column is free, while an unseen category lies off that surface and scores far above a known one. That large score is deliberate and is not new behaviour -- columns with three or more categories were always retained in full, so an unseen value has always scored this way. Truncating the binary case was the inconsistency. Co-authored-by: Isaac <no-reply@databricks.com>
`make build` failed with `Unexpected '', expected '-c', '-e', '-r' or the start of a requirement at .build-constraints.txt:7`. Line 7 is a `# via` comment wrapped in escape sequences. `make lock-dependencies` pipes `uv pip compile` into the file. uv suppresses colour when its output is not a terminal, but this shell exports FORCE_COLOR=3, which overrides that, so regenerating the file while bumping the mlflow floor wrote twelve escape sequences into it and uv's own requirements parser then rejected the result. That breaks the release wheel, not just a local build. The file is now byte-identical to main: the build-system requires are hatchling and hatch-fancy-pypi-readme, which an mlflow floor cannot affect, so it should never have changed in that commit at all. Stripped rather than re-resolved, so the pins cannot drift against a different --exclude-newer window; all seven pinned requirements are unchanged. `make build` exits 0 and produces both the wheel and the sdist. Found by building the wheel to re-run the demo notebooks, which is not a step CI performs on a pull request -- so nothing would have reported this before the release. Co-authored-by: Isaac <no-reply@databricks.com>
Only uv.lock conflicted; everything else auto-merged, and the merged pyproject.toml correctly carries both sides -- main's `requires-python >=3.10,<3.13`, its litellm ceiling and its ruff bump, alongside this branch's mlflow floor. Resolved by taking main's lock as the base and re-running `make lock-dependencies`, because a lockfile is generated rather than authored and hand-merging 4,500 hunks of it would produce a file no tool would ever emit. Basing on main keeps the churn to what the mlflow floor actually forces. Resolves to mlflow 3.15.2 and ruff 0.12.12, so both sides' intent survives. Two things checked rather than assumed, both because they bit today: - **No private-proxy URLs.** The only hosts in uv.lock are pypi.org and files.pythonhosted.org. The target's perl step rewrites them, which is exactly why the lock is regenerated through make rather than by calling uv directly -- an ad-hoc `uv run` earlier this session did leak pypi-proxy.cloud.databricks.com into the working tree, and that copy was discarded, never committed. - **No ANSI escape codes in .build-constraints.txt**, run with NO_COLOR because this shell exports FORCE_COLOR=3, which is how colour codes got committed into it earlier and broke `make build`. The file is byte-identical to main. ruff moves from ~=0.3.4 to ~=0.12.0 here, which is a nine-minor jump in the linter, so `make fmt` is re-run against this branch's new code separately from this integration commit. Co-authored-by: Isaac <no-reply@databricks.com>
…needs for our own detector
Raising the mlflow floor to 3.x broke `profile="timeseries"` outright. MLflow 3 validates a saved
sklearn model against skops' set of trusted types and refuses anything it does not recognise:
MlflowException: The saved sklearn model references untrusted types.
Untrusted types found in the file: ['...timeseries_detector.MahalanobisDetector']
`IsolationForest` is recognised, so the default profile was unaffected and every unit test still
passed. The fleet demo failed on a cluster while the tabular demo succeeded, which is exactly the
split the diagnosis predicts, and it reproduces locally against mlflow 3.15.2.
Naming cloudpickle restores what MLflow 2 did by default, so this is the format every DQX model has
always been written with rather than a new one, and it works identically on both major versions.
`skops_trusted_types` would be narrower but exists only on MLflow 3, so the shim would need a second
version branch. The trust model is unchanged and already documented: DQX loads models only from the
Unity Catalog registry the caller owns.
The regression test round-trips rather than only saving, because scoring loads the model back and a
format that writes but cannot read would move the failure from training to the first scored batch.
Scores and feature contributions must both come back identical.
Two existing tests stubbed `log_model` with fakes that did not accept the new argument; they now
assert it is passed, on both branches. Fixing them surfaced a second problem worth recording: on
MLflow 3 `mlflow.sklearn` is a LazyLoader until something touches it, and patching an attribute on the
loader materialises the real module underneath, so the patch landed on the loader while the code under
test called the real `log_model` and tried to reach a tracking server. Importing `mlflow.sklearn`
explicitly makes the patch target and the call target the same object.
Co-authored-by: Isaac <no-reply@databricks.com>
… stop deciding its trend Found by reading the demo's cell output rather than its exit code: the run logged `ConvergenceWarning: lbfgs failed to converge ... max_iter=400` from the Huber fit, with sklearn's own advice to raise the cap or scale the data. The design columns were already scaled (seconds / span, harmonics in [-1, 1]) but the response was not, and HuberRegressor solves for the coefficients and a scale parameter jointly, so a metric in its own large units is badly conditioned. The warning turned out to be the mild symptom. Measured against a known slope across 20 seeds: a metric in billions (bytes) 99.89% wrong (sd 0.00%) against 0.13%, new wins 20/20 a contaminated sample, 5% at 6x 1.32% against 0.14%, new wins 20/20 millions, and negatives 8.37% and 0.89% against 0.26% heavy-tailed (lognormal) noise 0.31% against 0.28%, new wins 10/20 -- a coin flip That error went straight into the residual `baseline_over_time` exists to produce: a metric measured in billions had essentially no trend removed at all. So this generalises rather than fitting the demo. Rates near 1e-6, small magnitudes, integer counts, zero-crossing metrics and a large offset with tiny variation are all a wash to within a tenth of a point. A single seed appeared to regress on heavy-tailed noise, which is why that case was repeated across 20 -- it is a coin flip, and reporting the n=1 result would have been wrong. Exactly invertible for the unpenalised problem, because the model is linear, so persisted coefficients stay in the metric's own units and scoring is untouched. The one real change is that HUBER_ALPHA's effective regularisation was magnitude-dependent and is now scale-free, which is the behaviour worth having but is not a strict no-op. Median and MAD rather than mean and standard deviation, for the same reason the loss is Huber at all, with a documented fallback when more than half the values are identical. make fmt exit 0, unit 2713 passing. Co-authored-by: Isaac <no-reply@databricks.com>
The docstring said "the ridged pseudo-inverse gives the zero-variance direction no weight". There is no pseudo-inverse in the detector and it never forms an inverse at all: a ridge floor is added as a fraction of the average variance, the result is factored by Cholesky, and diag(inverse) comes from the squared column norms of L-inverse. The ridge is what makes a singular covariance factorable in the first place, so it is doing the work the comment credited elsewhere. Also corrects an implication: both dummies of a retained pair vary, so neither is dropped by the constant-feature mask, which keys on per-feature spread rather than on directions in feature space. The measurement the test makes is unchanged and still passes -- scores identical to the one-dummy encoding, an unseen category far above a known one. Only the explanation was wrong, from confusing this detector with the benchmark harness's Mahalanobis, which does use np.linalg.pinv. Found by an agent fact-checking a teaching deck against the source, which is a better review than re-reading my own comment would have been. Co-authored-by: Isaac <no-reply@databricks.com>
…w reported The docstring said "a linear metric first chose one changepoint where a bent metric chose six". That is the reviewer's number, quoted from their report. Our own reproduction on the straight-and-bent pair the unit test uses gives **0 changepoints against 3**, which is what the commit that fixed this said, so the docstring and the commit message contradicted each other and the docstring carried a figure nobody here had measured. Re-confirmed just now: straight-first 0, bent-first 3, and both 3 once every metric is read. The docstring now names the test that reproduces it, so the number has somewhere to be checked against. Both figures demonstrate the same defect, so nothing about the fix changes. The point is that publishing a measurement we did not take is the exact habit this review round was about, and it survived into a docstring while the correct number sat in the commit message beside it. The second figure in that docstring (rescaling one metric by 1e6 moving the count from three to six) is ours, from the scale-invariance test that failed before the response was standardised, and stands. Found by an agent cross-checking a code walkthrough against the source. Co-authored-by: Isaac <no-reply@databricks.com>
Prompted by the question "does our sampling harm the temporal fit, or sample insensitively to the trend?" -- worth measuring rather than reasoning about, and the measurement says the comment overstated one thing while the design is otherwise sound. Sampling does not harm the trend, and the reason is worth writing down because it is not obvious: `DataFrame.sample` is a uniform Bernoulli draw and the train split is `randomSplit`, neither ordered nor chronological, so training rows spread across the whole history and the slope estimator stays unbiased. Measured slope error, full table against the 24% DQX actually fits: 0.00% vs 0.00% at 200k rows, 0.01% vs 0.01% at 20k, 0.13% vs 0.27% at 2k. The fitted window pulls in by under 0.1% of span, so staleness boundaries are unaffected. What sampling does erode is bucket occupancy, and the comment leaned on exactly that: "each bucket contributes its median, which is robust before Huber even sees it". Rows per bucket, full against sampled, are 50 -> 12 at 200k rows, 5 -> 1.6 at 20k, and 1 -> 1 at 2k, where a median of one row is the row. So below roughly 100k rows that sentence describes nothing. It does not matter, because Huber is carrying the robustness: with 5% of rows at 6x normal the recovered slope error is 0.02% sampled against 0.01% full. But the comment credited the wrong mechanism, which is how someone later "optimises away" the thing that was actually load-bearing. Also records a bucketing consequence unrelated to sampling: width is span/4000, so at 200k hourly rows the width is 50h and a daily period is correctly rejected as unresolvable. Sub-bucket-width seasonality is unavailable on very long spans, by construction. Measurement reuses the shipped `candidate_periods`, `fit_temporal` and TEMPORAL_FIT_BUCKETS over 8 seeds per size, reproducing `_fit_temporal_from_buckets`' bucketing exactly. Co-authored-by: Isaac <no-reply@databricks.com>
It read as the knob for how much gets flagged, and it was not one. It set the estimator's `contamination`, which places only scikit-learn's own `predict`/`offset_` boundary; every DQX scoring path reads `-score_samples` and ranks it against training-score quantiles, so what gets flagged is decided by the check's `threshold`. A parameter whose name promises the opposite of what it does is worse than no parameter, and this release already carries breaking changes, so it goes now rather than being documented around. Behaviour is unchanged. `IsolationForestConfig.contamination` now defaults to 0.02 directly, which is exactly what the removed parameter filled in when unset, so a caller who passed nothing gets what they got before. Anyone who was setting it can set `params.algorithm_config.contamination` instead -- still honest, because that name promises only what it delivers, and it still matters to anyone loading the registered model and calling `predict` themselves. Gone with it: `apply_expected_anomaly_rate_if_default_contamination`, the parameter on `AnomalyEngine.train` and `build_context`, the field on `AnomalyTrainingContext`, and its range check in `validate_training_params`. Not in `AnomalyConfig` or the workflow, so no run-config or YAML break. The two docstring examples that used it to "adjust expected anomaly rate for specific use cases" now show `threshold` on the check, which is the thing those examples were reaching for. Tests: the three unit tests for the removed mapping helper become two for the config default and the explicit escape hatch; the two range-check tests become one asserting contamination is still validated, which matters more now that it is the only route. The integration test that asserted the parameter reached the persisted hyperparameters is kept and rewritten against the default, because it is the only coverage that the algorithm config flows into `training.hyperparameters` at all. Verified: fmt exit 0, unit 2709 passing, and the nine training-validation integration tests pass against a live workspace. Co-authored-by: Isaac <no-reply@databricks.com>
The removal itself landed without touching the guide, and the guide's upgrade section opened with "If you never used `segment_by`, nothing here affects you" -- which that removal made false. Anyone passing `expected_anomaly_rate` now gets a TypeError regardless of whether they ever touched `segment_by`. Rewritten so the section states what actually needs action: every model must be retrained whether or not any of this was used, and calls passing either removed parameter will raise. Adds a migration bullet saying nothing replaces it and why -- `threshold` on the check is how you change what gets flagged, and `params.algorithm_config.contamination` is the escape hatch for anyone who loads the registered model and calls `predict` themselves. Also an FAQ entry, because the TypeError names the parameter but not the fix, and the obvious guess (reach for contamination) is usually the wrong one. Checked repo-wide rather than assumed: no demo, notebook, Studio app, MCP server or reference page ever passed it, and the only remaining mentions are comments explaining the removal. The generated API pages regenerate clean from the docstrings. Co-authored-by: Isaac <no-reply@databricks.com>
…was made An earlier commit today rewrote the `TEMPORAL_FIT_BUCKETS` comment to say plainly that bucketing bounds driver memory and that Huber, not the per-bucket median, is what carries robustness. It missed the identical claim in `_fit_temporal_from_buckets`' own docstring, 200 lines further down the same file: "the per-bucket median is itself robust, so gross outliers are attenuated before the Huber fit ever sees them". So the file contradicted itself, with the measurements on one side and the withdrawn claim on the other. That is worse than the original overstatement: a reader who finds the docstring first has no way to know it is the stale one. Now points at the constant, which carries the numbers, and says outright that bucketing is for memory rather than robustness. Found by an agent building a code walkthrough, which read both sites and refused to paper over the disagreement. Grepping for the claim I was withdrawing, rather than editing the site I happened to be looking at, would have caught it at the time. Co-authored-by: Isaac <no-reply@databricks.com>
Third-review finding, and a gap my own earlier fix opened. Stage 1 of that work made a schema legal that had not been before: a caller may pass their own `amount_rel_baseline` column alongside `amount`, because the collision validator only refuses a name the transform would actually generate, and with no `baseline_by` nothing generates it. The reverse mapper never learned that. It decomposed the name by suffix, credited `amount`, and labelled the feature "amount vs its group baseline" in a model with no group baseline at all -- attributing one user column's contribution to a different column, and making redaction of `amount` sweep up an unrelated one. Two changes, both about asking the metadata rather than the string: - An exact source-column match now wins over suffix decomposition. A name in `column_infos` is a column that was analysed, so identity cannot be wrong, and it cannot mask a genuine derived feature because `validate_generated_feature_names` refuses that collision. - Suffix resolution is gated on the basis having run: `_rel_baseline` only decomposes when `baseline_by` is recorded, `_rel_time` only when `baseline_over_time` is. Both were already persisted. The module docstring said the resolution order "must not be reordered", justified by a numeric column named `revenue_freq` where `revenue` does not exist. That case resolves correctly under either order, so it never argued for suffix-first. Corrected, with the reason that does hold. Adds `source_blocks`, mapping each source to its engineered descendants. Redaction gets it for free because `engineered_from` is already defined in terms of `source_column`, and the next commit needs it: explaining one engineered feature at a time is unsound when several carry the same source. The gate immediately caught three test fixtures describing models that cannot exist -- a baseline-relative feature listed with no `baseline_by` recorded -- in the redaction, explainer and naming suites. Fixed rather than worked around; a fixture that cannot occur in production is not protecting anything. Verified with the review's own probe: `amount_rel_baseline` now resolves to itself and labels as itself. Co-authored-by: Isaac <no-reply@databricks.com>
…eered feature Feature engineering gives one source column several engineered views: the metric itself, its deviation from its group's baseline, its deviation from its expected level at that time. The leave-one-out attribution then measures almost nothing for any single view, because dropping one leaves another copy of the same information behind -- and normalising those small numbers hands nearly all of the apparent blame to an unrelated metric. Measured on a correlated pair: adding one affine duplicate of x moved the reported cause from 99.8% x to 99.9% y while the score itself did not move (65.44813 vs 65.44816). So the detector found the right row and then named the wrong column, in business language, with no signal that anything was off. That compounds with the LLM prompt's rule to call a metric abnormal when contributions are "concentrated in a single metric": the guard fires on the innocent one. MahalanobisDetector.block_contributions marginalises a whole group of features out at once, (Pd)_G' inv(P_GG) (Pd)_G. Sigma^-1 is never formed -- L^-1 is recovered once per call from the stored Cholesky factor and each P_GG is built from its columns, so nothing new is persisted and models trained before this keep loading. A single-feature block reduces exactly to the existing a_i = z_i^2/(Sigma^-1)_ii, asserted at rtol=1e-12 rather than argued, so a model with no derived features explains identically to before. Block drops are not additive and the docstring says so: overlapping information belongs to no single block. Blocks come from the persisted metadata via source_block_indices, computed once on the driver and closed over by the UDF. The emitted map is therefore keyed by the source column the caller passed, which fixes the defect by construction -- one entry per source, so duplicate views cannot dilute each other -- and is more useful, since a reader gets their own column names rather than amount_rel_time. Verified end to end: the probe that produced 100% y now returns 99.7% x, matching the undisturbed two-feature model. Scope is the correlation-aware detector only. TreeSHAP's correct block aggregation is a signed sum, and the tree path still discards sign, so blocking there would add magnitudes that should have cancelled. That asymmetry is documented rather than hidden, and the signed-attribution fix it depends on is tracked separately. Two consequences of the key change, both checked rather than assumed. Redaction stays exact: it already unions the source column with every feature derived from it, and the source column is now itself the key. Label lookup falls through to the raw key, which for a source column is the user's own column name -- the intended outcome, not a gap. Both docstrings asserting that contribution keys are always engineered names are corrected, as are the guide's schema table and its "How it works" step. Co-authored-by: Isaac <no-reply@databricks.com>
…t contain The prompt receives contribution magnitudes, severity percentiles, group size, an unsigned drift score and metadata. Nothing in that carries a sign. Attribution is a squared quantity, so a row displaced one way and a row displaced equally the other way produce the identical score and the identical contribution map -- (8, 0.5) and (-8, -0.5) both score 65.387 with the same map. The few-shot exemplars asserted direction anyway: "sits far above the norm" and "Inflated amount fields overstate revenue". A few-shot example is an instruction, and a smaller serving model copies its shape, so those two lines made confident, business-language, half-of-the-time-backwards claims the house style. Both are rewritten so every clause is checkable against that exemplar's own inputs, and the instructions now name the absence of direction explicitly and list the words it rules out. "Be direct, avoid hedging" was the rule that made this feel sanctioned, so it is reconciled rather than left to interpretation: being direct means stating plainly what the inputs contain, and does not license asserting a direction, a cause or a value they do not. The correlation-aware reading was overclaiming in the other direction. It instructed the model to "describe the pattern as a broken relationship", but a high contribution there is equally consistent with one metric moving a long way on its own. The input cannot separate those, so the text now says so and refuses both readings rather than mandating one. The guard it already carried -- do not call a metric abnormal unless the contributions concentrate on it -- is only trustworthy now that attribution is keyed by source column, since concentration used to be able to land on an innocent metric. Neither exemplar showed attribution_basis, the field that decides how contributions may be described, so the model had no demonstration of the field it is told to follow. Both now state it, with different values, so one reading cannot become the assumed default. Two smaller honesty fixes in the same pass. "confidence" is agreement between ensemble members that differ only by random seed on identical training data; it measures score stability, not whether the flag is right or whether the data has drifted, and the prompt now says that and forbids presenting it as confidence in the finding. And the header says "why the model flagged this group" rather than describing a root-cause pattern, because the inputs cannot establish a cause. Snapshot regenerated. Five new tests pin the properties rather than the wording: no directional phrase in either exemplar response, both readings demonstrated, the two reconciliation clauses present, the correlation-aware text refusing to assert, and the confidence caveat intact. The existing attribution-semantics test kept its docstring -- it records a real observed failure, an LLM reporting "Abnormal coolant flow" for rows whose every reading sat inside its healthy range -- with its assertions repointed at those protections so a rewording that keeps the guarantees passes. Co-authored-by: Isaac <no-reply@databricks.com>
The comment justifying full one-hot retention said an unseen value "scored as a large deviation" and that "the detectors can tell apart" a known value from an unrecognised one. The plural was wrong, and it was a comment written in this branch. Measured on the same binary column, with every category retained: IsolationForest unseen 0.458 known 0.442 / 0.467 -> ordinary, between the two known encodings Mahalanobis unseen 2,000,000 known 0.9 Isolation Forest splits one feature at a time, so no tree can express "these indicators sum to zero", and an all-zeros row sits inside every individual indicator's observed range. Retaining the categories puts the distinction in the encoding, which is what the encoding can do; acting on it needs a model that reads features jointly. So an unrecognised category is not reliably caught under profile="tabular", and the comment and the test docstring that implied otherwise now both say so. The guide gains the user-facing half of this in the following commit, which is where its other prose changes land. The retention argument itself comes out stronger, not weaker. Pruning the anticorrelated dummy -- exactly what a correlation-threshold feature filter would do -- collapses the correlation-aware separation from 2,000,000-against-0.9 to 1.1-against-0.9, while leaving every score on ordinary rows untouched. The redundancy is not waste being tolerated: it *is* the off-support constraint, and no in-sample metric would notice its removal. That is now the sharpest argument on record against pruning correlated features here. Two tests make both halves executable rather than asserted in prose. Also lands the mirrored-row test, which belongs to the prompt work in the previous commit but lives in this file: a row and its mirror produce identical scores, identical per-feature contributions and identical block contributions. That is the fact making the prompt's no-direction rule necessary, pinned at the source of the numbers rather than only as an assertion about prompt text. Co-authored-by: Isaac <no-reply@databricks.com>
…d three real limits The guide led with 65.4% against 96.8% incident coverage, measured by fitting a 30% sample of the very rows being scored. Under that split 272 of the 327 labelled incidents had rows in training. So the number answers "can the model re-find anomalies it was shown", which is fair for scanning a table you already have, but it is not evidence for the guide's own quickstart -- train once, then check new or incoming data. The guide now reports the chronological split only: train on an earlier period, score a later one. 57.5% against 65.3% coverage at a 1% budget, average precision 0.277 against 0.423. Deliberately *less* prose, not more: a reader of this page is deciding whether to use "timeseries" on their data and cannot adjudicate a protocol taxonomy, so the retrospective table, the held-out-rows digression and the whole "why a clean training split looks worse" admonition are gone, and the sentence that actually helps stays -- choose on the shape of anomaly you expect, not on an expected accuracy gap. The full picture, both protocols and the overlap counts, belongs in the PR and the benchmark archive. One correction that follows from the switch: round 2 flipped the headline from average precision to coverage because AP was a wash -- under the leakier protocol. Chronologically AP is the stronger signal again, +0.146 or +53% against +7.8 points of coverage. The per-machine statistics that justified the coverage framing (+31 points, 22 wins to 1) belong to the retired protocol and are not carried over, because there are no per-machine chronological numbers to replace them with. The drift attribution is softened too: two protocols disagreeing does not isolate drift as the sole cause. Three limitations are documented rather than fixed, because each is a real property of the design and not a defect: baseline_by conditions on level, not on relationship. One model is fitted and it learns one set of relationships shared across groups -- the right trade when groups differ in scale and agree in behaviour. Measured on a counterexample where group A has y rising with x and group B has y falling, both centred identically: the pooled model scores a row that is impossible for B at 1.3, where a model fitted on B alone scores it 260. Since the medians match, group-relative features are identical and the transform changes nothing. Users whose groups genuinely behave differently are told to train per group. The same limit applies to baseline_over_time, which removes one shared trend. An unrecognised category is caught by "timeseries" and generally not by "tabular", which is the user-facing half of the previous commit. No accuracy number in that section reveals it, so it is stated plainly and points at is_in_list and foreign_key. The temporal selection criterion is blind to constant forecast bias, and that is fine. It subtracts the residual median, so it measures spread. But a constant offset in a _rel_time column cancels before it can reach a score: the correlation-aware detector centres on the training mean, and IsolationForest draws split thresholds from each feature's observed range so shifting a column shifts its candidates with it. Measured against no offset, offsets of 5 and 500 move the correlation-aware score by 1.7e-14 and 1.9e-12 relative -- floating-point residue from the centring, far below the quantile spacing that decides a severity percentile -- and leave IsolationForest bit-identical. Non-constant error inflates residual spread, which the criterion does penalise, so the blind spot is exactly the case that cannot matter. Pinned as a test, asserted at a tolerance rather than at bit equality, because bit equality is false: an earlier draft of this claim said "bit-identical on both detectors" and the test written to prove it failed. Co-authored-by: Isaac <no-reply@databricks.com>
The schema table linked to #how-it-works, but the heading is "How it works under the hood", so the anchor is #how-it-works-under-the-hood. Docusaurus fails the build on broken anchors rather than warning, so this took the whole docs job down. Introduced two commits ago, in the same change that added the sentence. It also took the five codecov statuses with it: .github/codecov.yml sets require_ci_to_pass with wait_for_ci, so those report failure when any CI job fails, whatever the coverage actually is. Verified with `make docs-build` locally, which is what CI runs and what I should have run before pushing the change that broke it. A cross-check of every in-page anchor in that file against every heading found no others. Co-authored-by: Isaac <no-reply@databricks.com>
…lising feature, or one member Three defects in feature attribution, all of which could name the wrong column, confidently, in business language, on a row the detector had flagged correctly. One fix is the prerequisite for the other two, so they land together. TreeSHAP's sign was being discarded rather than read. It explains the ensemble's average *path length*, not a score -- its base value is a path length around 12 -- and a row is anomalous when it is isolated in few splits, so a NEGATIVE value drives the anomaly and a positive one argues the row is ordinary. Using the magnitude added the second to the first and reported the sum as one number. The review that raised this had the sign backwards, and implementing it as described would have been far worse than the status quo. Measured on 200 rows each carrying exactly one deliberately anomalous feature, the largest value named the true culprit 199/200 under the orientation that ships, 199/200 using magnitude alone, and 0/200 using the positive side -- which reliably names the most ordinary feature instead. So magnitude was very nearly right; on rows anomalous enough to be shown, the evidence is 97.8% anomaly-driving at a severity of 95 and 99.5% at 99. Correcting it left the top driver unchanged on every row measured and reordered the ranks below it on roughly a quarter of rows. The map stays non-negative and sums to 100, so this is not a schema change. Orientation happens where the values are produced; the clip happens once, at the end, in format_shap_contributions. That ordering is load-bearing rather than tidy. Averaging across ensemble members and summing a source column's views both rely on SHAP's exact additivity, and a value arguing the row is normal has to be able to cancel one arguing it is not: two views at +3 and -1 net to 2 but clip first and they sum to 3; two members at +4 and -2 average to 1 but clip first and it is 2. An earlier draft of this change clipped at the point of production, which would have made both operations decompositions of nothing while every test still passed. A row with no anomaly-driving evidence was given 1/n for every key -- "every feature contributed equally", which is an explanation with no input behind it and the more misleading of the two failures because the numbers look unremarkable. Those rows now carry the all-null map that a row with a null feature already produced, and which every consumer already renders as unknown. Zero-valued entries are dropped from messages, prompts and the pattern key too, rather than being listed as "quantity (0%)": exact zeros went from rare to common when the normalising side stopped earning a share. An ensemble scored with every member and explained with one. The reported score is the mean across members and so is confidence_std, but contributions came from models[0], which is not a representative member -- they differ only by random seed, so it is whichever trained first. They disagree more than that framing suggests: on 200 rows with the default three members, the least-agreeing pair named a different top driver on 130 of them and the closest pair still differed on 92. Contributions are now the mean, which decomposes the mean path length exactly. The entry point takes the models behind the score rather than a model, so the mistake cannot be remade by omission, and misaligned members raise instead of being averaged into a confident, plausible, wrong answer. Cost is about 7% more at three members, because attribution runs only on rows above the threshold; there is deliberately no knob for it, since the honest lever is enable_contributions=False and it already exists. Both detectors now report one entry per source column. The correlation-aware one already did, from the previous commit's joint marginalisation; a tree model gets the plain sum within each block, which is exact because SHAP is additive and which the corrected sign is what makes possible. The failure it fixes is different in shape from the correlation-aware one -- the evidence is split rather than misdirected, 49.3% and 50.7% across two views of a column whose true share is 100% -- and it still changes the answer: with three views, a column whose true share was 74.5% left each view near 25%, handing the top spot to a derived view of itself while an unrelated single-view column sat tied alongside. The ensemble path never received blocks at all, and the default profile *is* the ensemble, so that plumbing is added. One knock-on worth stating: the shared integration contract asserted contribution keys were a subset of the *engineered* feature names. That was true only by accident on an all-numeric fixture, where a column's only feature is named after it, and false the moment a categorical appeared. It now asserts the actual contract, the caller's own columns. Both demos taught that a contribution reads "X vs its expected level at that time". It no longer does, and that is the trade this makes: naming the column right costs the signal of which comparison objected, which is now carried by baseline_over_time being set rather than by the key. Both prints say so. Also brought into line: compute_contributions_for_matrix carried both original defects in the same module, which is how one module came to hold two answers to what a negative SHAP value means; the correlation-aware detector's docstring claimed every consumer takes abs(); and the guide documented enable_contributions as defaulting to False when it is True. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Row anomaly detection: a second detector, and the end of
segment_byCloses #1484 and #1490.
TL;DR
Row anomaly detection gains three orthogonal choices, and loses one path:
profile"tabular"(Isolation Forest) or"timeseries"(Mahalanobis)"tabular", unchangedbaseline_bybaseline_over_timesegment_byandAnomalyParams.max_segment_modelsare removed.baseline_byreplaces them but is not a rename:segment_bytrained one model per group,baseline_byconditions one pooled model. Scores differ.segmentation→grouping, and a pre-release registry table is migrated in place on the next retrain.expected_anomaly_rateis removed. It set the estimator'scontamination, which never changed what DQX flags, so its name promised the opposite of what it did. Nothing replaces it:thresholdon the check is the alert budget. Default behaviour is unchanged.Beta, so formats were allowed to change without a migration path. Migration steps are in the user guide.
profile="timeseries"Isolation Forest splits on one feature at a time, which makes it strong on tabular data and close to blind when the anomaly is a broken relationship between two readings that are both in range .. there is no single feature to split on.
Measured on the full Server Machine Dataset (28 entities, 38 metrics, 708,420 rows, 327 labelled incidents), trained on an earlier period and scored on a later one .. which is what a user does when they train once and check new data.
@1%budget"tabular""timeseries"Average precision is the larger relative gain, at +53% against +7.8 points of coverage. An incident counts as surfaced if the detector flags at least one of its rows, so coverage answers "would I have been paged".
Which protocol, and why the headline changed
Earlier versions of this description led with 65.4% against 96.8%, measured by fitting a 30% sample of the very rows being scored. That protocol is legitimate for one question .. can a model re-find anomalies it has already seen, which is what scanning a table you already have amounts to .. but under it 272 of the 327 incidents had rows in training, 271 shared rows with the held-out set, and 7,027 of the 29,444 anomalous rows were fitted. So it is not evidence for scoring data that arrives later, which is what the quickstart tells you to do. Same code, same data, 31 points of difference from the split alone.
The table above therefore reports the chronological split, and this is the second correction to the same section. Round 2 flipped the headline from average precision to coverage because average precision was a wash (+0.006, 14W/14L) .. under the leakier protocol. Chronologically it is the stronger signal again, which is where it started. The per-entity statistics that justified the coverage framing (+31.4 points, sd 29.5, 22 wins to 1) belong to the retired protocol and are not carried over, because there are no per-entity chronological figures to replace them with.
Two protocols differing also does not isolate drift as the cause, and an earlier draft said it did.
Both protocols come from one archived run. The comparison is between estimators on raw metric matrices .. DQX's own feature engineering is not part of it. A benchmark through the real pipeline needs a workspace and 28 tables, and is tracked separately.
Two earlier sets of figures were withdrawn rather than adjusted: 79%/33%, from a harness truncating each entity to 4,000 rows, and 57.3%/67.6%, which mixed three runs one of which saved no output. Both were found by review.
No
"auto": the choice cannot be verified without labels, and silently swapping estimators would move every score people have set thresholds against.baseline_byThe same value can be fine in one group and wrong in another. One group's volume dropping 80% behind a flat daily total scored 45.1 .. the 45th percentile, which no threshold recovers.
baseline_byadds each metric's deviation from its own group's median, on one pooled model, so cost does not grow with group count. The same collapse now scores above 95, and PR-AUC on a contextual anomaly goes from 0.0376 to 0.5703 while an already-globally-extreme anomaly stays at 1.0000.Rows whose group was absent from training return a null score with
is_new_baseline = true, rather than 0.0 .. which was the most normal-looking value in the table, and the wrong thing to say when the truth is "could not judge this".baseline_over_timeA metric that has grown for a year sits outside the range it trained on, so ordinary rows start being flagged; and a value that is ordinary against the whole year can be badly wrong for where the trend had reached. On a fixture where every metric is rolled back together, correlations intact and every value inside the training range,
"tabular"surfaces 0.0% of incidents and"timeseries"0.8%. With the temporal baseline, 100%.DQX fits each metric's expected level as a function of time and persists the coefficients, so it extends to timestamps the training window never contained. That is why a time-bucket
baseline_bycannot do this job.baseline_by. With both set, the fit runs on the group-relative value. On groups trending at different rates, one pooled fit on the relative value reaches 101% of per-group fits; fitting raw values collapses from 80% to 29% as slopes diverge.profile: worth +39% mean event coverage on the tree detector and +40% on the correlation-aware one, harming neither. A third axis, not a third detector.Four internal rules are measured rather than chosen, each pinned by a test carrying its number: a seasonal term needs six complete cycles (a daily period over 2.8 days cost 15 points of coverage; apparent seasonal strength rises as cycle count falls, so variance explained is the wrong gate and is inverted); the fit is Huber, not least squares (with 5% of rows at 6x normal, ridge recovered slope 0.0694 against a true 0.0500, Huber 0.0502); changepoints are chosen on a held-out tail (in-sample R² was identical from 0 to 25 changepoints while false flags one window out ranged 1.2% to 100%); and the response is standardised before fitting, because the design columns were already scaled while the metric was in its own units and Huber solves for coefficients and a scale parameter jointly. That last one is the largest model-quality fix here: across 20 seeds, a metric measured in billions had its slope 99.89% wrong (sd 0.00%) against 0.13%, winning 20 of 20, and a contaminated sample went from 1.32% to 0.14%, also 20 of 20. Small magnitudes, rates near 1e-6, negatives, integer counts and zero-crossing metrics are a wash, and heavy-tailed noise is a 10/20 coin flip. It surfaced from a
ConvergenceWarningin a demo's cell output, which is the sort of thing an exit code hides.Severity between the quantile knots
threshold=98did not flag 2% of rows, andthreshold=99.9flagged nothing at all. Severity was interpolated linearly in score space, and a score quantile function is convex in the tail, so a straight line from p95 to p99 sits above the true quantile the whole way.thresholdNow interpolated in tail probability, which is what a percentile is:
severity = 100 - 5 * 5**-u,u = (score - q95) / (q99 - q95). Same distributions give 1.85% to 2.11% at 98 and 0.43% to 0.59% at 99.5.Chosen over adding knots for two reasons: it persists nothing new, so models already in a registry are corrected by being scored (new keys would have broken every pre-existing model, since
extract_quantile_pointsraises on a missing key); and it is exact at p95 and p99, so 90/95/99 are fixed points and the default configuration is byte-identical. It also breaks a tie .. the old expression clamped everything past the training maximum to exactly 100, which was 20% of a scored batch on average.Above roughly p99 the tail assumes exponential decay, which held on four synthetic distributions and real telemetry but overshoots on a heavy tail (0.21% against 0.10% nominal at 99.9), so the guide says to read 99.5+ as "much stricter than 99" rather than as a budget.
What this does not fix: scoring a later period with a model fitted on an earlier one flagged 9.8% to 21.7% of rows at
threshold=95, against 4.9% to 5.0% on same-period held-out data. Calibration is sound; the input moved. That is a retraining signal, and no training-relative scheme can fix it without a global pass the streaming contract forbids.Why Mahalanobis, and what was rejected
Chosen by measurement. The metric is event coverage at a fixed alert budget, not PR-AUC: SMD's anomalous rows sit in incidents of median length 6 and max 1041, so point-wise PR-AUC mostly measures whether the one huge incident was found. No point-adjusted F1 either .. Kim et al. (AAAI 2022) showed random scores reach state of the art under it.
It also gives exact, never-negative attribution: leave-one-out contribution has a closed form for this detector, so contributions come free with no SHAP dependency. The naive signed decomposition was rejected because its terms can go negative, and a feature that reduced the distance is not a driver of the anomaly .. it would now be dropped rather than misreported, which silently removes evidence from an explanation whose remaining shares no longer describe the distance they came from. Non-negativity has to come from the formula, not from a clip. Being deterministic, it trains one model rather than an ensemble (
ensemble_sizeis ignored andconfidence_stdunavailable, both logged).Rejected alternatives, with the numbers so nobody re-argues them
PCA 95% → Mahalanobisreaches 74.5% coverage against the shipping 65.3%, 15W/5L. Rejected anyway on three grounds independent of that: k is unchoosable (same fixture swings ROC-AUC 0.48 to 1.00 on k alone, and retained variance is anti-correlated with what the data wants); it collapses on the very anomaly class the second detector exists for (recall 81.6% → 0.8% on a correlation break); and it destroys per-feature attribution.segment_by). Worst of three configs on SMD (PR-AUC 0.1416 against 0.1536 conditioned), and failing in a way the average hides: one entity produced a 56% false alarm rate because each per-group model calibrates on its own rows. Cost is linear in groups .. 90 groups took ~88 minutes against 0.17s for one conditioned model.Explaining a flagged row
Detection and explanation were reviewed separately, and the explanation is where the defects were. Each
of these could name the wrong column, confidently, in business language, on a row the detector had
flagged correctly.
One column, one entry. Feature engineering can derive three features from one numeric column .. the
metric, its deviation from its group's baseline, its deviation from its expected level at that time ..
and contributions were reported per feature, which answers a question nobody asked. The correlation-aware
detector's per-view drops are each almost nothing, because dropping one view leaves another copy of the
same information: adding one affine duplicate of a metric moved the reported cause from 99.8% that
metric to 99.9% an unrelated one while the score did not move. Blocking by source restores 99.7%/0.3%.
For a tree model the failure is different in shape .. the evidence is split rather than misdirected,
49.3% and 50.7% across two views of a column whose true share is 100% .. and still changes the answer,
because with three views a column whose true share was 74.5% left each view near 25%, handing the top
spot to a derived view of itself with an unrelated single-view column tied alongside. Both detectors now
group by source column, by an exact joint marginalisation for the one and a plain sum for the other,
which is exact because SHAP is additive.
Provenance is read from the recorded transform, not from spelling. A caller's own column named
amount_rel_baselinewas decomposed intoamountand labelled "amount vs its group baseline" on a modelwith no grouping at all .. crediting one column's contribution to another, and making redaction of
amountsweep up an unrelated column. Suffixes now resolve only when the transform that generates themactually ran, and an exact source-name match wins over decomposition.
The SHAP orientation was wrong in kind, and the review's reading of it was backwards. TreeSHAP on an
isolation forest explains the ensemble's average path length, not a score, so a negative value
shortens the path and drives the anomaly. Review reported this as
abs()turning normalising evidenceinto drivers, which implies the positive side is the anomaly-driving one; it is the opposite. Measured on
200 rows each carrying exactly one deliberately anomalous feature, the largest value named the true
culprit 199/200 under the orientation that ships, 199/200 using magnitude alone, and 0/200 using
the positive side .. so implementing the finding as described would have destroyed attribution accuracy.
Magnitude alone was very nearly right, since on rows anomalous enough to be shown the evidence is 97.8%
anomaly-driving at a severity of 95 and 99.5% at 99. It was still wrong in kind, adding evidence of
normality to evidence of anomaly, and correcting it left the top driver unchanged on every row measured
while reordering the ranks below it on roughly a quarter of rows. Not a schema change: the map stays
non-negative and sums to 100.
A row with no anomaly-driving evidence said every feature contributed equally. When the attribution
totalled zero the map was filled with
1 / n.. an explanation with no input behind it, and the moremisleading of the two failures because the numbers look unremarkable. Such rows now carry the all-null
map that a row with a null feature already produced, which every consumer already renders as
unknown.Zero-valued entries are also dropped from messages, prompts and the pattern key, rather than being
listed as
quantity (0%).An ensemble scored with every member and explained with one. The reported score is the mean across
members and so is
confidence_std, but contributions came frommodels[0].. arbitrary, since membersdiffer only by random seed. They disagree more than that framing suggests: on 200 rows with the default
three members, the least-agreeing pair named a different top driver on 130 of them and the closest
pair still differed on 92. Contributions are now the mean across members, which decomposes the mean path
length exactly, and the entry point takes the models behind the score rather than a model, so the mistake
cannot be remade by omission. It costs about 7% more at three members, because attribution runs only
on rows above the threshold. Averaging makes the explanation unbiased between members, not stable .. with
three members it stays noisy, and the fix for that is more members.
The prompt asserted a direction the inputs do not contain. Contributions are unsigned magnitudes, so
a row displaced one way and a row displaced equally the other produce the identical score and the
identical map ..
(8, 0.5)and(-8, -0.5)both score 65.387. The few-shot exemplars nonetheless said"sits far above the norm" and "Inflated amount fields overstate revenue", and a smaller serving model
copies the shape of its examples. Both are rewritten so every clause is checkable against that
exemplar's own inputs; the instructions name the absence of direction explicitly; and "be direct, avoid
hedging" is reconciled rather than left to interpretation.
confidenceis relabelled as agreementbetween seeds rather than confidence in the finding, and the header says the model is explaining what it
measured rather than diagnosing a root cause.
Novelty is not guaranteed on the default profile. Retaining every one-hot category puts an unseen
value's distinction in the encoding, but acting on it needs a model that reads features jointly. The
correlation-aware detector scores an unseen category at 2,000,000 against 0.9 for a known one; Isolation
Forest scores it 0.458 against 0.442 and 0.467, i.e. ordinary. A comment claiming both detectors could
tell them apart is corrected, and the guide says so. The same probe strengthens the case against pruning
correlated features: dropping the anticorrelated dummy collapses that separation to 1.1 against 0.9 while
leaving ordinary rows untouched, so the redundancy is the off-support constraint and no in-sample metric
would notice its removal.
Limitations documented rather than fixed
baseline_byconditions on level, not on relationship. One pooled model learns one set ofrelationships. On a counterexample where group A has
yrising withxand group B has it falling,both centred identically, the pooled model scores a row that is impossible for B at 1.3 where a
model fitted on B alone scores it 260 .. and since the medians match, group-relative features are
identical, so the transform changes nothing. Groups that behave differently rather than sitting at
different levels want one model each.
baseline_over_timehas the same limit: it removes one sharedtrend.
because a constant offset in a
_rel_timefeature cancels before it can reach a score .. offsets of 5and 500 move the correlation-aware score by 1.7e-14 and 1.9e-12 relative and leave Isolation Forest
bit-identical. Non-constant error inflates residual spread, which the criterion does penalise, so the
blind spot is exactly the case that cannot matter. Written down so nobody "fixes" it.
Breaking changes
segment_byremoved, along withAnomalyParams.max_segment_models.segmentation→grouping. A pre-release table is migrated in place on the next retrain and the old column is gone afterwards, rather than both being carried forever by amergeSchemaappend. A permission failure is translated into an error naming the grant needed._dq_info[].anomaly.segmentremoved;is_new_baseline,new_baseline_key,is_stale_baselineandstale_baseline_horizonadded. The struct is wider, so appending to an existing_dq_infotable needsmergeSchema.amountalongsideamount_rel_baseline), instead of silently replacing it and fitting on two copies of the derived value._dq_info[].anomaly.contributionsis keyed by source column under both profiles, where"tabular"previously keyed by engineered feature. A query readingsignup_hour_sinnow readssignup. The map staysmap<string,double>, non-negative and summing to 100, so only the key vocabulary changes .. and a column that argued the row was normal now shows0rather than a small positive share. A flagged row with no anomaly-driving evidence carries an all-null map instead of an invented even split.expected_anomaly_rateremoved fromtrain(). It supplied the estimator'scontamination, which places only scikit-learn's ownpredict/offset_boundary; DQX ranksscore_samplesagainst training quantiles, so the parameter never changed which rows were flagged. Documenting around a name that promises a detection rate would have preserved the trap, so it is gone.contaminationnow defaults to 0.02 directly, which is exactly what the parameter filled in, so a call that never passed it behaves identically. To change how much is flagged, setthresholdon the check; to setcontaminationbecause you load the registered model and callpredictyourself, pass it throughparams.algorithm_config.mlflowfloor moves to>=3.13,<4, to pick up a Databricks unified-auth fix that exists in no 2.x release. One consequence needed handling: MLflow 3 validates saved sklearn models against skops' trusted types and refusesMahalanobisDetector, DQX's own class, soprofile="timeseries"failed at registration while every unit test passed.serialization_format="cloudpickle"is now named explicitly, which is what MLflow 2 did by default, so this restores existing behaviour rather than introducing new.Docs, demos, tests
drift_thresholddoes not warn about trend).dqx_row_anomaly_detection_demo.pyis replaced by two problem-led notebooks (card transactions that pass every rule but are jointly implausible; machine telemetry where every metric stays in band while the relationship breaks), both covered by a parametrised e2e test. Both print a threshold sweep with precision beside the ceiling that alert count allows, rather than a single detection rate, and both train on the whole table becauseDataFrame.sampledraws per partition and is not reproducible across partition counts.Unit 2754 passing, mypy clean, pylint 10.00/10,
make fmtandmake buildclean,make docs-buildclean.Still open
sample_fraction, because the sample is drawn per partition. A deterministic sample is a small change tosample_dfbut moves every existing user's model.0.16.0in some places and0.17.0in others; which is right depends on the release this lands in, so a maintainer needs to settle it.is_new_baselineis the precedent to follow.make anomaly, against a live workspace) and both demo notebooks need a re-run, because the contributions map's keys changed for the default profile. Unit tests cannot reach the registry or MLflow paths, so that gate is the signal that matters and this stays a draft until it is green.This pull request and its description were written by Isaac.