diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 8e98bd02a..16c0b5c47 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -626,7 +626,12 @@ jobs: # The run fails if performance degrades by more than 25%. # Tests are run sequentially to reduce variability. # Do at least 5 rounds to get more stable results. - UV_FROZEN=1 uv run --all-extras pytest tests/perf -v -n 1 \ + # Anomaly benchmarks are deselected here: --benchmark-compare-fail applies to the whole + # invocation and cannot be scoped per test, and those benchmarks are dominated by MLflow + # and Unity Catalog control-plane latency, so they would trip a 25% mean gate on variance + # alone. They still produce a baseline in the step above; only the timing comparison skips + # them. Their detection quality is gated by assertions in tests/integration_anomaly/. + UV_FROZEN=1 uv run --all-extras pytest tests/perf -v -n 1 -m "not anomaly" \ --benchmark-storage=$BENCHMARKS_DIR \ --benchmark-compare=baseline \ --benchmark-compare-fail=mean:25% \ diff --git a/demos/dqx_demo_anomaly_correlation_fleet.py b/demos/dqx_demo_anomaly_correlation_fleet.py new file mode 100644 index 000000000..6b9367f1b --- /dev/null +++ b/demos/dqx_demo_anomaly_correlation_fleet.py @@ -0,0 +1,775 @@ +# Databricks notebook source +# MAGIC %md +# MAGIC # ⚙️ When Every Gauge Reads Normal and the Machine Still Fails +# MAGIC +# MAGIC ## Learn Row Anomaly Detection on Machine Telemetry in 10 Minutes +# MAGIC +# MAGIC **What you'll do:** +# MAGIC - Generate healthy telemetry where the metrics move together, as real machines do +# MAGIC - Break the *relationship* between three of them while every reading stays in range +# MAGIC - Prove no threshold or range check could ever catch it +# MAGIC - Train with `profile="correlation"` and read the explanation +# MAGIC +# MAGIC **Dataset**: Eight metrics per reading across four CNC machines (no domain expertise required) +# MAGIC + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ## The problem: a broken relationship is not an extreme value +# MAGIC +# MAGIC A plant records spindle load, motor current, coolant flow, bearing temperature, vibration, +# MAGIC hydraulic pressure, air pressure and throughput. Every metric has a safe band, and every band has +# MAGIC an alert. +# MAGIC +# MAGIC A bearing fails. Afterwards the logs show every gauge sat inside its band for two hours +# MAGIC beforehand. No alert fired. +# MAGIC +# MAGIC But something *was* visible: **motor current stopped tracking spindle load**. On a healthy machine +# MAGIC those rise and fall together, because cutting harder draws more current. For two hours load was +# MAGIC high while current sat mid-band. Both readings were ordinary. Their relationship was not. +# MAGIC +# MAGIC ## "Normal" compared to what? +# MAGIC +# MAGIC Every anomaly check answers one question: *is this row normal?* The useful part is the follow-up, +# MAGIC **normal compared to what?** DQX gives you three independent answers, and you can use any combination +# MAGIC of them. Nothing here requires knowing any ML. +# MAGIC +# MAGIC #### 1. Compared to the rest of the table — which detector? (`profile`) +# MAGIC +# MAGIC | Your data | `profile` | An anomaly looks like | +# MAGIC |---|---|---| +# MAGIC | **Independent records.** Card payments, insurance claims, customer records, product listings. | `"tabular"` (default) | A row whose values, or combination of values, is unusual | +# MAGIC | **Repeated measurements of the same things.** Machine sensors, server metrics, patient vitals, smart meters. | `"correlation"` | Metrics that normally move **together** stop doing so, each staying in its own range | +# MAGIC +# MAGIC This notebook uses `"correlation"`, because the bearing story above is exactly its case. The default +# MAGIC detector splits on one column at a time, so a broken relationship between two in-range values is close +# MAGIC to invisible to it. On the **Server Machine Dataset**, 28 machines of real telemetry with labelled +# MAGIC incidents, trained the way DQX trains, the correlation-aware detector surfaces **96.8%** of incidents +# MAGIC inside an alert budget of 1% of rows against **65.4%** for the default, and it wins on 22 of the 28 +# MAGIC machines while losing on 1. See the guide for the full protocol and for the metrics where the two are +# MAGIC level. +# MAGIC +# MAGIC #### 2. Compared to its own group (`baseline_by`) +# MAGIC +# MAGIC The same number can be fine in one group and wrong in another, so comparing everything against one +# MAGIC table-wide normal hides a whole class of problem: +# MAGIC +# MAGIC - **Retail** — £8,000 of sales is a good day for a small branch and a collapse for a flagship. +# MAGIC - **Payments** — £900 is ordinary for electronics and absurd for a coffee shop. +# MAGIC - **Healthcare** — a lab's reference range differs from another lab's for the same assay. +# MAGIC - **SaaS** — a 2% error rate is normal for one tenant's integration and an incident for another's. +# MAGIC +# MAGIC #### 3. Compared to its own past (`baseline_over_time`) +# MAGIC +# MAGIC When the normal level itself moves, a value that looks fine against the whole history can be wrong for +# MAGIC where things had actually got to: +# MAGIC +# MAGIC - **Manufacturing** — a wearing bearing runs hotter every month; 71°C is fine at week 1 and a warning +# MAGIC at week 40. +# MAGIC - **Subscriptions** — revenue that has grown all year makes last January's figure a bad yardstick. +# MAGIC - **Energy** — demand climbs through a heatwave, so yesterday is the only fair comparison. +# MAGIC - **Logistics** — a new depot ramps for months before its throughput means anything. +# MAGIC +# MAGIC **Section 5 covers this one**, on a dataset that genuinely trends, and measures whether it is worth +# MAGIC turning on before turning it on. It is off by default, because on flat data it measures *worse*. +# MAGIC +# MAGIC ## What DQX does with a timestamp column +# MAGIC +# MAGIC Three different things, depending on what you tell it. Worth knowing up front, because the default is +# MAGIC the one people least expect: +# MAGIC +# MAGIC | You... | DQX... | Use when | +# MAGIC |---|---|---| +# MAGIC | name `columns` explicitly and omit it | ignores it | the clock has nothing to do with what you are looking for | +# MAGIC | list it in `columns` | derives seven calendar features: hour, day of week and month as sine/cosine pairs, plus a weekend flag | 3am is suspicious and 3pm is not | +# MAGIC | pass it as `baseline_over_time` | treats it as the **axis** each metric is measured along, never as a feature | the normal level moves over time | +# MAGIC +# MAGIC Pass no `columns` at all and DQX discovers them for you, which *includes* any timestamp, so the +# MAGIC middle row is what you get by default. It says so at training time rather than leaving you to find out. +# MAGIC +# MAGIC Note what the correlation-aware profile does *not* need: **a timestamp, or any row order.** It models +# MAGIC how metrics relate to each other, not how they behave over time, and it never reads the previous row. +# MAGIC That is what keeps it valid on a streaming DataFrame. +# MAGIC + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ## Prerequisites: Install DQX with Anomaly Support +# MAGIC +# MAGIC ```python +# MAGIC %pip install 'databricks-labs-dqx[anomaly]' +# MAGIC ``` +# MAGIC +# MAGIC **Note**: On ML Runtime or Serverless most dependencies are already present. +# MAGIC + +# COMMAND ---------- +# DBTITLE 1,Install DQX + +dbutils.widgets.text("test_library_ref", "", "Test Library Ref") + +if dbutils.widgets.get("test_library_ref") != "": + %pip install 'databricks-labs-dqx[anomaly] @ {dbutils.widgets.get("test_library_ref")}' +else: + %pip install 'databricks-labs-dqx[anomaly]' + +%restart_python + +# COMMAND ---------- +# DBTITLE 1,Configure catalog and schema + +dbutils.widgets.text("demo_catalog", "main", "Catalog Name") +dbutils.widgets.text("demo_schema", "default", "Schema Name") + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ## Section 1: Setup & Healthy Telemetry +# MAGIC +# MAGIC | Column | Type | Description | +# MAGIC |---|---|---| +# MAGIC | `machine_id` | string | `CNC-01` … `CNC-04` | +# MAGIC | `reading_seq` | int | Reading order — for reference only, the model never uses it | +# MAGIC | `spindle_load` … `throughput` | double | The eight metrics | +# MAGIC | `is_incident` | double | Ground truth, for this demo only — never given to the model | +# MAGIC +# MAGIC Healthy readings are driven by **two hidden factors**: how hard the machine is working, and how hot +# MAGIC it is running. Every metric is a mix of the two plus its own noise, which is exactly why they move +# MAGIC together — and what a correlation-aware detector learns. +# MAGIC + +# COMMAND ---------- +# DBTITLE 1,Setup engines + +import numpy as np +import pyspark.sql.functions as F +from databricks.sdk import WorkspaceClient + +from databricks.labs.dqx.anomaly.anomaly_engine import AnomalyEngine +from databricks.labs.dqx.anomaly.check_funcs import has_no_row_anomalies +from databricks.labs.dqx.config import AnomalyParams, InputConfig, OutputConfig +from databricks.labs.dqx.engine import DQEngine +from databricks.labs.dqx.rule import DQDatasetRule + +catalog = dbutils.widgets.get("demo_catalog") +schema = dbutils.widgets.get("demo_schema") + +ws = WorkspaceClient() +dq_engine = DQEngine(ws) +anomaly_engine = AnomalyEngine(ws) + +print(f"✅ Setup complete — writing to {catalog}.{schema}") + +# COMMAND ---------- +# DBTITLE 1,Prepare a clean model registry + +# Drop the registry so each run of this notebook starts from nothing. Without this, re-running leaves +# every previous run's rows behind and the "registered model" cell below shows a pile of stale +# configurations rather than the one just trained. +registry_table = f"{catalog}.{schema}.dqx_anomaly_models" +spark.sql(f"DROP TABLE IF EXISTS {registry_table}") + +print(f"📋 Model registry: {registry_table}") +print("✅ Registry reset — ready for this run's model") + +# COMMAND ---------- +# DBTITLE 1,A helper for reading detection quality honestly + + +def report_quality(scored_df, label_col: str, severity_col, budget: float, thresholds=(90, 95, 98, 99)): + """Print recall, precision and the best precision the alert count allows, at several thresholds. + + Precision alone is unreadable. If a threshold raises 65 alerts and only 24 rows are genuinely wrong, no + model can exceed 24/65 = 37% however well it ranks, so the ceiling is printed beside what was achieved: + where the two are equal the ranking is optimal and only the alert count is costing anything. + + Expect more alerts than the threshold's share of rows, and do not read that as a fault. `threshold=95` + means "above the 95th percentile of severity seen *during training*", so a batch that contains real + problems clears that line more often than 5% of the time. Measured on this notebook's own data, the + alert rate among the genuinely healthy readings is 5.2% against 5.0% nominal; the rest of the total is + the faults being found. + """ + total = scored_df.count() + faults = scored_df.filter(F.col(label_col) == 1.0).count() + print(f"🎚️ {total:,} rows, {faults} of them genuinely wrong ({faults / total:.1%}).\n") + print("Threshold | Alerts | Caught | Precision | Best possible | Recall") + print("-" * 68) + for threshold in thresholds: + alerts = scored_df.filter(severity_col >= threshold) + n_alerts = alerts.count() + n_caught = alerts.filter(F.col(label_col) == 1.0).count() + ceiling = min(n_alerts, faults) / n_alerts if n_alerts else 0.0 + precision = n_caught / n_alerts if n_alerts else 0.0 + recall = n_caught / faults if faults else 0.0 + marker = " ← used above" if abs(threshold - budget) < 0.01 else "" + print( + f"{threshold:9} | {n_alerts:6} | {n_caught:3}/{faults:<3} | {precision:8.1%} | " + f"{ceiling:11.1%} | {recall:6.1%}{marker}" + ) + + +print("✅ Helper ready") + +# COMMAND ---------- +# DBTITLE 1,How the metrics relate to each other + +METRICS = [ + "spindle_load", + "motor_current", + "coolant_flow", + "bearing_temp", + "vibration", + "hydraulic_pressure", + "air_pressure", + "throughput", +] + +# Each row is one metric's sensitivity to (work rate, heat). Note motor_current tracks spindle_load +# closely, and air_pressure is mostly independent — not everything correlates, which is realistic. +LOADINGS = np.array( + [[0.95, 0.10], [0.90, 0.15], [0.35, 0.80], [0.30, 0.90], [0.70, 0.45], [0.80, 0.20], [0.25, 0.30], [0.85, 0.25]] +).T +BASELINES = np.array([62.0, 18.5, 24.0, 58.0, 2.4, 145.0, 6.2, 480.0]) +SCALES = np.array([9.0, 2.6, 3.4, 6.5, 0.45, 12.0, 0.35, 55.0]) +SCHEMA = "machine_id string, reading_seq int, " + ", ".join(f"{m} double" for m in METRICS) + ", is_incident double" + +print(f"📊 {len(METRICS)} metrics driven by 2 hidden factors") + +# COMMAND ---------- +# DBTITLE 1,Generate telemetry + + +def generate_telemetry(n_rows: int, seed: int, break_correlation: bool = False): + """Telemetry driven by two shared latent factors. + + With *break_correlation*, a contiguous block has three metrics decoupled by **permuting their values + among those rows** — the same values, reordered. Every metric's own distribution is preserved + exactly, so only the joint behaviour changes. + """ + rng = np.random.default_rng(seed) + factors = rng.normal(0.0, 1.0, size=(n_rows, 2)) + values = BASELINES + SCALES * (factors @ LOADINGS + rng.normal(0.0, 0.18, size=(n_rows, len(METRICS)))) + labels = np.zeros(n_rows) + + if break_correlation: + start, length = int(n_rows * 0.72), max(4, int(n_rows * 0.04)) + block = values[start : start + length, :3] + values[start : start + length, :3] = np.roll(block, shift=1, axis=0) + labels[start : start + length] = 1.0 + + rows = [(f"CNC-{(i % 4) + 1:02d}", i, *[float(v) for v in values[i]], float(labels[i])) for i in range(n_rows)] + return spark.createDataFrame(rows, SCHEMA) + + +# COMMAND ---------- +# DBTITLE 1,Create the training table + +# Nothing is cached in this notebook: PERSIST is unsupported on serverless compute, which is what most +# readers will run this on. These frames are small local relations built from seeded RNGs, so +# recomputation is both cheap and deterministic. +print("🔄 Generating healthy telemetry...\n") + +healthy_df = generate_telemetry(4000, seed=5) +healthy_table = f"{catalog}.{schema}.fleet_telemetry_healthy" +healthy_df.write.mode("overwrite").option("overwriteSchema", "true").saveAsTable(healthy_table) + +print("📊 Sample of healthy readings:") +display(healthy_df.limit(10)) + +print(f"\n✅ {healthy_df.count():,} readings saved to {healthy_table}") + +# COMMAND ---------- +# DBTITLE 1,Confirm the metrics really do move together + +PAIRS = [ + ("spindle_load", "motor_current", "cutting harder draws more current"), + ("bearing_temp", "coolant_flow", "coolant responds to heat"), + ("spindle_load", "air_pressure", "barely related — and that is realistic"), +] + +correlations = spark.table(healthy_table).select( + *[F.round(F.corr(left, right), 3).alias(f"{left}__{right}") for left, right, _ in PAIRS] +).first() + +print("🔍 Correlations on healthy data — the premise this detector relies on:\n") +print(f"{'metric pair':<34}{'correlation':>13} why") +print("-" * 78) + +for left, right, reason in PAIRS: + print(f"{left + ' vs ' + right:<34}{correlations[f'{left}__{right}']:>13.3f} {reason}") + +print("\n💡 Strong pairs are what the detector exploits. Not every metric relates to every other,") +print(" and real telemetry looks exactly like this.") + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ## Section 2: The Incident +# MAGIC +# MAGIC Two hours in which spindle load, motor current and coolant flow stop tracking each other, while +# MAGIC every single reading stays inside the range it has always occupied. +# MAGIC + +# COMMAND ---------- +# DBTITLE 1,Inject the correlation break + +print("🔄 Generating a batch containing the incident...\n") + +incident_df = generate_telemetry(1500, seed=77, break_correlation=True) +incident_table = f"{catalog}.{schema}.fleet_telemetry_incident" +incident_df.write.mode("overwrite").option("overwriteSchema", "true").saveAsTable(incident_table) + +total_readings = incident_df.count() +injected = incident_df.filter(F.col("is_incident") == 1.0).count() + +print(f"✅ {total_readings:,} readings saved to {incident_table}") +print(f" {injected} of them during the incident") + +# COMMAND ---------- +# DBTITLE 1,Prove no range check could catch it + +readings = spark.table(incident_table) +during = readings.filter(F.col("is_incident") == 1.0) +outside = readings.filter(F.col("is_incident") == 0.0) + +print("🔍 Each metric's healthy range vs its range during the incident:\n") +print(f"{'metric':<18}{'healthy range':>22}{'during incident':>22} verdict") +print("-" * 74) + +for metric in METRICS[:3]: + healthy = outside.select(F.min(metric).alias("lo"), F.max(metric).alias("hi")).first() + broken = during.select(F.min(metric).alias("lo"), F.max(metric).alias("hi")).first() + inside = broken["lo"] >= healthy["lo"] and broken["hi"] <= healthy["hi"] + healthy_range = f"{healthy.lo:.1f} – {healthy.hi:.1f}" + broken_range = f"{broken.lo:.1f} – {broken.hi:.1f}" + print(f"{metric:<18}{healthy_range:>22}{broken_range:>22} {'inside ✅' if inside else 'OUTSIDE'}") + +print("\n⚠️ Every incident reading sits inside the healthy range for its own metric.") +print(" No threshold, no range check and no per-metric z-score can separate these rows.") + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ## Section 3: Train with `profile="correlation"` +# MAGIC +# MAGIC One word selects the correlation-aware detector. Everything else is unchanged: the same automatic +# MAGIC feature engineering, the same registry, the same `has_no_row_anomalies` check, the same +# MAGIC contributions and AI explanations. +# MAGIC +# MAGIC It measures how far a reading sits from normal **once the relationships between metrics are +# MAGIC accounted for**. A load/current pair that never co-occurs on healthy data is far away in that +# MAGIC space even though each value sits mid-range. +# MAGIC +# MAGIC `baseline_by=[]` keeps the comparison across the whole fleet, because these four machines are +# MAGIC interchangeable and share one operating envelope. Pass `baseline_by=["machine_id"]` instead when +# MAGIC each machine has its own normal. +# MAGIC + +# COMMAND ---------- +# DBTITLE 1,Train the model + +print("🎯 Training the correlation-aware model...\n") + +model_name = f"{catalog}.{schema}.fleet_telemetry_monitor" + +trained = anomaly_engine.train( + df=spark.table(healthy_table), + model_name=model_name, + registry_table=registry_table, + columns=METRICS, + baseline_by=[], + profile="correlation", + # Whole table rather than the default sample: 4,000 readings is small enough that sampling only makes + # the numbers printed below vary between runs, because the sample is drawn per partition. + params=AnomalyParams(sample_fraction=1.0), +) + +print(f"\n✅ Model trained: {trained}") + +# COMMAND ---------- +# DBTITLE 1,The registry records which detector was used + +print("📋 Registered model:\n") + +display( + spark.table(registry_table) + .filter(F.col("identity.model_name") == trained) + .selectExpr( + "identity.model_name", + "identity.algorithm", + "training.hyperparameters['covariance'] as covariance", + "training.training_rows", + ) +) + +print("💡 Scoring reads the algorithm back off the registry, so you never repeat the choice.") +print(" It also trains a single model rather than an ensemble, because it is deterministic.") + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ## Section 4: Score and Read the Explanation +# MAGIC +# MAGIC The detector explains itself by leaving each metric out in turn and reporting how much of the +# MAGIC anomaly disappears — so contributions work here with no SHAP involved. +# MAGIC + +# COMMAND ---------- +# DBTITLE 1,Apply the anomaly check + +print("🔍 Scoring the incident batch...\n") + +anomaly_check = [ + DQDatasetRule( + criticality="error", + check_func=has_no_row_anomalies, + check_func_kwargs={ + "model_name": trained, + "registry_table": registry_table, + "threshold": 95.0, + }, + ) +] + +# One DQX call: name the input table, name the output table. Writing the result rather than keeping a +# lazy DataFrame also matters here — AI explanations call an LLM through ai_query *inside* the scoring +# plan, so each action on an unmaterialised result would call the model again. +scored_table = f"{catalog}.{schema}.fleet_scored" + +dq_engine.apply_checks_and_save_in_table( + input_config=InputConfig(location=incident_table), + output_config=OutputConfig(location=scored_table, mode="overwrite", options={"overwriteSchema": "true"}), + checks=anomaly_check, +) + +scored = spark.table(scored_table) +anomaly = F.element_at(F.col("_dq_info"), 1).getField("anomaly") +flagged = scored.filter(anomaly.getField("is_anomaly")) + +print(f"✅ Scoring complete — {flagged.count()} of {total_readings:,} readings flagged") + +# COMMAND ---------- +# DBTITLE 1,Which relationships broke + +caught = flagged.filter(F.col("is_incident") == 1.0).count() +n_alerts = flagged.count() +print(f"🔝 Caught {caught} of the {injected} incident readings — none of which any range check could see.") +print(f" That cost {n_alerts} alerts on {total_readings:,} readings, so {caught / n_alerts:.0%} of them were real.") +print(" The next cell sweeps the threshold, which is the honest way to read that number.\n") + +display( + flagged.select( + "machine_id", + "reading_seq", + F.round("spindle_load", 1).alias("spindle_load"), + F.round("motor_current", 1).alias("motor_current"), + anomaly.getField("severity_percentile").alias("severity"), + anomaly.getField("contributions").alias("contributions"), + ) + .orderBy(F.desc("severity")) + .limit(10) +) + +# COMMAND ---------- +# DBTITLE 1,What the alert budget actually buys + +report_quality(scored, "is_incident", anomaly.getField("severity_percentile"), budget=95.0) + +print("\n💡 More alerts than 5% of rows is correct, not a fault: the extra ones are the faults being") +print(" found. The alert count grows with the size of the problem instead of being capped at a fixed") +print(" share, which is what you want from a quality check.") +print(" What the table is for is the tradeoff. Raising the threshold here buys a large drop in false") +print(" alarms for a small loss of recall, because severity ranks the incident readings well above the") +print(" healthy ones. Severity is stored for every row, so producing this costs nothing and no") +print(" rescoring, and it is how to pick the number on your own data rather than inheriting 95.") + +# COMMAND ---------- +# DBTITLE 1,Why each group was flagged, in plain language + +# One explanation per *pattern*, not per row: readings driven by the same broken relationship share a +# single ai_query call, so cost scales with how many distinct problems there are rather than with how many +# readings have them. Grouping the display the same way is the only way to see that. +print("🤖 AI explanations. One call per pattern, however many readings share it:\n") + +display( + flagged.groupBy( + anomaly.getField("ai_explanation").getField("narrative").alias("narrative"), + anomaly.getField("ai_explanation").getField("business_impact").alias("impact"), + anomaly.getField("ai_explanation").getField("action").alias("action"), + ) + .agg(F.count("*").alias("readings"), F.min("machine_id").alias("example_machine")) + .filter(F.col("narrative").isNotNull()) + .orderBy(F.desc("readings")) +) + +print("💡 Note the wording: broken *relationships*, not abnormal metrics. DQX tells the model which") +print(" detector produced the contributions, so the explanation describes the right kind of problem.") + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ## Section 5: (Optional) A Third Question — Is This Normal *For Now*? +# MAGIC +# MAGIC Everything above compared each reading against the fleet's normal. There is a third question, and a +# MAGIC maintenance engineer asks it constantly: +# MAGIC +# MAGIC > *Bearing temperature is 71°C. That is fine for this machine. Is it fine for **1,800 hours in**?* +# MAGIC +# MAGIC A wearing bearing has a **rising baseline**. A reading that is ordinary against the whole service +# MAGIC history can be well above where the wear curve had actually got to. `baseline_over_time` fits each +# MAGIC metric's expected level as a function of time and compares against *that*. +# MAGIC +# MAGIC | Question | Argument | +# MAGIC |---|---| +# MAGIC | Unusual on its own, or in combination? | `profile` | +# MAGIC | Unusual for its own group? | `baseline_by` | +# MAGIC | Unusual for its own point in time? | `baseline_over_time` | +# MAGIC +# MAGIC **This needs a different dataset, and that is the lesson.** The telemetry above is stationary by +# MAGIC construction — no trend, no timestamp — which is exactly the shape where a temporal baseline has +# MAGIC nothing to remove. So we generate a short service history that genuinely wears. + +# COMMAND ---------- +# DBTITLE 1,Generate a service history with a rising baseline + +import datetime + +WEAR_START = datetime.datetime(2025, 1, 6) + + +def generate_wear_history(n_hours: int, seed: int, late_fault: bool = False): + """Bearing temperature and vibration that drift upward as the bearing wears. + + With *late_fault*, a block near the end is held at the level it had 600 hours earlier. Every value + stays inside the range the whole history covers, so nothing about it is extreme -- it is simply wrong + for how worn the bearing should be by then. + """ + rng = np.random.default_rng(seed) + hours = np.arange(n_hours) + temp = 52.0 + 0.011 * hours + 4.0 * np.sin(2 * np.pi * hours / 24.0) + rng.normal(0, 1.1, n_hours) + vib = 1.7 + 0.0006 * hours + rng.normal(0, 0.08, n_hours) + labels = np.zeros(n_hours) + + if late_fault: + start, length = int(n_hours * 0.80), max(6, int(n_hours * 0.05)) + temp[start : start + length] -= 0.011 * 600 + vib[start : start + length] -= 0.0006 * 600 + labels[start : start + length] = 1.0 + + rows = [ + (WEAR_START + datetime.timedelta(hours=int(h)), float(temp[i]), float(vib[i]), float(labels[i])) + for i, h in enumerate(hours) + ] + return spark.createDataFrame(rows, "reading_ts timestamp, bearing_temp double, vibration double, is_incident double") + + +wear_train = f"{catalog}.{schema}.bearing_wear_history" +wear_test = f"{catalog}.{schema}.bearing_wear_recent" +generate_wear_history(24 * 45, seed=11).write.mode("overwrite").saveAsTable(wear_train) +generate_wear_history(24 * 20, seed=12, late_fault=True).write.mode("overwrite").saveAsTable(wear_test) + +print(f"📊 45 days of hourly service history, and 20 days of recent readings with a fault") + +# COMMAND ---------- +# DBTITLE 1,Measure whether the data trends before deciding + +# The decision comes first, and it is measurable. Subtracting a fitted expectation from a metric with no +# temporal structure removes real signal and adds the fit's own error, so this is not a free switch. +from databricks.labs.dqx.anomaly.temporal_advisory import measure_trend_strength + +WEAR_METRICS = ["bearing_temp", "vibration"] +wear_trend = measure_trend_strength(spark.table(wear_train), "reading_ts", WEAR_METRICS) +flat_trend = measure_trend_strength( + spark.table(healthy_table).withColumn("fake_ts", F.expr("timestamp('2025-01-06') + make_interval(0,0,0,0,reading_seq)")), + "fake_ts", + METRICS, +) + +print(f"📊 Wear history: {wear_trend:.1%} of variance explained by trend and seasonality ✅ use it") +print(f"📊 Fleet telemetry: {flat_trend:.1%} ⚠️ nothing to remove — DQX would warn, so leave it off") + +# COMMAND ---------- +# DBTITLE 1,Train with a time axis + +# reading_ts is named as the axis, so it is NOT a feature: no cyclical calendar columns are derived from +# it. That matters — those help a calendar-contextual anomaly and measurably hurt otherwise. +wear_model = f"{catalog}.{schema}.bearing_wear_model" + +anomaly_engine.train( + df=spark.table(wear_train), + model_name=wear_model, + registry_table=registry_table, + columns=WEAR_METRICS, + baseline_over_time="reading_ts", + baseline_by=[], + params=AnomalyParams(sample_fraction=1.0), +) + +print(f"🎯 Trained with an expected level per metric over time") + +# COMMAND ---------- +# DBTITLE 1,Score, and see what "wrong for now" looks like + +# The default, deliberately, even though a tighter cutoff scores better on this fixture. An earlier draft +# pinned 99 from one run's sweep and the next run's model ranked slightly differently, at which point 99 was +# dropping real faults. A demo that hardcodes a tuned number teaches the wrong lesson anyway: score at the +# default, read the sweep the next cell prints, then choose. That is the loop, and it is cheap because +# severity is stored for every row. +WEAR_THRESHOLD = 95 + +wear_checks = [ + { + "criticality": "error", + "check": { + "function": "has_no_row_anomalies", + "arguments": { + "model_name": wear_model, + "registry_table": registry_table, + "threshold": WEAR_THRESHOLD, + }, + }, + } +] +wear_scored = f"{catalog}.{schema}.bearing_wear_scored" +dq_engine.apply_checks_by_metadata_and_save_in_table( + input_config=InputConfig(location=wear_test), + output_config=OutputConfig(location=wear_scored, mode="overwrite"), + checks=wear_checks, +) + +anomaly = F.col("_dq_info")[0].getField("anomaly") +wear_result = spark.table(wear_scored) +caught = wear_result.filter(anomaly.getField("is_anomaly") & (F.col("is_incident") == 1.0)).count() +total = wear_result.filter(F.col("is_incident") == 1.0).count() +print(f"🔍 Caught {caught} of {total} rows that were wrong for how worn the bearing should have been.\n") + +# The ranking is what to judge, and a sweep is the only way to see it. On this fixture almost every fault +# sits above almost every healthy reading, so a much tighter cutoff costs little or no recall while removing +# most of the false alarms. That is the shape worth recognising: when precision equals its ceiling at every +# row count, the model has ranked correctly and the cutoff is the only decision left. +report_quality(wear_result, "is_incident", anomaly.getField("severity_percentile"), budget=WEAR_THRESHOLD) + +# COMMAND ---------- +# DBTITLE 1,Read the contributions, which name the expected level + +print("💡 Contributions name the metric, once, however many ways the model compared it.") +print(" Read them as 'this metric mattered', not 'this metric's value was extreme': every one of") +print(" these readings sits inside the history's own range, and it is baseline_over_time being set") +print(" that tells you the comparison they failed.") +display( + spark.table(wear_scored) + .filter(anomaly.getField("is_anomaly")) + .select( + "reading_ts", + "bearing_temp", + anomaly.getField("severity_percentile").alias("severity"), + anomaly.getField("contributions").alias("contributions"), + anomaly.getField("is_stale_baseline").alias("extrapolating"), + ) + .orderBy(F.desc("severity")) + .limit(8) +) + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ### When *not* to reach for `baseline_over_time` +# MAGIC +# MAGIC The cell that measured both datasets is the point of this section. The honest cases against it: +# MAGIC +# MAGIC - **A largely stationary metric**, like the fleet telemetry above. On real server telemetry that +# MAGIC arrives already normalised, the same transform measured *worse* than leaving it off. +# MAGIC - **A short training window.** A daily shape needs several complete days to be identifiable at all. +# MAGIC DQX fits one only where the window supports it, and logs the period it skipped and why. +# MAGIC - **With `profile="tabular"`, keep other datetime columns out of `columns`.** Calendar features on +# MAGIC top of the residual measured worse on every anomaly shape tested. +# MAGIC +# MAGIC It is also **not a forecaster**. It models the level expected *at* a time; it does not predict the +# MAGIC next value, and it never reads the previous row — which is what keeps scoring valid on a stream. +# MAGIC +# MAGIC `is_stale_baseline` marks rows past the window the expectation was fitted on. The score is still +# MAGIC produced, because near the boundary it is still accurate; treat the flag as a signal to retrain. + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ## Summary & Next Steps +# MAGIC +# MAGIC **Key takeaways:** +# MAGIC - Some failures are **broken relationships**, not extreme values. Every gauge reads normal and the +# MAGIC machine is still in trouble. +# MAGIC - Per-metric alerts cannot see those, however well tuned — verified above, not asserted. +# MAGIC - `profile="correlation"` switches to a detector that models how metrics move together. One word; +# MAGIC everything else is unchanged. +# MAGIC - It needs no timestamp column and trains a single model rather than an ensemble. +# MAGIC +# MAGIC **Apply to your data:** +# MAGIC ```python +# MAGIC model = anomaly_engine.train( +# MAGIC df=spark.table("your_catalog.your_schema.your_metrics"), +# MAGIC model_name="your_catalog.your_schema.your_model", +# MAGIC registry_table="your_catalog.your_schema.dqx_anomaly_models", +# MAGIC profile="correlation", +# MAGIC ) +# MAGIC +# MAGIC # Then score straight from one table into another. +# MAGIC dq_engine.apply_checks_and_save_in_table( +# MAGIC input_config=InputConfig(location="your_catalog.your_schema.new_readings"), +# MAGIC output_config=OutputConfig(location="your_catalog.your_schema.scored"), +# MAGIC checks=checks, +# MAGIC ) +# MAGIC ``` +# MAGIC +# MAGIC **What this does not do**, so you can plan around it: +# MAGIC - **Trend.** A steadily growing metric eventually leaves the range it was trained on. Model a rate +# MAGIC or a ratio rather than a running level, and retrain on a schedule. +# MAGIC - **A single metric.** With nothing to correlate against, use a rule or a threshold. +# MAGIC - **Forecasting.** DQX judges readings against learned normal; it does not predict the next value. +# MAGIC +# MAGIC The SMD figures quoted earlier are a large gain on the data this detector is for, and they are *not* +# MAGIC state of the art for multivariate time-series anomaly detection. Published figures near 0.80 F1 use +# MAGIC point adjustment, which [Kim et al. (AAAI 2022)](https://arxiv.org/abs/2109.05257) showed random +# MAGIC scores also reach, so they are not a fair comparison in either direction. +# MAGIC + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ### 📚 Resources +# MAGIC +# MAGIC - [Choosing a profile](https://databrickslabs.github.io/dqx/docs/guide/row_anomaly_detection#choosing-a-profile) +# MAGIC - [Benchmarks](https://databrickslabs.github.io/dqx/docs/reference/benchmarks#anomaly-benchmarks) — measured detection quality and timings +# MAGIC - [Row Anomaly Detection guide](https://databrickslabs.github.io/dqx/docs/guide/row_anomaly_detection) +# MAGIC +# MAGIC ### 🎉 You're Ready! +# MAGIC +# MAGIC You now understand: +# MAGIC - ✅ The difference between an extreme value and a broken relationship +# MAGIC - ✅ When to reach for `profile="correlation"` instead of the default +# MAGIC - ✅ Why it needs no timestamp column +# MAGIC - ✅ How to read contributions and AI explanations for a correlation break +# MAGIC +# MAGIC **Start watching the relationships, not just the gauges!** 🚀 +# MAGIC diff --git a/demos/dqx_demo_anomaly_tabular_transactions.py b/demos/dqx_demo_anomaly_tabular_transactions.py new file mode 100644 index 000000000..3b6b04fcb --- /dev/null +++ b/demos/dqx_demo_anomaly_tabular_transactions.py @@ -0,0 +1,765 @@ +# Databricks notebook source +# MAGIC %md +# MAGIC # 🔍 Finding the Transactions Your Rules Will Never Catch +# MAGIC +# MAGIC ## Learn Row Anomaly Detection on Business Records in 10 Minutes +# MAGIC +# MAGIC **What you'll do:** +# MAGIC - Write the quality rules a good payments team would already have +# MAGIC - Watch them pass a batch that contains real problems +# MAGIC - Train a DQX anomaly model with no thresholds and no labels +# MAGIC - Read *why* each row was flagged, in plain language +# MAGIC +# MAGIC **Dataset**: Card transactions across twelve merchant categories (no domain expertise required) +# MAGIC + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ## The problem: a row can be wrong without any value being wrong +# MAGIC +# MAGIC A payments team has good rules. Amount is positive and under the card limit. Quantity is at least +# MAGIC one. The merchant category is one they support. Every rule passes, every day, and the dashboards +# MAGIC are green. +# MAGIC +# MAGIC Then a reconciliation breaks, and someone finds a **£4 grocery basket with 38 items** in it, and a +# MAGIC **£900 coffee**. Both were inside every threshold. Neither was flagged. +# MAGIC +# MAGIC **Known vs unknown issues** +# MAGIC - **Known unknowns**: nulls, ranges, formats. Write a rule — it is cheap, clear and versioned. +# MAGIC - **Unknown unknowns**: a combination of values that is individually ordinary and jointly absurd. +# MAGIC There is no single column to write the rule against. +# MAGIC +# MAGIC **"Normal" also depends on context.** £900 is unremarkable for electronics and absurd for coffee. A +# MAGIC threshold that catches the coffee rejects half the laptops. DQX handles this with `baseline_by`, +# MAGIC which judges every row against **its own group's** normal rather than the whole table's. +# MAGIC +# MAGIC Use rules *and* anomaly detection. Rules catch what you can describe; anomaly detection covers +# MAGIC what is left. +# MAGIC + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ## Prerequisites: Install DQX with Anomaly Support +# MAGIC +# MAGIC ```python +# MAGIC %pip install 'databricks-labs-dqx[anomaly]' +# MAGIC ``` +# MAGIC +# MAGIC **Note**: On ML Runtime or Serverless most dependencies are already present. +# MAGIC + +# COMMAND ---------- +# DBTITLE 1,Install DQX + +dbutils.widgets.text("test_library_ref", "", "Test Library Ref") + +if dbutils.widgets.get("test_library_ref") != "": + %pip install 'databricks-labs-dqx[anomaly] @ {dbutils.widgets.get("test_library_ref")}' +else: + %pip install 'databricks-labs-dqx[anomaly]' + +%restart_python + +# COMMAND ---------- +# DBTITLE 1,Configure catalog and schema + +dbutils.widgets.text("demo_catalog", "main", "Catalog Name") +dbutils.widgets.text("demo_schema", "default", "Schema Name") + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ## Section 1: Setup & Data Generation +# MAGIC +# MAGIC | Column | Type | Description | +# MAGIC |---|---|---| +# MAGIC | `transaction_id` | string | Unique transaction reference | +# MAGIC | `transaction_time` | timestamp | When the card was used | +# MAGIC | `amount` | double | Total basket value, GBP | +# MAGIC | `item_count` | int | Items in the basket | +# MAGIC | `merchant_category` | string | One of twelve categories — the **baseline group** | +# MAGIC | `channel` | string | `chip_and_pin`, `contactless` or `online` | +# MAGIC | `is_anomaly` | double | Ground truth, for this demo only — never given to the model | +# MAGIC +# MAGIC Each category has its own **typical basket**: a coffee is a couple of pounds for one item, a laptop +# MAGIC several hundred for one, a weekly shop tens of pounds across dozens. That structure is the point — +# MAGIC it is what makes a single global threshold useless. +# MAGIC + +# COMMAND ---------- +# DBTITLE 1,Setup engines + +from datetime import datetime, timedelta + +import numpy as np +import pyspark.sql.functions as F +from databricks.sdk import WorkspaceClient + +from databricks.labs.dqx.anomaly.anomaly_engine import AnomalyEngine +from databricks.labs.dqx.anomaly.check_funcs import has_no_row_anomalies +from databricks.labs.dqx.check_funcs import is_in_range, is_not_null +from databricks.labs.dqx.config import AnomalyParams, InputConfig, OutputConfig +from databricks.labs.dqx.engine import DQEngine +from databricks.labs.dqx.rule import DQDatasetRule, DQRowRule + +catalog = dbutils.widgets.get("demo_catalog") +schema = dbutils.widgets.get("demo_schema") + +ws = WorkspaceClient() +dq_engine = DQEngine(ws) +anomaly_engine = AnomalyEngine(ws) + +print(f"✅ Setup complete — writing to {catalog}.{schema}") + +# COMMAND ---------- +# DBTITLE 1,Prepare a clean model registry + +# Drop the registry so each run of this notebook starts from nothing. Without this, re-running leaves +# every previous run's rows behind and the "registered model" cell below shows a pile of stale +# configurations rather than the one just trained. +registry_table = f"{catalog}.{schema}.dqx_anomaly_models" +spark.sql(f"DROP TABLE IF EXISTS {registry_table}") + +print(f"📋 Model registry: {registry_table}") +print("✅ Registry reset — ready for this run's model") + +# COMMAND ---------- +# DBTITLE 1,Typical basket per merchant category + +# (typical unit price, typical item count). These are the patterns a model has to learn; +# nobody writes them down as rules. +CATEGORY_BASKETS = { + "coffee_shop": (3.20, 1.4), + "grocery": (2.10, 24.0), + "fuel": (68.00, 1.0), + "electronics": (420.00, 1.1), + "pharmacy": (8.50, 2.6), + "restaurant": (23.00, 2.2), + "clothing": (38.00, 2.4), + "transport": (2.80, 1.0), + "streaming": (9.99, 1.0), + "hardware": (14.00, 3.8), + "books": (11.00, 1.7), + "gym": (42.00, 1.0), +} +CHANNELS = ("chip_and_pin", "contactless", "online") +START = datetime(2024, 1, 1) +SCHEMA = ( + "transaction_id string, transaction_time timestamp, amount double, " + "item_count int, merchant_category string, channel string, is_anomaly double" +) + +print(f"📊 {len(CATEGORY_BASKETS)} merchant categories, each with its own basket shape") + +# COMMAND ---------- +# DBTITLE 1,The three shapes of implausible row + + +def make_implausible(rng, category: str, items: int): + """Return (category, item_count, amount) for a row that is ordinary per column and absurd overall.""" + kind = rng.integers(3) + if kind == 0: + # A grocery-sized basket at a coffee-shop price. £4 and 38 items are each ordinary + # somewhere in this table; together they are not. + return category, int(rng.integers(30, 45)), round(rng.uniform(3.0, 6.0), 2) + if kind == 1: + # An electronics-sized amount on a single coffee — still inside the global amount range. + return "coffee_shop", 1, round(rng.uniform(600.0, 950.0), 2) + # A plausible amount and count, for the wrong category: a £420 single grocery item. + return "grocery", 1, round(rng.uniform(380.0, 460.0), 2) + + +# COMMAND ---------- +# DBTITLE 1,Generate transactions + + +def generate_transactions(n_rows: int, seed: int, inject: bool = False): + """Transactions whose amount and item count follow their category's basket shape.""" + rng = np.random.default_rng(seed) + categories = list(CATEGORY_BASKETS) + rows = [] + + for i in range(n_rows): + category = categories[rng.integers(len(categories))] + unit_price, typical_items = CATEGORY_BASKETS[category] + items = max(1, int(rng.normal(typical_items, max(0.4, typical_items * 0.25)))) + amount = round(items * unit_price * rng.uniform(0.82, 1.18), 2) + is_anomaly = 0.0 + + if inject and rng.random() < 0.02: + category, items, amount = make_implausible(rng, category, items) + is_anomaly = 1.0 + + when = START + timedelta(days=int(rng.integers(0, 90)), hours=int(rng.integers(7, 22))) + channel = CHANNELS[rng.integers(len(CHANNELS))] + rows.append((f"TXN{i:06d}", when, amount, items, category, channel, is_anomaly)) + + return spark.createDataFrame(rows, SCHEMA) + + +# COMMAND ---------- +# DBTITLE 1,Create the training table + +print("🔄 Generating three months of clean history...\n") + +history_df = generate_transactions(6000, seed=11) +history_table = f"{catalog}.{schema}.card_transactions_history" +history_df.write.mode("overwrite").option("overwriteSchema", "true").saveAsTable(history_table) + +print("📊 Sample of historical transactions:") +display(history_df.limit(10)) + +print(f"\n✅ {history_df.count():,} transactions saved to {history_table}") + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ## Section 2: The Rules a Good Team Already Has +# MAGIC +# MAGIC These are sensible rules, not strawmen — amount present, amount in range, item count in range. +# MAGIC They are exactly what you should write, and they will catch a great deal of real breakage. +# MAGIC +# MAGIC They will not catch what we are about to inject. +# MAGIC + +# COMMAND ---------- +# DBTITLE 1,Define the rules + +rules = [ + DQRowRule(check_func=is_not_null, column="amount", criticality="error"), + DQRowRule( + check_func=is_in_range, + column="amount", + check_func_kwargs={"min_limit": 0.01, "max_limit": 2000.0}, + criticality="error", + ), + DQRowRule( + check_func=is_in_range, + column="item_count", + check_func_kwargs={"min_limit": 1, "max_limit": 60}, + criticality="error", + ), +] + +print(f"✅ {len(rules)} rules defined") + +# COMMAND ---------- +# DBTITLE 1,Generate a new batch containing real problems + +# Nothing is cached in this notebook: PERSIST is unsupported on serverless compute, which is what most +# readers will run this on. These frames are small local relations built from seeded RNGs, so +# recomputation is both cheap and deterministic. +print("🔄 Generating a new batch with problems injected...\n") + +new_df = generate_transactions(1500, seed=99, inject=True) +new_table = f"{catalog}.{schema}.card_transactions_new" +new_df.write.mode("overwrite").option("overwriteSchema", "true").saveAsTable(new_table) + +total_new = new_df.count() +injected = new_df.filter(F.col("is_anomaly") == 1.0).count() + +print(f"✅ {total_new:,} new transactions saved to {new_table}") +print(f" {injected} of them jointly implausible") + +# COMMAND ---------- +# DBTITLE 1,Apply the rules + +print("🔍 Applying the rule-based checks...\n") + +rule_results = dq_engine.apply_checks(new_df, rules) +caught_by_rules = rule_results.filter(F.col("_errors").isNotNull() & (F.col("is_anomaly") == 1.0)).count() + +print(f"⚠️ Rules caught {caught_by_rules} of the {injected} implausible transactions.") +print(" Every injected row sits inside every threshold — each value is ordinary on its own.") +print(" Widening the rules cannot help; tightening them would reject legitimate transactions.") + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ## Section 3: Train the Anomaly Model +# MAGIC +# MAGIC Note what is **not** passed: no thresholds, no per-category limits, no labels. DQX learns the +# MAGIC patterns from the history table. +# MAGIC +# MAGIC `baseline_by=["merchant_category"]` is the one modelling decision, and it says what a payments +# MAGIC analyst already knows: **judge each transaction against its own category**. DQX then adds, for every +# MAGIC metric, its deviation from that category's own median — so one model can hold "£900 is normal for +# MAGIC electronics and extreme for coffee". +# MAGIC +# MAGIC `profile="tabular"` is the default and is right for independent records like these. Use +# MAGIC `profile="correlation"` for repeated multivariate measurements such as machine telemetry — see the +# MAGIC companion notebook. +# MAGIC +# MAGIC **Why `transaction_time` is not in `columns`.** A datetime column becomes seven features (cyclical +# MAGIC hour, day of week and month, plus a weekend flag). That is valuable when *when* something happened +# MAGIC carries meaning — off-hours activity, weekend spikes. In this dataset it does not, so those seven +# MAGIC features would be noise, and noise costs you twice: it dilutes the columns that do carry signal, and +# MAGIC it manufactures "unusual timing" alerts that spend your alert budget. Measured on this data, +# MAGIC excluding it lifts recall at threshold 98 from **77% to 100%** and precision from 38% to 50%. +# MAGIC +# MAGIC The rule is general: **feed a column only if it relates to the anomalies you care about.** Every +# MAGIC extra column adds features, and features you do not need make the ones you do harder to see. Note +# MAGIC that auto-discovery — `train()` with no `columns` — would have included the timestamp here. +# MAGIC + +# COMMAND ---------- +# DBTITLE 1,Train the model + +print("🎯 Training the anomaly model...\n") + +model_name = f"{catalog}.{schema}.card_transactions_monitor" + +trained = anomaly_engine.train( + df=spark.table(history_table), + model_name=model_name, + registry_table=registry_table, + # transaction_time is deliberately excluded — see the note above. + columns=["amount", "item_count"], + baseline_by=["merchant_category"], + profile="tabular", + # By default DQX trains on a sample, which is what makes training a table of a billion rows + # affordable. On 6,000 it only adds variance: the sample is seeded, but it is drawn per partition, so + # a different partition count draws different rows and the numbers printed below move between runs. + params=AnomalyParams(sample_fraction=1.0), +) + +print(f"\n✅ Model trained: {trained}") + +# COMMAND ---------- +# DBTITLE 1,What DQX engineered for you + +print("📋 Registered model and its engineered features:\n") + +display( + spark.table(registry_table) + .filter(F.col("identity.model_name") == trained) + .selectExpr( + "identity.algorithm", + "training.columns", + "grouping.baseline_by", + "training.training_rows", + "from_json(features.feature_metadata, 'engineered_feature_names array')" + ".engineered_feature_names as engineered_features", + ) +) + +print("💡 Note the `_rel_baseline` features — each metric's deviation from its own category's median.") +print(" Four features from two columns, and every one of them carries signal.") + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ## Section 4: Score and Triage +# MAGIC +# MAGIC One check. Feature contributions and AI explanations are **on by default**. +# MAGIC + +# COMMAND ---------- +# DBTITLE 1,Apply the anomaly check + +print("🔍 Scoring the new batch...\n") + +anomaly_check = [ + DQDatasetRule( + criticality="error", + check_func=has_no_row_anomalies, + check_func_kwargs={ + "model_name": trained, + "registry_table": registry_table, + "threshold": 95.0, + }, + ) +] + +# One DQX call: name the input table, name the output table. Writing the result rather than keeping a +# lazy DataFrame also matters here — AI explanations call an LLM through ai_query *inside* the scoring +# plan, so each action on an unmaterialised result would call the model again. +scored_table = f"{catalog}.{schema}.transactions_scored" + +dq_engine.apply_checks_and_save_in_table( + input_config=InputConfig(location=new_table), + output_config=OutputConfig(location=scored_table, mode="overwrite", options={"overwriteSchema": "true"}), + checks=anomaly_check, +) + +scored = spark.table(scored_table) +anomaly = F.element_at(F.col("_dq_info"), 1).getField("anomaly") +flagged = scored.filter(anomaly.getField("is_anomaly")) + +print(f"✅ Scoring complete — {flagged.count()} of {total_new:,} rows flagged") + +# COMMAND ---------- +# DBTITLE 1,Which columns combined badly + +caught = flagged.filter(F.col("is_anomaly") == 1.0).count() +print(f"🔝 Anomaly detection caught {caught} of the {injected} implausible transactions.\n") + +display( + flagged.select( + "transaction_id", + "merchant_category", + "amount", + "item_count", + anomaly.getField("severity_percentile").alias("severity"), + anomaly.getField("contributions").alias("contributions"), + ) + .orderBy(F.desc("severity")) + .limit(10) +) + +# COMMAND ---------- +# DBTITLE 1,Why each group was flagged, in plain language + +# One explanation per *pattern*, not per row: rows driven by the same combination of features share a +# single ai_query call, so the cost scales with how many distinct problems there are rather than with how +# many rows have them. Grouping the display the same way is the only way to see that. +print("🤖 AI explanations. One call per pattern, however many rows share it:\n") + +display( + flagged.groupBy( + anomaly.getField("ai_explanation").getField("top_features").alias("pattern"), + anomaly.getField("ai_explanation").getField("narrative").alias("narrative"), + anomaly.getField("ai_explanation").getField("action").alias("action"), + ) + .agg(F.count("*").alias("transactions"), F.collect_list("merchant_category")[0].alias("example_category")) + .filter(F.col("narrative").isNotNull()) + .orderBy(F.desc("transactions")) +) + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ## Section 5: (Optional) Tune the Threshold +# MAGIC +# MAGIC **The threshold is an alert budget, not a confidence score.** `threshold=95` means "flag the rows +# MAGIC above the 95th percentile of *training* severity" — so on 1,500 rows it flags roughly 75 before a +# MAGIC single anomaly exists. A row at severity 97 is not "97% likely to be a problem"; it is in the top 3% +# MAGIC most unusual. This is the most commonly misread number in the feature. +# MAGIC +# MAGIC A batch that contains real problems therefore flags **more** than 5%, and that is correct rather +# MAGIC than a fault: roughly 5% of the ordinary rows, plus the anomalies on top. The alert count grows with +# MAGIC the size of the problem instead of being capped at a fixed share of the table. +# MAGIC +# MAGIC That also puts a hard ceiling on precision. Ask for the top 5% of 1,500 rows and you get 75 alerts; +# MAGIC if only 30 rows are genuinely bad, the best precision anyone could achieve is 30/75 = **40%**. The +# MAGIC table below prints that ceiling next to what the model actually achieved, which is the only fair way +# MAGIC to read the number. +# MAGIC +# MAGIC Severity is computed for **every** row, so you can count would-be anomalies at other thresholds +# MAGIC without rescoring. +# MAGIC + +# COMMAND ---------- +# DBTITLE 1,Threshold tradeoffs + + +def report_thresholds(scored_df, label: str, thresholds=(90.0, 95.0, 98.0), label_col: str = "is_anomaly"): + """Print alerts, catch rate and precision against its ceiling, at several thresholds. + + A helper rather than a loop because Section 6 reports a second model with it, and two models are only + comparable if both are measured the same way. + """ + severity_col = F.element_at(F.col("_dq_info"), 1).getField("anomaly").getField("severity_percentile") + is_planted = F.col(label_col) == 1.0 + planted = scored_df.filter(is_planted).count() + + print(f"🎚️ {label}\n") + print("Threshold | Alerts | Caught | Precision | Best possible | Recall") + print("-" * 68) + for threshold in thresholds: + alerts = scored_df.filter(severity_col >= threshold) + n_alerts = alerts.count() + n_caught = alerts.filter(is_planted).count() + # The ceiling: you cannot be more precise than "every alert is a real anomaly". + ceiling = min(1.0, planted / n_alerts) if n_alerts else 0.0 + precision = n_caught / n_alerts if n_alerts else 0.0 + print( + f" {threshold:>5.0f} | {n_alerts:>6d} | {n_caught:>4d}/{planted:<3d}|" + f" {precision:>6.1%} | {ceiling:>7.1%} | {n_caught / planted:>5.1%}" + ) + + +report_thresholds(scored, "Testing different thresholds:") + +print("\n💡 Read precision against the ceiling, not against 100%. Where the two are equal, every") +print(" planted anomaly is inside the model's ranking and no alert is wasted — the ranking is") +print(" optimal for that budget. A tighter threshold then trades recall for precision; it does") +print(" not reveal a better model.") +print("\n Where precision sits *below* the ceiling, something different is happening: ordinary rows are") +print(" outranking real anomalies. That is a statement about the ranking, not about the budget, and no") +print(" choice of threshold fixes it — the feature set or the comparison basis is what needs attention.") + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ## Section 6: (Optional) Time As A *Basis*, Not A Feature +# MAGIC +# MAGIC Everything so far compared each transaction against its merchant category's normal. Amounts in this +# MAGIC dataset do not trend, so no time axis was needed. Plenty of payments data does trend: a processor's +# MAGIC volume grows as merchants are onboarded, a subscription book compounds, a seasonal retailer ramps. +# MAGIC +# MAGIC When the normal level itself moves, `baseline_over_time` names a column as the **axis** each metric is +# MAGIC measured along. This is the part worth internalising: +# MAGIC +# MAGIC | | What DQX does with the column | What it detects | +# MAGIC |---|---|---| +# MAGIC | timestamp listed in `columns` | turns it into seven calendar features (hour, day of week, month, weekend) | 3am is odd and 3pm is not | +# MAGIC | timestamp passed as `baseline_over_time` | **never a feature.** It is the axis; DQX fits each metric's expected level along it | this value is wrong for *where the trend had got to* | +# MAGIC +# MAGIC The failure below is one every data team recognises: a feed partially breaks, volumes quietly revert to +# MAGIC an earlier level, and every number stays inside the year's range. No range check fires. Nothing is +# MAGIC extreme. It is only wrong for the point in time it arrived at. + +# COMMAND ---------- +# DBTITLE 1,A processor whose volume grows, and a week where a feed silently reverts + +# 18 months of daily counts, growing as merchants are onboarded. The fault holds one week at the level of +# roughly five months earlier -- inside the range the year covers, so nothing about it is out of bounds. +DAILY_START = datetime(2024, 1, 1) + + +def generate_daily_volume(n_days: int, seed: int, stalled_week: bool = False): + """Daily transaction count and settled value for one processor, trending upward.""" + rng = np.random.default_rng(seed) + days = np.arange(n_days) + count = 4_000 + 9.0 * days + 300.0 * np.sin(2 * np.pi * days / 7.0) + rng.normal(0, 120, n_days) + settled = count * (26.0 + 0.004 * days) + rng.normal(0, 4_000, n_days) + labels = np.zeros(n_days) + + if stalled_week: + start = int(n_days * 0.82) + count[start : start + 7] -= 9.0 * 150 + settled[start : start + 7] -= 9.0 * 150 * 26.0 + labels[start : start + 7] = 1.0 + + rows = [ + ( + DAILY_START + timedelta(days=int(day)), + float(count[i]), + float(settled[i]), + float(labels[i]), + ) + for i, day in enumerate(days) + ] + return spark.createDataFrame(rows, "settlement_date timestamp, txn_count double, settled_value double, is_anomaly double") + + +volume_history = f"{catalog}.{schema}.processor_daily_history" +volume_recent = f"{catalog}.{schema}.processor_daily_recent" +generate_daily_volume(540, seed=21).write.mode("overwrite").option("overwriteSchema", "true").saveAsTable(volume_history) +generate_daily_volume(180, seed=22, stalled_week=True).write.mode("overwrite").option( + "overwriteSchema", "true" +).saveAsTable(volume_recent) + +print("📊 18 months of daily history, and 6 months of recent days containing one stalled week") + +# COMMAND ---------- +# DBTITLE 1,Check the data actually trends before reaching for the parameter + +# The decision comes first and it is measurable. Subtracting a fitted expectation from a metric with no +# structure over time removes real signal and adds the fit's own error, so this is not a free switch. +from databricks.labs.dqx.anomaly.temporal_advisory import measure_trend_strength + +VOLUME_METRICS = ["txn_count", "settled_value"] +volume_trend = measure_trend_strength(spark.table(volume_history), "settlement_date", VOLUME_METRICS) +amount_trend = measure_trend_strength(spark.table(history_table), "transaction_time", ["amount", "item_count"]) + +print(f"📊 Daily processor volume: {volume_trend:.1%} of variance explained by trend and seasonality ✅ use it") +print(f"📊 Individual transactions: {amount_trend:.1%} ⚠️ nothing to remove, which is why Sections 1-5 do not") + +# COMMAND ---------- +# DBTITLE 1,Train with a time axis + +# settlement_date is named as the axis, so it is NOT a feature: no calendar columns are derived from it, and +# nothing about the hour or weekday enters the model. DQX fits each metric's expected level along it instead. +volume_model = f"{catalog}.{schema}.processor_volume_monitor" + +volume_trained = anomaly_engine.train( + df=spark.table(volume_history), + model_name=volume_model, + registry_table=registry_table, + columns=VOLUME_METRICS, + baseline_over_time="settlement_date", + baseline_by=[], + params=AnomalyParams(sample_fraction=1.0), +) + +print(f"\n🎯 Trained with an expected level per metric over time") + +# COMMAND ---------- +# DBTITLE 1,Score, and see what "wrong for now" looks like + +volume_scored = f"{catalog}.{schema}.processor_daily_scored" + +dq_engine.apply_checks_and_save_in_table( + input_config=InputConfig(location=volume_recent), + output_config=OutputConfig(location=volume_scored, mode="overwrite", options={"overwriteSchema": "true"}), + checks=[ + DQDatasetRule( + criticality="error", + check_func=has_no_row_anomalies, + check_func_kwargs={ + "model_name": volume_trained, + "registry_table": registry_table, + "threshold": 95.0, + "enable_ai_explanation": False, + }, + ) + ], +) + +volume_result = spark.table(volume_scored) +volume_anomaly = F.element_at(F.col("_dq_info"), 1).getField("anomaly") +stalled = volume_result.filter(F.col("is_anomaly") == 1.0).count() +caught = volume_result.filter(volume_anomaly.getField("is_anomaly") & (F.col("is_anomaly") == 1.0)).count() +print(f"🔍 Caught {caught} of the {stalled} days when the feed had quietly reverted\n") + +report_thresholds(volume_result, "Daily volume, judged against its own trend:", label_col="is_anomaly") + +# COMMAND ---------- +# DBTITLE 1,Read the contributions, which name the expected level + +print("💡 The contributions name the metric, not the comparison that objected to it. Every one of") +print(" these counts sits inside the range the history covers, so what is wrong is their position") +print(" against the trend rather than their value -- but the map says 'this metric mattered', and") +print(" it is baseline_over_time being set that tells you the comparison it mattered against.\n") + +display( + volume_result.filter(volume_anomaly.getField("is_anomaly")) + .select( + "settlement_date", + F.round("txn_count").alias("txn_count"), + volume_anomaly.getField("severity_percentile").alias("severity"), + volume_anomaly.getField("contributions").alias("contributions"), + volume_anomaly.getField("is_stale_baseline").alias("extrapolating"), + ) + .orderBy(F.desc("severity")) + .limit(8) +) + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ### The four ways DQX can treat a timestamp +# MAGIC +# MAGIC | Ask this | Use | The timestamp becomes | +# MAGIC |---|---|---| +# MAGIC | Is this row odd on its own, or in combination? | `profile` | nothing | +# MAGIC | Is it odd for its own group? | `baseline_by` | nothing | +# MAGIC | Is it odd for the time of day or day of week? | list it in `columns` | seven calendar features | +# MAGIC | Is it odd for its own point in time? | `baseline_over_time` | the axis, never a feature | +# MAGIC +# MAGIC They compose. Set `baseline_by` and `baseline_over_time` together and each metric is judged against +# MAGIC what its own group's history says to expect at that moment, still on one pooled model. +# MAGIC +# MAGIC ### When to leave it off +# MAGIC +# MAGIC - **A metric that does not trend**, like the individual transaction amounts in Sections 1 to 5. The +# MAGIC cell above measures both, and DQX warns when a training window shows too little structure over time. +# MAGIC - **A short window.** A weekly shape needs several complete weeks before it is identifiable at all; +# MAGIC DQX fits one only where the window supports it, and logs the period it skipped and why. +# MAGIC - **If you list a timestamp in `columns` by accident** — which is what auto-discovery does for you — +# MAGIC DQX warns at training time and names both escapes. Measured on this dataset, seven calendar features +# MAGIC derived from a meaningless timestamp cost a quarter of the detections at a fixed alert budget. +# MAGIC +# MAGIC It is also **not a forecaster.** It models the level expected *at* a time; it does not predict the next +# MAGIC value and never reads the previous row, which is what keeps scoring valid on a stream. +# MAGIC +# MAGIC `is_stale_baseline` marks rows past the window the expectation was fitted on. The score is still +# MAGIC produced, because near the boundary it remains accurate; treat the flag as a signal to retrain. + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ## Summary & Next Steps +# MAGIC +# MAGIC **Key takeaways:** +# MAGIC - Rules catch what you can name in advance. They caught **none** of these rows, and no threshold +# MAGIC would have, because every individual value was ordinary. +# MAGIC - Row anomaly detection finds implausible **combinations**, with no thresholds to choose. +# MAGIC - `baseline_by` makes "normal" contextual, so one model covers a coffee shop and an electronics store. +# MAGIC - Contributions and AI explanations tell you *which columns combined badly*, so a flagged row is +# MAGIC actionable rather than merely suspicious. +# MAGIC - The threshold is an **alert budget**, not a confidence score. Judge precision against the ceiling +# MAGIC that budget implies. +# MAGIC - A timestamp can be a **feature** or an **axis**, and the difference matters. Section 6 uses one as +# MAGIC an axis via `baseline_over_time` to catch a feed that silently reverted to an earlier level, with +# MAGIC every value still inside the year's range. +# MAGIC +# MAGIC **Apply to your data:** +# MAGIC ```python +# MAGIC model = anomaly_engine.train( +# MAGIC df=spark.table("your_catalog.your_schema.your_table"), +# MAGIC model_name="your_catalog.your_schema.your_model", +# MAGIC registry_table="your_catalog.your_schema.dqx_anomaly_models", +# MAGIC baseline_by=["your_grouping_column"], # judge each row against its own group +# MAGIC ) +# MAGIC +# MAGIC checks = [ +# MAGIC DQDatasetRule( +# MAGIC criticality="error", +# MAGIC check_func=has_no_row_anomalies, +# MAGIC check_func_kwargs={ +# MAGIC "model_name": model, +# MAGIC "registry_table": "your_catalog.your_schema.dqx_anomaly_models", +# MAGIC }, +# MAGIC ) +# MAGIC ] +# MAGIC +# MAGIC # Name the input table and the output table — DQX reads, scores and writes in one call. +# MAGIC dq_engine.apply_checks_and_save_in_table( +# MAGIC input_config=InputConfig(location="your_catalog.your_schema.new_data"), +# MAGIC output_config=OutputConfig(location="your_catalog.your_schema.scored"), +# MAGIC checks=checks, +# MAGIC ) +# MAGIC ``` +# MAGIC +# MAGIC **Optional next steps:** +# MAGIC - Add `drift_threshold=3.0` to be warned when the input distribution moves away from training. +# MAGIC - Quarantine flagged rows with `apply_checks_and_split` instead of tagging them in place. +# MAGIC - Schedule retraining as "normal" changes — new products, new pricing, new processes. +# MAGIC + +# COMMAND ---------- + +# MAGIC %md +# MAGIC --- +# MAGIC +# MAGIC ### 📚 Resources +# MAGIC +# MAGIC - [Row Anomaly Detection guide](https://databrickslabs.github.io/dqx/docs/guide/row_anomaly_detection) +# MAGIC - [`has_no_row_anomalies` reference](https://databrickslabs.github.io/dqx/docs/reference/quality_checks#row-anomaly-detection) +# MAGIC - [Choosing a profile](https://databrickslabs.github.io/dqx/docs/guide/row_anomaly_detection#choosing-a-profile) +# MAGIC +# MAGIC ### 🎉 You're Ready! +# MAGIC +# MAGIC You now understand: +# MAGIC - ✅ Why rule-based checks cannot catch implausible combinations +# MAGIC - ✅ How to train an anomaly model with no thresholds and no labels +# MAGIC - ✅ How `baseline_by` makes "normal" depend on context +# MAGIC - ✅ How to read contributions and AI explanations to triage a flagged row +# MAGIC +# MAGIC **Start finding the rows your rules miss!** 🚀 +# MAGIC diff --git a/demos/dqx_row_anomaly_detection_demo.py b/demos/dqx_row_anomaly_detection_demo.py deleted file mode 100644 index 6576ca56f..000000000 --- a/demos/dqx_row_anomaly_detection_demo.py +++ /dev/null @@ -1,854 +0,0 @@ -# Databricks notebook source -# MAGIC %md -# MAGIC # 📊 Row Anomaly Detection Demo -# MAGIC -# MAGIC ## Learn Row Anomaly Detection in 15 Minutes -# MAGIC -# MAGIC **Quickstart (5–10 minutes):** -# MAGIC - Train an anomaly model on sample data using DQX Row Anomaly Detection Engine -# MAGIC - Apply checks and see flagged anomalies -# MAGIC - View severity percentiles and top contributors -# MAGIC -# MAGIC **Dataset**: Simple sales transactions (universally relatable, no domain expertise required) -# MAGIC - -# COMMAND ---------- - -# MAGIC %md -# MAGIC --- -# MAGIC -# MAGIC ## What is Row Anomaly Detection? -# MAGIC -# MAGIC - Standard rule-based checks catch *known* issues (nulls, ranges, formats). -# MAGIC - Row anomaly detection finds *unknown* patterns in rows across multiple columns. -# MAGIC - Use both together for better coverage. -# MAGIC -# MAGIC **Why row anomaly detection** -# MAGIC - Learns "normal" from data -# MAGIC - Flags deviations without manual rules and thresholds -# MAGIC - Complements rule-based checks rather than replacing them -# MAGIC -# MAGIC **Known vs Unknown Issues** -# MAGIC - **Known unknowns**: rule‑based checks (nulls, ranges, formats). -# MAGIC - **Unknown unknowns**: multi‑column or subtle patterns you didn’t anticipate. -# MAGIC -# MAGIC **Data Quality Monitoring (DQM) vs DQX Row Anomaly detection** -# MAGIC - **[Data Quality Monitoring (DQM)](https://docs.databricks.com/aws/en/data-quality-monitoring/anomaly-detection)**: uses table‑level signals such as row counts and commit patterns. -# MAGIC - **DQX Anomaly**: look for row‑level patterns within the data (per‑record anomalies with explanations). -# MAGIC - DQM and DQX each provide distinct capabilities. Together, they complement one another to deliver comprehensive coverage across the full spectrum of data quality checks. -# MAGIC - -# COMMAND ---------- - -# MAGIC %md -# MAGIC --- -# MAGIC -# MAGIC ## Prerequisites: Install DQX with Anomaly Support -# MAGIC -# MAGIC ```python -# MAGIC %pip install 'databricks-labs-dqx[anomaly]' -# MAGIC dbutils.library.restartPython() -# MAGIC ``` -# MAGIC -# MAGIC **What's included in `[anomaly]` extras:** -# MAGIC - `scikit-learn` - Machine learning algorithms used for row anomaly detection -# MAGIC - `mlflow` - Model tracking and registry -# MAGIC - `shap` - Feature contributions for explainability -# MAGIC - `cloudpickle` - Model serialization -# MAGIC -# MAGIC **Note**: If you are using ML Runtime or Serverless compute, most dependencies are already pre-installed. -# MAGIC - -# COMMAND ---------- -# DBTITLE 1,Prerequisites: Install DQX with Anomaly Support - -dbutils.widgets.text("test_library_ref", "", "Test Library Ref") - -if dbutils.widgets.get("test_library_ref") != "": - %pip install 'databricks-labs-dqx[anomaly] @ {dbutils.widgets.get("test_library_ref")}' -else: - %pip install databricks-labs-dqx[anomaly] - -%restart_python - -# COMMAND ---------- -# DBTITLE 1,Prerequisites: Configure test catalog and schema - -default_catalog = "main" -default_schema = "default" - -# Configure widgets for catalog and schema -dbutils.widgets.text("demo_catalog", default_catalog, "Catalog Name") -dbutils.widgets.text("demo_schema", default_schema, "Schema Name") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ## Section 1: Setup & Data Generation -# MAGIC - -# COMMAND ---------- -# DBTITLE 1,Setup engines - -import pyspark.sql.functions as F -from pyspark.sql.types import * -from datetime import datetime, timedelta -import random -import numpy as np - -from databricks.labs.dqx.anomaly.anomaly_engine import AnomalyEngine -from databricks.labs.dqx.anomaly.check_funcs import has_no_row_anomalies -from databricks.labs.dqx.engine import DQEngine -from databricks.labs.dqx.rule import DQDatasetRule, DQRowRule -from databricks.labs.dqx.check_funcs import is_not_null, is_in_range -from databricks.sdk import WorkspaceClient - -# Initialize DQX engines -ws = WorkspaceClient() -anomaly_engine = AnomalyEngine(ws) -dq_engine = DQEngine(ws) - -# Set seeds for reproducibility for demo purposes -random.seed(42) -np.random.seed(42) - -print("✅ Setup complete!") - -# COMMAND ---------- -# DBTITLE 1,Data Generation - -# Generate historical (training) data -def generate_historical_sales_data( - num_rows: int = 1000, -): - """ - Generate historical sales data (no synthetic anomalies). - """ - data = [] - categories = ["Electronics", "Clothing", "Food", "Books", "Home"] - regions = ["North", "South", "East", "West"] - - # Regional pricing patterns (normal baseline) - region_patterns = { - "North": {"base_amount": 200, "quantity": 5}, - "South": {"base_amount": 150, "quantity": 4}, - "East": {"base_amount": 180, "quantity": 4}, - "West": {"base_amount": 220, "quantity": 6}, - } - - start_date = datetime(2024, 1, 1, 9, 0) # Jan 1, 2024, 9am - - for i in range(num_rows): - transaction_id = f"TXN{i:06d}" - category = random.choice(categories) - region = random.choice(regions) - pattern = region_patterns[region] - - # Generate timestamp (mostly business hours weekdays) - days_offset = random.randint(0, 90) # 3 months of data - hours_offset = random.randint(0, 9) # 9am-6pm = 9 hours - date = start_date + timedelta(days=days_offset, hours=hours_offset) - - # Skip weekends for normal transactions - if date.weekday() >= 5: # Saturday=5, Sunday=6 - date = date - timedelta(days=date.weekday() - 4) # Move to Friday - - # Normal transaction (tighter variance for more consistent patterns) - amount = round(pattern["base_amount"] * random.uniform(0.85, 1.15), 2) - quantity = max(1, int(np.random.normal(pattern["quantity"], 1))) - - # Ensure valid ranges (skip for injected nulls/negatives) - if amount is not None: - amount = max(10, min(10000, amount)) - if quantity is not None: - quantity = max(1, min(150, quantity)) # Allow bulk orders up to 150 - - data.append((transaction_id, date, amount, quantity, category, region)) - - return data - -# Generate historical data -print("🔄 Generating historical (training) data...\n") -train_rows = 5000 -historical_data = generate_historical_sales_data(num_rows=train_rows) - -schema = StructType([ - StructField("transaction_id", StringType(), False), - StructField("date", TimestampType(), False), - StructField("amount", DoubleType(), True), - StructField("quantity", IntegerType(), True), - StructField("category", StringType(), False), - StructField("region", StringType(), False), -]) - -df_train = spark.createDataFrame(historical_data, schema) - -print("📊 Sample of sales transactions:") -display(df_train.orderBy("date")) - -total_train = df_train.count() -print(f"\n✅ Generated {total_train} historical transactions (for training)") - -# COMMAND ---------- -# DBTITLE 1,Save Test Data - -# Get catalog and schema from widgets -catalog = dbutils.widgets.get("demo_catalog") -schema_name = dbutils.widgets.get("demo_schema") - -print(f"📂 Using catalog: {catalog}") -print(f"📂 Using schema: {schema_name}\n") - -train_table = f"{catalog}.{schema_name}.sales_transactions_train" -df_train.write.mode("overwrite").saveAsTable(train_table) - -print(f"✅ Training data saved to: {train_table}") - -# COMMAND ---------- - -# Set up registry table for tracking trained models (always use fully qualified table name) -registry_table = f"{catalog}.{schema_name}.anomaly_model_registry_101" -print(f"📋 Model registry table: {registry_table}") - -# Clean up any existing registry from previous runs -spark.sql(f"DROP TABLE IF EXISTS {registry_table}") -print(f"✅ Registry ready for new models") - - -# COMMAND ---------- - -# MAGIC %md -# MAGIC --- -# MAGIC -# MAGIC ## Section 2: Train the Anomaly Model -# MAGIC -# MAGIC We’ll run: -# MAGIC - Simple rule checks (nulls, ranges) -# MAGIC - Row anomaly detection for unusual multi‑column patterns -# MAGIC -# MAGIC In DQX you can run all types of rules in the same run. - -# COMMAND ---------- -# DBTITLE 1,Train the Anomaly Model - -# Train row anomaly detection model with zero configuration -print("🎯 Training row anomaly detection model...") -print(" DQX will automatically discover patterns in your data\n") - -model_name_auto = f"{catalog}.{schema_name}.sales_auto" # stored in Unity Catalog and must be fully qualified name -model_uri_auto = anomaly_engine.train( - df=spark.table(train_table), - model_name=model_name_auto, - registry_table=registry_table # must be fully qualified table name: catalog.schema.table_name -) - -print(f"✅ Model trained successfully!") -print(f" Model URI: {model_uri_auto}") - -# View what DQX created for you -print(f"\n📋 Trained Models:\n") - -display( - spark.table(registry_table) - .filter(F.col("identity.model_name").contains(model_name_auto)) - .select( - "identity.model_name", - "training.columns", - "segmentation.segment_by", - "segmentation.segment_values", - "training.training_rows", - "training.training_time", - "identity.status" - ) - .orderBy("identity.model_name") -) - -print("\n💡 DQX auto-discovered patterns and registered a model for scoring.") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ### Optional: View Models in the UI -# MAGIC -# MAGIC Your models are stored in Unity Catalog and registered within MLflow. -# MAGIC If you want to inspect them, open **Catalog Explorer** or **Experiments**. -# MAGIC - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ### Section 3: Generate new data containing some anomalies -# MAGIC - -# COMMAND ---------- -# DBTITLE 1,Generate new data containing Anomalies - -def inject_anomalies_and_dq_issues( - base_rows: list[tuple], - anomaly_rate: float = 0.02, - dq_null_amount_rate: float = 0.01, - dq_null_quantity_rate: float = 0.005, - dq_negative_amount_rate: float = 0.005, -): - """ - Take clean (normal) rows and inject anomalies + simple DQ issues. - """ - rows = [] - for idx, row in enumerate(base_rows): - transaction_id, date, amount, quantity, category, region = row - transaction_id = f"NEW{idx:06d}" - is_synthetic_anomaly = False - - dq_roll = random.random() - if dq_roll < dq_null_amount_rate: - amount = None - elif dq_roll < dq_null_amount_rate + dq_null_quantity_rate: - quantity = None - elif dq_roll < dq_null_amount_rate + dq_null_quantity_rate + dq_negative_amount_rate: - amount = -abs(amount) - elif random.random() < anomaly_rate: - is_synthetic_anomaly = True - anomaly_type = random.choices( - ["extreme_scale", "mismatch_pair", "timing_spike"], - weights=[3, 3, 2], - )[0] - - if anomaly_type == "extreme_scale": - amount = round(amount * random.uniform(15, 25), 2) - quantity = int(quantity * random.uniform(15, 25)) - elif anomaly_type == "mismatch_pair": - # Large amount with tiny quantity (or vice versa) - if random.random() < 0.5: - amount = round(amount * random.uniform(12, 20), 2) - quantity = max(1, int(quantity * random.uniform(0.05, 0.2))) - else: - amount = round(amount * random.uniform(0.05, 0.2), 2) - quantity = int(quantity * random.uniform(12, 20)) - else: - # Off-hours + large spike - amount = round(amount * random.uniform(10, 18), 2) - quantity = int(quantity * random.uniform(10, 18)) - date = date.replace(hour=random.choice([2, 3, 4, 22, 23])) - - if amount is not None: - amount = max(10, min(10000, amount)) - if quantity is not None: - quantity = max(1, min(150, quantity)) - - rows.append((transaction_id, date, amount, quantity, category, region, is_synthetic_anomaly)) - return rows - -print("🔄 Generating new data with injected anomalies...\n") - -new_rows = 1000 -anomaly_rate = 0.02 -dq_null_amount_rate = 0.01 -dq_null_quantity_rate = 0.005 -dq_negative_amount_rate = 0.005 -dq_issue_rate = dq_null_amount_rate + dq_null_quantity_rate + dq_negative_amount_rate - -new_data_base = generate_historical_sales_data(num_rows=new_rows) -new_data = inject_anomalies_and_dq_issues( - base_rows=new_data_base, - anomaly_rate=anomaly_rate, - dq_null_amount_rate=dq_null_amount_rate, - dq_null_quantity_rate=dq_null_quantity_rate, - dq_negative_amount_rate=dq_negative_amount_rate, -) - -new_schema = StructType([ - StructField("transaction_id", StringType(), False), - StructField("date", TimestampType(), False), - StructField("amount", DoubleType(), True), - StructField("quantity", IntegerType(), True), - StructField("category", StringType(), False), - StructField("region", StringType(), False), - StructField("is_synthetic_anomaly", BooleanType(), False), -]) - -df_new = spark.createDataFrame(new_data, new_schema) - -print("📊 Sample of new data:") -display(df_new.orderBy("date")) - -total_new = df_new.count() -print(f"\n✅ Generated {total_new} NEW transactions") -print(f" Injected anomalies: ~{int(total_new * anomaly_rate)} ({anomaly_rate*100:.0f}%)") -print(f" Injected rule issues: ~{int(total_new * dq_issue_rate)} ({dq_issue_rate*100:.1f}%)") - -new_table = f"{catalog}.{schema_name}.sales_transactions_new" -df_new.write.mode("overwrite").saveAsTable(new_table) -print(f"✅ New data saved to: {new_table}") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC ### Section 4: Apply checks including row anomaly detection -# MAGIC -# MAGIC Now apply row anomaly detection + rule-based checks to the **new data**. - -# COMMAND ---------- -# DBTITLE 1,Apply quality checks - -print("🔍 Applying quality checks to new data...\n") - -# Define all quality checks, use default criticality="error" -checks_combined = [ - # Rule-based checks for known issues and thresholds - DQRowRule(check_func=is_not_null, check_func_kwargs={"column": "transaction_id"}), - DQRowRule(check_func=is_not_null, check_func_kwargs={"column": "amount"}), - DQRowRule(check_func=is_in_range, check_func_kwargs={"column": "amount", "min_limit": 0, "max_limit": 100000}), - DQRowRule(check_func=is_not_null, check_func_kwargs={"column": "quantity"}), - DQRowRule(check_func=is_in_range, check_func_kwargs={"column": "quantity", "min_limit": 1, "max_limit": 1000}), - - # Row anomaly detection for unusual patterns - DQDatasetRule( - check_func=has_no_row_anomalies, - check_func_kwargs={ - "model_name": model_name_auto, - "registry_table": registry_table - } - ) -] - -df_valid, df_quarantine = dq_engine.apply_checks_and_split(df_new, checks_combined) - -display(df_quarantine) - -print("\n💡 Summary:") -print(" • We trained on historical data and applied checks on new data.") -print(" • Default threshold 95 flags the top 5% most unusual records.") -print(" • Threshold is a percentile cutoff — tune it based on your data and alert tolerance.") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC --- -# MAGIC -# MAGIC ## Section 5a: (Optional) Review Results to understand why some records are anomalous -# MAGIC -# MAGIC You’ll see flagged anomalies, severity percentiles, and top contributors. -# MAGIC -# MAGIC This section is optional. Skip if you only want the quickstart. -# MAGIC -# MAGIC In the quarantine dataset we can find the regular `_error` and `_warnings` reporting columns, and `_dq_info` column (array of structs). The first check's info is at `_dq_info[0]`; `_dq_info[0].anomaly` includes: -# MAGIC - `severity_percentile` (0–100): percentile of anomaly severity -# MAGIC - `score`: raw model score (diagnostic only) -# MAGIC - `contributions`: feature-level explanations - -# COMMAND ---------- -# DBTITLE 1,Review Results - -df_quarantine = df_quarantine.filter( - F.col("_dq_info").getItem(0).getField("anomaly").getField("is_anomaly") == True -) -score_col = F.col("_dq_info").getItem(0).getField("anomaly").getField("score") -severity_col = F.col("_dq_info").getItem(0).getField("anomaly").getField("severity_percentile") -percentile_band = ( - F.when(severity_col >= 98, F.lit("p98+ (top 2%)")) - .when(severity_col >= 95, F.lit("p95-98 (top 5%)")) - .when(severity_col >= 90, F.lit("p90-95 (top 10%)")) - .otherwise(F.lit(" 0: - recall = synthetic_caught / synthetic_total * 100 - print(f"\n✅ Synthetic anomalies injected: {synthetic_total}") - print(f" Synthetic anomalies caught: {synthetic_caught} ({recall:.1f}% recall)") -print(f"\n🔝 Top 10 anomalies:\n") - -display(df_quarantine.orderBy(severity_col.desc()).select( - "transaction_id", "date", "amount", "quantity", "category", "region", - F.round(severity_col, 1).alias("severity_percentile"), - F.round(score_col, 3).alias("anomaly_score"), - percentile_band.alias("severity_band"), -).limit(10)) - -# COMMAND ---------- - -# MAGIC %md -# MAGIC --- -# MAGIC ## Section 5b: AI explanations -# MAGIC -# MAGIC AI explanations are **on by default** — the checks in Section 4 already produced -# MAGIC `_dq_info[0].anomaly.ai_explanation` (a plain-language `narrative`, `business_impact`, -# MAGIC `action`, and the deterministic `top_features` pattern). The LLM call runs **inside Spark** -# MAGIC via the SQL `ai_query` function against a Databricks Model Serving endpoint — no extra -# MAGIC dependency, and rows are grouped so the model is called **once per group** (capped by -# MAGIC `max_groups`). This requires **Databricks serverless compute or Databricks Runtime 15.4 LTS -# MAGIC or above** (where `ai_query` is available). If `ai_query` is unavailable or no endpoint is -# MAGIC reachable, explanations are skipped with a warning and scoring still completes. -# MAGIC -# MAGIC This cell just shows how to override the endpoint or turn explanations off — none of these -# MAGIC kwargs are required. - -# COMMAND ---------- -# DBTITLE 1,Apply checks (AI explanations are on by default) - -checks_with_ai = [ - DQDatasetRule( - check_func=has_no_row_anomalies, - check_func_kwargs={ - "model_name": model_name_auto, - "registry_table": registry_table, - # All optional — explanations + contributions are on by default: - # "enable_ai_explanation": False, # turn explanations off - # "ai_explanation_llm_model_config": {"model_name": "databricks-claude-sonnet-4-5"}, # override endpoint - # "redact_columns": ["region"], # keep sensitive names out of the prompt - # "max_groups": 500, # cap on LLM calls per run - }, - ) -] - -df_ai = dq_engine.apply_checks(df_new, checks_with_ai) - -anomaly = F.col("_dq_info").getItem(0).getField("anomaly") -explanation = anomaly.getField("ai_explanation") -print("🤖 Top anomalies with AI explanations:\n") -display( - df_ai.filter(anomaly.getField("is_anomaly") == True) - .orderBy(anomaly.getField("severity_percentile").desc()) - .select( - "transaction_id", - "amount", - "quantity", - F.round(anomaly.getField("severity_percentile"), 1).alias("severity_percentile"), - explanation.getField("top_features").alias("top_features"), - explanation.getField("narrative").alias("why_flagged"), - explanation.getField("business_impact").alias("business_impact"), - explanation.getField("action").alias("suggested_action"), - ) - .limit(10) -) - -# COMMAND ---------- - -# MAGIC %md -# MAGIC --- -# MAGIC ## Section 6: (Optional) Threshold Tradeoffs -# MAGIC -# MAGIC This section is optional. Skip if you only want the quickstart. - -# COMMAND ---------- -# DBTITLE 1,Threshold Tradeoffs - -print("📌 Summary:") -print(" • Default threshold = 95 (top 5%).") -print(" • Raise it to reduce alerts; lower it to catch more.") -print(" • The right setting depends on your data distribution and risk tolerance.") - -# (Optional) Quick normal vs anomaly sanity check -print("🔍 Sanity check (severity < 95 vs ≥ 95):\n") -normal_count = df_scored.filter(severity_col < 95).count() -anomaly_count = df_scored.filter(severity_col >= 95).count() -print(f" Normal: {normal_count} ({normal_count/total_scored*100:.1f}%)") -print(f" Anomaly: {anomaly_count} ({anomaly_count/total_scored*100:.1f}%)") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC --- -# MAGIC -# MAGIC ### Tuning the Threshold -# MAGIC -# MAGIC Threshold is a percentile cutoff: -# MAGIC - Lower (e.g., 90) = more alerts -# MAGIC - Higher (e.g., 98) = fewer alerts -# MAGIC -# MAGIC We already scored all records, so you can change thresholds without re‑scoring. -# MAGIC - -# COMMAND ---------- -# DBTITLE 1,Tuning the Threshold - -# Try different thresholds -print("🎚️ Testing Different Thresholds:\n") -print("Threshold | Anomalies | % of Data | Interpretation") -print("-" * 70) - -thresholds = [90, 95, 98] -total_count = total_scored - -for threshold in thresholds: - anomaly_count = df_scored.filter(severity_col >= threshold).count() - percentage = (anomaly_count / total_count) * 100 - - if threshold < 95: - interpretation = "Sensitive (more alerts)" - elif threshold == 95: - interpretation = "Balanced (default)" - else: - interpretation = "Strict (fewer alerts)" - - print(f" {threshold:>3d} | {anomaly_count:4d} | {percentage:5.1f}% | {interpretation}") - -print("\n💡 Start at 95, then explore thresholds on your data to balance noise vs. missed anomalies.") - -# COMMAND ---------- -# DBTITLE 1,Tuning the Threshold - -# Borderline slice (optional) -borderline = df_scored.filter((severity_col >= 90) & (severity_col < 95)).orderBy(severity_col.desc()) -print(f"\nBorderline (90-<95) examples: {borderline.count()}") -display(borderline.select( - "transaction_id", "amount", "quantity", - F.round(severity_col, 1).alias("severity_percentile"), - F.round(score_col, 3).alias("score"), -).limit(5)) - -# COMMAND ---------- - -# MAGIC %md -# MAGIC --- -# MAGIC -# MAGIC ## Section 7: (Optional) Manual Column Selection -# MAGIC -# MAGIC Skip this if you only want the quickstart. -# MAGIC -# MAGIC We will train a model with specific columns. While applying the row anomaly detection check, only the columns the model was trained on will be used. -# MAGIC -# MAGIC This is important in production when you need strict feature control. By default, all supported columns are used. - -# COMMAND ---------- -# DBTITLE 1,Training with Manual Column Selection - -print("🎯 Training model with manual column selection...\n") -model_name_manual = f"{catalog}.{schema_name}.sales_manual" # stored in Unity Catalog and must be fully qualified name -model_uri_manual = anomaly_engine.train( - df=spark.table(train_table), - columns=["amount", "quantity"], # Explicitly specify numeric columns only - model_name=model_name_manual, - registry_table=registry_table -) - -print(f"✅ Manual model trained!") -print(f" Model URI: {model_uri_manual}") -print(f"\n💡 Manual selection is useful in production when you want strict feature control.") - -# COMMAND ---------- -# DBTITLE 1,Manual Column Selection - -# Score with manual model -print("🔍 Scoring with manual model...\n") - -checks_manual = [ - DQDatasetRule( - check_func=has_no_row_anomalies, - check_func_kwargs={ - "model_name": model_name_manual, - "threshold": 95.0, - "registry_table": registry_table - # we don't specify which column to apply the anomaly check on; the same columns that were selected for the training are used - } - ) -] - -df_valid, df_quarantine_manual = dq_engine.apply_checks_and_split(df_new, checks_manual) - -print(f"⚠️ Manual model found {df_quarantine_manual.count()} anomalies") -print(f" (Auto model found {df_quarantine.count()} anomalies)") -print(f"\n🔝 Top 5 anomalies from manual model:\n") - - -_dq_info_severity = F.col("_dq_info").getItem(0).getField("anomaly").getField("severity_percentile") -_dq_info_score = F.col("_dq_info").getItem(0).getField("anomaly").getField("score") -display(df_quarantine_manual.orderBy(_dq_info_severity.desc()).select( - "transaction_id", "amount", "quantity", "date", - F.round(_dq_info_severity, 1).alias("severity_percentile"), - F.round(_dq_info_score, 3).alias("score") -).limit(5)) - -print("\n💡 Different features → different anomalies. That’s expected.") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC --- -# MAGIC -# MAGIC ## Section 8: (Optional) Feature Contributions -# MAGIC -# MAGIC Skip this if you only want the quickstart. -# MAGIC -# MAGIC ### Advanced Options (Reference) -# MAGIC -# MAGIC **Scoring options (`has_no_row_anomalies`):** -# MAGIC - `threshold` (float, 0–100): percentile cutoff (default 95) -# MAGIC - `enable_contributions` (bool): feature contributions in `_dq_info[0].anomaly` -# MAGIC - `enable_confidence_std` (bool): confidence estimate (std dev across ensemble) -# MAGIC - `drift_threshold` (float): drift detection sensitivity -# MAGIC - `row_filter` (str): SQL filter applied before scoring -# MAGIC -# MAGIC **Training options (`AnomalyEngine.train` / `AnomalyParams`):** -# MAGIC - `columns` (list[str]): explicit feature list (disables auto‑discovery) -# MAGIC - `segment_by` (list[str]): explicit segmentation columns -# MAGIC - `sample_fraction`, `max_rows`: training sample controls -# MAGIC - `ensemble_size`: number of models in the ensemble -# MAGIC - `expected_anomaly_rate`: expected anomaly rate for calibration -# MAGIC -# MAGIC These are optional — the demo uses defaults for simplicity. -# MAGIC - -# COMMAND ---------- -# DBTITLE 1,Advanced Options (Reference) - -# Score with feature contributions -print("🔍 Scoring with feature contributions (explainability)...\n") - -checks_with_contrib = [ - DQDatasetRule( - check_func=has_no_row_anomalies, - check_func_kwargs={ - "model_name": model_name_manual, - "threshold": 95.0, - "enable_contributions": True, # on by default; shown here for clarity - "registry_table": registry_table - } - ) -] - -df_with_contrib = dq_engine.apply_checks(df_new, checks_with_contrib) - -print("✅ Scored with feature contributions!") -print("\n🎯 Top Anomalies with Explanations:\n") - -# Filter by _errors column (standard DQX pattern) to get flagged anomalies -anomalies_explained = df_with_contrib.filter( - F.size(F.col("_errors")) > 0 -).orderBy(F.col("_dq_info").getItem(0).getField("anomaly").getField("severity_percentile").desc()).limit(5) - -_dq_severity = F.col("_dq_info").getItem(0).getField("anomaly").getField("severity_percentile") -_dq_score = F.col("_dq_info").getItem(0).getField("anomaly").getField("score") -_dq_contrib = F.col("_dq_info").getItem(0).getField("anomaly").getField("contributions") -display(anomalies_explained.select( - "transaction_id", - "amount", - "quantity", - F.date_format("date", "yyyy-MM-dd HH:mm").alias("date"), - F.round(_dq_severity, 1).alias("severity_percentile"), - F.round(_dq_score, 3).alias("score"), - _dq_contrib.alias("contributions").alias("why_anomalous") -)) - -print("\n💡 Contributions show which features most influenced the anomaly.") -print(" Focus on features with the highest % contribution.") - -# COMMAND ---------- -# DBTITLE 1,Advanced Options (Reference) - -# Show one detailed example -print("🔎 Detailed Example - Top Anomaly:\n") - -# Extract the columns for easier access -anomalies_flattened = anomalies_explained.select( - "transaction_id", - "amount", - "quantity", - "date", - F.col("_dq_info").getItem(0).getField("anomaly").getField("severity_percentile").alias("severity_percentile"), - F.col("_dq_info").getItem(0).getField("anomaly").getField("score").alias("score"), - F.col("_dq_info").getItem(0).getField("anomaly").getField("contributions").alias("contributions"), -) - -top_anomaly = anomalies_flattened.first() - -print(f"Transaction ID: {top_anomaly['transaction_id']}") -print(f"Severity Percentile: {top_anomaly['severity_percentile']:.1f}") -print(f"Anomaly Score (raw): {top_anomaly['score']:.3f}") -print(f"\nTransaction Details:") -print(f" Amount: ${top_anomaly['amount']:.2f}") -print(f" Quantity: {top_anomaly['quantity']}") -print(f" Date: {top_anomaly['date']}") -print(f"\nFeature Contributions:") - -contributions = top_anomaly['contributions'] -if contributions: - # Sort by contribution value - sorted_contrib = sorted(contributions.items(), key=lambda x: abs(x[1]), reverse=True) - for feature, value in sorted_contrib[:3]: # Top 3 - print(f" {feature}: {abs(value):.1f}% contribution") - - print(f"\n🎯 Investigation Tip:") - top_feature = sorted_contrib[0][0] - if "amount" in top_feature: - print(f" → Check for pricing errors or incorrect price feeds") - elif "quantity" in top_feature: - print(f" → Investigate bulk order or inventory issue") - elif "date" in top_feature or "hour" in top_feature: - print(f" → Review transaction timing - off-hours activity?") -else: - print(" (No detailed contributions available)") - -# COMMAND ---------- - -# MAGIC %md -# MAGIC --- -# MAGIC -# MAGIC ## Summary & Next Steps -# MAGIC -# MAGIC **Key takeaways:** -# MAGIC - You can apply row anomaly detection and rule-based checks together. -# MAGIC - Start with threshold 95 (default), tune as needed. -# MAGIC - Use contributions to triage anomalies faster. -# MAGIC -# MAGIC **Apply to your data:** -# MAGIC ```python -# MAGIC # Replace with your table -# MAGIC model = anomaly_engine.train( -# MAGIC df=spark.table("your_catalog.your_schema.your_table"), -# MAGIC model_name="your_catalog.your_schema.your_model_name", -# MAGIC registry_table="your_catalog.your_schema.dqx_anomaly_models", -# MAGIC ) -# MAGIC -# MAGIC checks = [ -# MAGIC has_no_row_anomalies( -# MAGIC model_name="your_catalog.your_schema.your_model_name", -# MAGIC registry_table="your_catalog.your_schema.dqx_anomaly_models", -# MAGIC ) -# MAGIC ] -# MAGIC df_scored = dq_engine.apply_checks(your_df, checks) -# MAGIC ``` -# MAGIC -# MAGIC **Optional next steps:** -# MAGIC - Add segmentation (`segment_by` option for training), drift detection, and scheduled scoring. -# MAGIC - Automate retraining and alerting. - -# COMMAND ---------- - -# MAGIC %md -# MAGIC --- -# MAGIC -# MAGIC ### 📚 Resources -# MAGIC -# MAGIC - [DQX Row Anomaly Detection Documentation](https://databrickslabs.github.io/dqx/guide/row_anomaly_detection) -# MAGIC - [API Reference](https://databrickslabs.github.io/dqx/reference/quality_checks#has_no_row_anomalies) -# MAGIC - [Data Quality Monitoring (DQM)](https://docs.databricks.com/aws/en/data-quality-monitoring/anomaly-detection/#-table-quality-details) -# MAGIC -# MAGIC ### 🎉 You're Ready! -# MAGIC -# MAGIC You now understand: -# MAGIC - ✅ What row anomaly detection is and when to use it -# MAGIC - ✅ How to implement it with minimal configuration -# MAGIC - ✅ How to interpret and tune results -# MAGIC - ✅ How to integrate it into production -# MAGIC -# MAGIC **Start detecting anomalies in your data today!** 🚀 -# MAGIC diff --git a/docs/dqx/docs/demos.mdx b/docs/dqx/docs/demos.mdx index 2071401cf..9c358f095 100644 --- a/docs/dqx/docs/demos.mdx +++ b/docs/dqx/docs/demos.mdx @@ -49,7 +49,8 @@ Import the following notebooks in the Databricks workspace to try DQX out: * [DQX Demo Notebook for Data Quality Summary Metrics](https://github.com/databrickslabs/dqx/blob/v0.16.0/demos/dqx_demo_summary_metrics.py) - demonstrates how to generate summary-level data quality metrics when validating data with DQX. * [DQX Demo Notebook for Actions and Alerting](https://github.com/databrickslabs/dqx/blob/v0.16.0/demos/dqx_demo_alerting.py) - demonstrates how to react to data quality problems by firing alerts (driver log, with optional Slack) when summary metrics cross a threshold. * [DQX Demo Notebook for Profiling and Applying Checks at Scale on Multiple Tables](https://github.com/databrickslabs/dqx/blob/v0.16.0/demos/dqx_multi_table_demo.py) - demonstrates how to use DQX as a library at scale to apply checks on multiple tables. -* [DQX Demo Notebook for Row Anomaly Detection](https://github.com/databrickslabs/dqx/blob/v0.16.0/demos/dqx_row_anomaly_detection_demo.py) - comprehensive demo showing how to use DQX Row Anomaly Detection to detect unusual patterns in your data. +* [DQX Demo Notebook for Row Anomaly Detection on Transactions](https://github.com/databrickslabs/dqx/blob/v0.16.0/demos/dqx_demo_anomaly_tabular_transactions.py) - starts from a domain problem: card transactions that pass every rule but are jointly implausible. Shows why no threshold catches them, how `baseline_by` makes "normal" depend on the merchant category, and what the contributions tell you. Uses the default `tabular` profile. +* [DQX Demo Notebook for Row Anomaly Detection on Machine Telemetry](https://github.com/databrickslabs/dqx/blob/v0.16.0/demos/dqx_demo_anomaly_correlation_fleet.py) - starts from a domain problem: machine metrics that each stay inside their safe band while the *relationship* between them breaks. Verifies that no per-metric range check could catch it, then trains with `profile="correlation"` and reads the explanation. * [DQX Demo Notebook for AI-assisted checks generation](https://github.com/databrickslabs/dqx/blob/v0.16.0/demos/dqx_demo_ai_assisted_checks_generation.py) - demonstrates how to generate DQX rules/checks with LLM using natural language. * [DQX Demo Notebook for Data Contract Integration (ODCS)](https://github.com/databrickslabs/dqx/blob/v0.16.0/demos/dqx_demo_datacontract_odcs.py) - demonstrates how to generate DQX quality rules from ODCS (Open Data Contract Standard) data contracts, including predefined rules from schema constraints, explicit custom rules, and contract metadata tracking. * [DQX Demo Notebook for Spark Structured Streaming (Native End-to-End Approach)](https://github.com/databrickslabs/dqx/blob/v0.16.0/demos/dqx_streaming_demo_native.py) - demonstrates how to use DQX as a library with Spark Structured Streaming, using the built-in end-to-end method to handle both reading and writing. diff --git a/docs/dqx/docs/dev/docs_authoring.mdx b/docs/dqx/docs/dev/docs_authoring.mdx index 4e2e60612..2c9af46d8 100644 --- a/docs/dqx/docs/dev/docs_authoring.mdx +++ b/docs/dqx/docs/dev/docs_authoring.mdx @@ -224,8 +224,10 @@ feature documentation: The following components are available: -*`` — a status badge linked +*`` — a status badge linked to the matching section of the [Feature lifecycle](/docs/reference/feature_lifecycle) reference. + Stages run `experimental` → `alpha` → `beta` → GA as a feature matures. There is no `stage="ga"`: + a generally available feature carries no badge at all, so an untagged page is GA by definition. *`` — "Available since DQX v0.14.0", linked to that release's notes. *`` diff --git a/docs/dqx/docs/guide/row_anomaly_detection/index.mdx b/docs/dqx/docs/guide/row_anomaly_detection/index.mdx index 68ad895c4..4e3dd76ca 100644 --- a/docs/dqx/docs/guide/row_anomaly_detection/index.mdx +++ b/docs/dqx/docs/guide/row_anomaly_detection/index.mdx @@ -6,56 +6,23 @@ sidebar_position: 308 import Admonition from '@theme/Admonition'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -import Deck, { Slide } from '@site/src/components/Deck'; -import { AvailableSinceVersion, FeatureTags } from '@site/src/components/FeatureTags'; +import Deck from '@site/src/components/Deck'; +import { AvailableSinceVersion, FeatureLifecycleStage, FeatureTags } from '@site/src/components/FeatureTags'; # Row Anomaly Detection + -Use row anomaly detection to automatically find unusual rows in your data using ML (per‑record anomalies with explanations) without manually specifying thresholds so you can catch issues that rule-based checks miss. You provide recent good data; DQX trains a model and flags rows that don't fit typical patterns. No ML expertise required. Each flagged row includes an explainable breakdown of which columns drove the score, so you can see why it was flagged. - - - - Row anomaly detection in data quality — in a few minutes. - - - **Rules** catch things we already expect — like "bananas should be yellow" or "a batch must have at least 100 rows". But what about surprises we never thought to check for? That's where **anomaly detection** comes in. - - - For a single banana, we can write rules: check size, colour, flavour. These rules live in code — clear, testable, versioned. - - - Databricks Data Quality Monitoring (DQM) watches the big picture: did the data arrive? Is it fresh? Are the row counts about right? Think of it as checking the delivery trucks. - - - The truck arrived on time and the count looks right — great. But are the bananas inside actually good? DQM watches the delivery; nobody's inspecting the contents. That's the gap. - - - DQX learns what "normal" looks like from your good data, then checks every single row. No labels needed — it figures out what's unusual on its own. *"Is this banana weird?"* - - - Each banana becomes a set of numbers — size, colour, spots, bend. An Isolation Forest then tries to separate each point from the rest. If a banana is easy to isolate, it's probably odd. - - - An unusual banana gets separated in just a few steps — it sticks out. A normal banana is buried in the crowd and takes many steps to single out. - - - A score alone isn't enough — you want to know *why*. SHAP breaks it down: "too brown", "wrong size". So you can act on the insight straight away. - - - Everything you need to catch unusual rows — no ML expertise required. - - - Install DQX and start catching unusual rows in minutes. - - +Use row anomaly detection to find unusual rows in your data using ML (per‑record anomalies with explanations) without writing a rule or a numeric bound per column, so you can catch issues that rule-based checks miss. There is still one number to choose — a sensitivity, which DQX supplies a default for and which you should size against your own data. You provide recent good data; DQX trains a model and flags rows that don't fit typical patterns. No ML expertise required. Each flagged row includes an explainable breakdown of which columns drove the score, so you can see why it was flagged. + + ## What it is and why it helps -Use row anomaly detection to learn typical patterns from recent "good" data and highlight rows that look unusual when you consider multiple columns together. You do not need to define thresholds or write rules; the model learns from your data and surfaces what stands out. +Use row anomaly detection to learn typical patterns from recent "good" data and highlight rows that look unusual when you consider multiple columns together. You do not write a rule or pick a range per column; the model learns from your data and surfaces what stands out. You do choose how sensitive it is — see [Choosing a threshold](#how-to-choose-a-threshold). It works alongside your existing rule-based checks: - **Rules** catch the issues you can anticipate and describe (for example, "amount must be positive"). @@ -85,7 +52,7 @@ Because results are **explainable**, you don't just get a list of flagged rows. | **Known valid ranges** | Yes, easy to define | No (overkill) | Rules only | | **Compliance checks** | Yes, required for audit | Partial (not audit-friendly) | Rules only | | **Unusual combinations** | Partial (hard to specify) | Yes, natural fit | Anomaly + Rules | -| **Temporal patterns** | Partial (complex rules) | Yes, auto-learns | Anomaly + Rules | +| **Temporal patterns** | Partial (complex rules) | Partly, and only if configured | Anomaly + Rules | | **Unknown/unexpected issues** | No (can't anticipate) | Yes, discovers | Anomaly only | | **Cross-column dependencies** | Partial (very complex) | Yes, excels | Anomaly + Rules | @@ -109,6 +76,21 @@ Because results are **explainable**, you don't just get a list of flagged rows. **Use DQX for**: "Is this data unusual?" **Use domain models for**: "Is this data fraudulent, faulty, or malicious?" +### How well does it detect, and how to judge it + +Row anomaly detection scores each row **on its own**. It is not a time-series or sequence model and does not read a window of history to make a prediction. The high numbers you see on published anomaly-detection leaderboards are usually won by sequence models on time-series data, which is a different problem, so they are not a yardstick for DQX. The right yardstick is your own data. + +What it is reliably good at, and what it is not: + +- **Good at unusual _combinations_ across columns**: several values that are each individually fine but wrong together. Rules struggle to express that, and it is where anomaly detection earns its place. +- **Weaker than a plain rule when a single column is extreme in isolation.** A range check or an outlier rule catches that more reliably and more cheaply, so reach for a rule there. You can run both. + +Measured across ten classical tabular anomaly-detection datasets, it beats a random baseline on every one and a simple "largest z-score across columns" baseline on most. The exceptions are exactly the single-extreme-value case above. Detection quality on DQX's own synthetic fixtures, alongside training and scoring times, is published in [Benchmarks](/docs/reference/benchmarks). + + +Whatever a benchmark says, measure on **your** data before relying on a number. Train on a slice you consider good, score a slice you understand, and check that the rows it flags are ones you would actually want flagged. + + ## Complements Databricks Data Quality Monitoring [Databricks Data Quality Monitoring (DQM)](https://docs.databricks.com/aws/en/data-quality-monitoring/anomaly-detection) focuses on table-level signals like freshness and completeness. DQX row anomaly detection focuses on unusual rows and cross-column patterns. @@ -126,10 +108,10 @@ DQM and DQX each provide distinct capabilities. Together, they complement one an Each row is scored and enriched with: - **Severity percentile (0–100)**: how unusual the row is compared to training data. -- **Anomaly flag**: whether it crosses your chosen score threshold (default 95). You can tune this to control how many alerts you get. -- **Top contributors (explainability)**: which fields most influenced the anomaly score, so you can see *why* a row was flagged. Powered by SHAP, this turns a black-box score into an actionable insight. +- **Anomaly flag**: whether it crosses your chosen score threshold (default 95). The threshold is a percentile of the severity seen during *training*, so a batch with real problems in it produces more alerts than the threshold's share of rows. See [How to choose a threshold](#how-to-choose-a-threshold). +- **Top contributors (explainability)**: which fields most influenced the anomaly score, so you can see *why* a row was flagged. This turns a black-box score into an actionable insight. -You can tune the threshold and other options later if you need to reduce alert noise or catch more edge cases, but the defaults should work well for most use cases. +The defaults are a starting point, not a finished configuration. `threshold=95` is a sensitivity that has to be judged against your own data and how many rows someone can actually investigate — the section on [choosing a threshold](#how-to-choose-a-threshold) shows how to count alerts at several cutoffs before committing to one. ## Prerequisites @@ -222,12 +204,19 @@ Avoid ID-like columns (for example, `order_id`, `user_id`) in anomaly training. - name: orders input_config: location: catalog.schema.orders - anomaly_config: - columns: [amount, quantity] # optional; omit to use all supported columns - model_name: catalog.schema.orders_monitor - registry_table: catalog.schema.dqx_anomaly_models + # anomaly_config sits beside input_config, not inside it + anomaly_config: + columns: [amount, quantity] # optional; omit to auto-discover + model_name: catalog.schema.orders_monitor + registry_table: catalog.schema.dqx_anomaly_models + baseline_by: [region, product] # optional; judge each metric against its own group + profile: tabular # optional; "tabular" (default) or "correlation" + baseline_over_time: event_ts # optional; time column to fit each metric's level along ``` + Every key mirrors an argument of `anomaly_engine.train()`, so a model trained by hand can be reproduced + by a scheduled run and vice versa. Omitting a key is the same as omitting that argument. + Then trigger the anomaly-trainer workflow via the CLI: ```bash @@ -251,7 +240,7 @@ However, when you run a row anomaly detection check, DQX also adds a `_dq_info` The info column is an array of structs, with one element per anomaly detection check that was applied. -Feature contributions are on by default and are computed only for anomalous rows (severity at or above the threshold) — non-anomalous rows carry a `null` contributions map, so the SHAP cost scales with the number of anomalies, not the table size. Set `enable_contributions=False` to skip them entirely for the fastest scoring. Severity percentiles are always computed for every row, so you can still count would-be anomalies at other thresholds after scoring; but if you want contributions (and AI explanations) for rows below your enforcement threshold, score with the lower exploration threshold and filter afterwards: +Feature contributions are on by default and are computed only for anomalous rows (severity at or above the threshold). Non-anomalous rows carry a `null` contributions map, so the cost of computing them scales with the number of anomalies, not the table size. Set `enable_contributions=False` to skip them entirely for the fastest scoring. Severity percentiles are always computed for every row, so you can still count would-be anomalies at other thresholds after scoring; but if you want contributions (and AI explanations) for rows below your enforcement threshold, score with the lower exploration threshold and filter afterwards: ```python DQDatasetRule( @@ -260,7 +249,7 @@ DQDatasetRule( check_func_kwargs={ "model_name": "catalog.schema.orders_monitor", # fully qualified name "registry_table": "catalog.schema.dqx_anomaly_models", # fully qualified name - # "enable_contributions": False, # optional: skip SHAP (also disables AI explanations) + # "enable_contributions": False, # optional: skip contributions (also disables AI explanations) } ) ``` @@ -283,81 +272,405 @@ display( ) ``` -Scores are normalized into `severity_percentile` (0–100). The anomaly threshold is a percentile cutoff (default 95) that you should tune to your data. +Scores are normalized into `severity_percentile` (0–100). The anomaly threshold is a cutoff on that value (default 95), calibrated against the severity distribution seen during training, that you should tune to your data. ## How to choose a threshold -The threshold controls how many rows are flagged as anomalies (percentile cutoff; default 95). Lower values (for example 90) flag more rows; higher values (for example 98) flag fewer (only extreme anomalies). Start with the default (95). If you get too many alerts, raise the threshold; if you are missing issues you care about, lower it. +The threshold controls how many rows are flagged as anomalies (percentile cutoff; default 95). Lower values +(for example 90) flag more rows; higher values (for example 98) flag fewer (only extreme anomalies). Start +with the default (95). If you are missing issues you care about, lower it; if you are investigating more +rows than you have time for, raise it. -## How it works under the hood +### What the number means -For full parameter and schema details, see [Row Anomaly Detection in Quality Checks](/docs/reference/quality_checks#row-anomaly-detection). +`threshold=95` flags rows above the 95th percentile of the severity DQX saw **while training**. It does +*not* mean "flag 5% of the rows I am scoring" — and that one distinction explains most surprises: -### Architecture overview +| You see | Most likely because | Do this | +|---|---|---| +| A few more alerts than your budget | The batch contains real problems. Alert counts grow with the size of the problem instead of being capped at a fixed share | Nothing. This is the design working | +| Far more alerts than your budget | The input distribution moved away from what the model trained on | Find out *why* before retraining — see below | +| Low precision | The alert count itself caps it | Compare against that ceiling, not against 100% | -1. **Feature engineering** (automatic): DQX detects column types and creates features (numerical standardized, categorical one-hot or frequency-encoded, temporal expanded to hour/day/month/weekend). -2. **Smart sampling and training**: DQX samples your data (default 30%, capped at 1M rows), trains an ensemble of Isolation Forest models, and captures baseline statistics for drift detection. -3. **Model registry**: Models and metadata live in MLflow and a Delta table; segmented models use deterministic names (for example `__seg_region=US_tier=gold`). -4. **Scoring and explanation**: Raw scores are normalized to a 0–100 severity percentile. SHAP contributions (which features drove each score) are on by default and computed only for anomalous rows; set `enable_contributions=False` to skip them for the fastest scoring. -5. **Auto-discovery of columns and segments**: When you call `train()` without `columns` or `segment_by`, DQX automatically discovers both. It selects numeric columns with enough variance as features and may **auto-segment** when it finds suitable segment columns: categorical or low-cardinality columns with 2–50 distinct values, low null rate, and enough rows per segment (for example region, product category). If your auto-trained model is segmented, that is expected. To force a single global model, pass `segment_by=[]` (or omit segment columns from the data used for discovery) or set `columns` and `segment_by` explicitly. +Each row above, with the measurement behind it: -### Why Isolation Forest? +- **Real problems add alerts on top of the budget.** On correlated machine telemetry where 4.0% of rows were + genuinely faulty, `threshold=95` flagged 8.4% of the batch — 5.2% of the *normal* rows, which is the budget + delivered almost exactly, plus most of the faults on top. A batch with nothing wrong with it gives you + roughly your budget and no more, provided it resembles the training data. -Isolation Forest measures how "easy" it is to isolate a data point; anomalies are isolated in few splits, normal points need many. It is fast, handles mixed types, is robust to noise, and is explainable via SHAP. DQX uses it by default because it fits data quality use cases without tuning. +- **A moved distribution can multiply alerts several times over.** Scoring a later time period with a model + fitted on an earlier one flagged 9.8%–21.7% of rows depending on the detector, where the same model on + same-period held-out data flagged 4.9%–5.0%. The calibration was sound; the input had moved. Set + `drift_threshold=3.0` to be told this rather than inferring it from alert volume. -### Output structure and options + **Retraining is the last step, not the first.** A jump in alerts looks identical to a changed unit or + schema, a mis-set `baseline_by` / `baseline_over_time`, an upstream incident, or a real business change. + Rule those out first: retraining on data that contains a real problem teaches the model to accept it. -The `_dq_info` column is an array of structs (one element per dataset-level check that produces info; for row anomaly, one per `has_no_row_anomalies` check). Use `severity_percentile` for threshold decisions; raw `score` is for diagnostics only. Enable `drift_threshold` (for example `3.0`) to get warnings when the scoring distribution shifts from training so you know when to retrain. +- **Precision cannot exceed the true-fault share of your alerts.** With 65 alerts of which 24 rows are + genuinely wrong, no model beats 24/65 = 37%, however well it ranks. On one fixture precision hit that + ceiling exactly, with a perfect ranking underneath — the 24 highest-severity rows *were* the 24 faults. + Measured against 100%, that would have looked like a poor detector. -Anomalous records can be identified using the standard reporting columns (`_errors` and `_warnings`). The `_dq_info[0].anomaly.is_anomaly` field provides additional detail for in-depth analysis and is set to `False` for records that are not anomalous. +### Calibrating on your own data -### Schema of the info column (_dq_info) +`severity_percentile` is computed for every row, including rows below the threshold, so you can count +would-be alerts at other cutoffs without rescoring anything: -`_dq_info` has type **array of structs**. Each array element corresponds to one dataset-level check that writes info (for example one `has_no_row_anomalies` check). Element order matches the order of checks; the first anomaly check is at index `0`. +```python +scored = spark.table("catalog.schema.scored") +severity = F.element_at(F.col("_dq_info"), 1).getField("anomaly").getField("severity_percentile") +total = scored.count() -Each element is a **struct** with a shared “wide” schema. Currently, the only field populated by row anomaly detection is **`anomaly`**. Other check types may add more top-level fields in the future. +for cutoff in (90, 95, 98, 99, 99.5): + alerts = scored.filter(severity >= cutoff).count() + print(f"threshold {cutoff}: {alerts} alerts ({alerts / total:.2%} of rows)") +``` -**Nested `anomaly` struct** (when the check is row anomaly detection): +One precision caveat, because it decides which cutoffs this loop can answer. The published +`severity_percentile` carries exactly the precision the *scoring* threshold needed — one decimal for a +threshold of 95, two for 99.95 — which is what makes `severity >= threshold` agree with `is_anomaly`. So +counts are exact at any cutoff at or coarser than that precision, and **undercount** at a finer one: scored +at `threshold=95`, every severity in `[99.95, 100)` was published as `99.9`, so a cutoff of `99.95` misses +those rows. Stay at or below the scoring threshold's precision, or rescore at the finer threshold. The +`threshold` field sits next to `severity_percentile` in the same struct, so a scored table always says which +precision it carries. + +Pick the cutoff whose alert count matches how many rows someone can actually investigate. Both anomaly demos +under `demos/` print this table, and one of them chooses its threshold from it rather than inheriting the +default. + +A tuned threshold transfers only partly. Measured across machines in the same dataset, tuning on one half of +a batch and applying to the other cut the median error against the requested budget from 3.1x to 1.7x and +the spread across machines from 28 to 7.5 percentage points, but only 20-30% of machines landed within +double their budget. Treat a tuned number as a much better starting point than a guess, not as a guarantee. + +:::note[Very high thresholds are directional, not calibrated] +Severity is calibrated on a persisted grid of quantile knots, and above roughly 99 the value between knots +is interpolated in tail probability. That interpolation assumes the score distribution's tail decays +exponentially — which held on four synthetic distributions and on real telemetry, but a heavier tail +overshoots: on a lognormal score distribution `threshold=99.9` flagged 0.21% of rows against the 0.10% +requested, a factor of 2.1. + +So treat 99.5 and above as "much stricter than 99" rather than as a share you can budget against. At 99 and +below the calibration sits on the knots themselves and the budget holds as described above. +::: + +## Group-aware anomaly detection -| Field | Type | Description | -|--------|------|-------------| -| `check_name` | string | Always `"has_no_row_anomalies"` for this check. | -| `score` | double | Raw model score (0–1). Use for diagnostics only. | -| `severity_percentile` | double | Normalized score 0–100. **Use this for thresholds and ordering.** | -| `is_anomaly` | boolean | `true` if `severity_percentile` ≥ threshold. | -| `threshold` | double | Severity percentile threshold used (e.g. 95.0). | -| `model` | string | Full model name (e.g. Unity Catalog name). | -| `segment` | map<string, string> | Segment key-value pairs for segmented models; `null` for global models. | -| `contributions` | map<string, double> | Per-feature contribution percentages (0–100). On by default (`enable_contributions=True`); populated only for anomalous rows — `null` for non-anomalous rows or if you set it `False`. | -| `confidence_std` | double | Ensemble score standard deviation. Present when `enable_confidence_std=True`; `null` otherwise. | -| `ai_explanation` | struct | LLM-generated explanation for the row's `(segment, pattern)` group. On by default (`enable_ai_explanation=True`); `null` for non-anomalous rows, when disabled, or when no serving endpoint is reachable. See [AI explanations](#ai-explanations) below. | + + + -The nested `ai_explanation` struct (populated when AI explanations are on — the default): +Some values are only wrong *in context*. If one country's daily order volume drops 80% while the overall total holds steady (because other countries absorbed the difference), the collapsed number still sits comfortably inside the range other countries occupy normally. A model that compares every row against the whole table cannot see it. -| Field | Type | Description | -|--------|------|-------------| -| `narrative` | string | Plain-language description of why the group was flagged. | -| `business_impact` | string | Likely downstream impact if the rows are processed unchanged. | -| `action` | string | What an analyst should investigate. | -| `top_features` | string | Deterministic top-2 contributing features (e.g. `amount+quantity`) — the group's pattern key. | -| `group_size` | long | Number of anomalous rows in this `(segment, pattern)` group. | -| `group_avg_severity` | double | Mean `severity_percentile` across the group. | +Pass `baseline_by` to give DQX the right basis for comparison: + + +Grouping and time are **independent of the detector**. The examples in this section and the next leave +`profile` unset, so they use the default `"tabular"` detector and each shows one new argument at a time — +but every one of them works the same way with `profile="correlation"`. See +[Choosing a profile](#choosing-a-profile). + + +```python +anomaly_engine.train( + df, + model_name="catalog.schema.orders_model", + registry_table="catalog.schema.dqx_anomaly_models", + columns=["order_count"], + baseline_by=["country", "product"], +) +``` + +Now each metric is judged against its own group's baseline, so the collapse stands out even though its value is ordinary for the table as a whole. + + +`baseline_by` conditions on **level** — where each group's values normally sit — not on the **relationships** between metrics inside a group. One model is still fitted, and it learns one set of relationships shared across all groups. + +That is the right trade for the usual case, where groups differ in scale and agree in behaviour: one region's orders run ten times another's, but order count and revenue move together everywhere. It cannot express groups whose metrics relate to each other *differently*. Measured on a deliberate counterexample — group A where `y` rises with `x`, group B where `y` falls as `x` rises, both centred identically — the pooled model scores a row that is impossible for B at 1.3, where a model fitted on B alone scores it 260. Because the two groups have the same medians, group-relative features are identical and the transform changes nothing. + +If your groups genuinely behave differently rather than merely sitting at different levels, train one model per group and give each its own `model_name`. The same applies to `baseline_over_time`: it removes one shared trend, so groups trending in opposite directions or at different phases are not separated by it either. + + + +A baseline column is the basis of comparison, not a metric being compared, so it never becomes a model feature. Passing the same column in both `columns` and `baseline_by` is an error, and if you let DQX auto-discover `columns`, it drops your declared baseline columns from the feature list for you. Baseline columns must be string, integral, boolean, or date. Float, double, and decimal are rejected, because Spark and Python format floating-point values differently and the baseline lookup would silently match nothing. Bucket the value or cast it to a string first. + + +### Groups that appear after training + +A row whose group was never seen during training gets a **null** score and severity, plus `is_new_baseline = true` and the unrecognised key in `new_baseline_key`: + +```python +result.filter(F.col("_dq_info")[0].anomaly.is_new_baseline).select("_dq_info") +``` + +It is not flagged as a violation. Neither categorical encoder can represent an unseen value honestly: one-hot makes it look maximally normal, and frequency encoding makes it look maximally extreme. So DQX cannot judge the row, and "could not judge" is a different claim from "is anomalous". Previously such rows were scored 0.0 and silently passed, which is the most normal-looking value in the table. + +If an unrecognised group value is itself something you want to fail on, that is a membership question rather than an anomaly one: use `foreign_key` or `is_in_list` on the baseline column against your set of known values. + +## Comparing against time + + + + + +Some values are only wrong *for when they happened*. A metric that has grown steadily for a year sits +outside the range it trained on, so ordinary rows start looking anomalous; and a value that is perfectly +normal against the whole year can be badly wrong for where the trend had actually got to. Comparing +against the whole table cannot see either. + +Pass `baseline_over_time` to give DQX a time axis: + +```python +anomaly_engine.train( + df, + model_name="catalog.schema.revenue_model", + registry_table="catalog.schema.dqx_anomaly_models", + columns=["revenue", "orders"], + baseline_over_time="event_ts", +) +``` + +DQX fits each metric's expected level as a function of time, persists it with the model, and gives the +detector the *difference* between the observed value and that expectation. Because the expectation is a +function rather than a lookup table, it extends to timestamps the training window never contained. + +It is the third of three independent questions, and they compose. The column you name here is treated as an +**axis**, never as a feature, which is the distinction most worth holding on to: listing a timestamp in +`columns` instead expands it into seven calendar features and answers a completely different question. + +| Question | Argument | What it changes | +|---|---|---| +| Is this value unusual on its own, or in combination? | `profile` | Which detector scores the row | +| Is it unusual for its own group? | `baseline_by` | Each metric is compared against its own group's level | +| Is it unusual for the time of day or day of week? | list it in `columns` | A timestamp is expanded into seven calendar features | +| Is it unusual for its own point in time? | `baseline_over_time` | A timestamp becomes the comparison **axis**, never a feature | + +Pass no `columns` at all and auto-discovery includes any timestamp it finds, so the third row is what you get +by default. DQX warns at training time when that happens and names both ways out. + +Set both grouping and time and the expectation is fitted on the group-relative value rather than the raw +metric. That buys one specific thing, and it is worth being exact about which: + +- **Supported.** Groups sitting at different *levels* that share one trend shape. Every region's orders grow + about 10% a month from its own base: the group-relative transform removes the level difference, and one + fitted curve then describes all of them. One model is enough. +- **Not supported.** Groups whose trends have different *shapes* — one growing 10% a month while another + grows 50%, or two moving in opposite directions, or the same cycle at different phases. One coefficient + vector per metric is fitted, and removing a level difference does not make two different slopes into one. + Train a model per group, each with its own `model_name`. + +### When to use it, and when not to + +The parameter is not free, so this table is worth reading before reaching for it. + +| Situation | Recommendation | +|---|---| +| The metric carries a real trend, or a genuine daily or weekly shape | Use it | +| The metric is largely stationary | Leave it off. Subtracting a fitted expectation from data with no temporal structure removes signal and adds the fit's own error | +| You want to catch a value that is wrong *for a Saturday* | Use it, and keep the timestamp out of `columns` | +| Using `profile="tabular"` | Exclude other datetime columns from `columns`; calendar features on top of the residual measured worse on every anomaly shape tested | +| The training window is short | A seasonal term needs several complete cycles to be identifiable. DQX fits one only where the window supports it, and logs the period it skipped and why | + +DQX will not turn this on for you. Whether a metric's own history is worth comparing against is a +judgement about the data, and it cannot be verified without labelled anomalies. What DQX does instead is +tell you when the training window shows almost no structure over time, so you learn the parameter is +unlikely to help before you rely on it. + +### It is not a forecaster + +`baseline_over_time` models the level expected *at* a time. It does not predict the next value, and it +never looks at the previous row: every row is judged from its own timestamp alone, which is what keeps +scoring valid on a streaming DataFrame. A sudden jump that lands exactly where the trend expected is not +an anomaly to this feature. + +### Knowing when the model has run out of evidence + +A fitted expectation extrapolates, but its accuracy decays the further past the training window you go. +Rows beyond that window carry `is_stale_baseline`, and `stale_baseline_horizon` records where the +evidence ran out: + +```python +result.filter(F.col("_dq_info")[0].anomaly.is_stale_baseline).select("_dq_info") +``` + +The score is still produced, unlike the unseen-group case above. Near the boundary it is still accurate, +so nulling it would throw away a usable verdict; far out it is not, and the flag is how you tell the +difference. Treat it as a signal to retrain rather than as a violation. + +## Choosing a profile + + + + + +`profile` selects **how DQX decides a row is unusual**. It is the one modelling decision DQX asks you to +make, and it exists because a single algorithm cannot cover both cases well. It is a statement about the +method, not about the shape of your table: both profiles accept the same data, and either can be combined +with the group and time context below. + +| What makes a row unusual | `profile` | Typical data | +|---|---|---| +| Its *values*, or its combination of values, are unusual | `"tabular"` (default) | Independent records — transactions, orders, customers, events | +| Metrics that normally move **together** stop doing so, while each one stays inside its usual range | `"correlation"` | Multivariate metrics — machine telemetry, service metrics, sensor readings | + + +It needs **no timestamp column and no time-ordered data**, and it does **not** forecast. It models how your +metrics relate to each other within a row, so rows may arrive in any order. If what you want is a comparison +against a metric's own expected level over time, that is `baseline_over_time` below — an independent option +you can add to *either* profile. The name is the accessible label rather than a precise one: the detector +uses the full covariance between metrics, not pairwise correlation alone. + + +```python +anomaly_engine.train( + df, + model_name="catalog.schema.fleet_model", + registry_table="catalog.schema.dqx_anomaly_models", + profile="correlation", +) +``` + +Nothing else changes: the same automatic feature engineering, the same registry, the same +`has_no_row_anomalies` check, the same contributions and AI explanations. Scoring reads the choice back +off the model, so you never repeat it. -**Access in PySpark:** use `F.element_at(F.col("_dq_info"), 1)` for the first element (1-based), then `.getField("anomaly").getField("severity_percentile")` etc. Alternatively `F.col("_dq_info").getItem(0)` for 0-based index (see [Troubleshooting](/docs/guide/row_anomaly_detection/troubleshooting) for Spark Connect–friendly patterns). +### Why the choice matters -### AI explanations +**The default checks one column at a time. The other reads the whole row at once.** That single difference +decides which anomalies each one can see, so pick on the **shape of the anomaly you expect** rather than on an +expected accuracy gap. -AI explanations are **on by default**: each anomalous row gets a plain-language, LLM-generated explanation in `_dq_info[0].anomaly.ai_explanation` answering *why was this flagged, what's the impact, and what should I do* — without anyone reading raw SHAP percentages. Set `enable_ai_explanation=False` to turn it off (or `enable_contributions=False`, which disables explanations too, since they use the SHAP contributions as input). The explanation is AI-generated from the anomaly signal (feature names + SHAP + severity); it is **not** grounded in your catalog's table/column descriptions, so treat the business-impact and action as a starting point, not authoritative. +| If a bad row looks like this | Use | Why | +|---|---|---| +| An order of 8,000 units where the largest ever was 500 — one value, plainly out of range | `"tabular"` (default) | Checking columns one at a time is enough to catch it, and this is the only profile that reports `confidence_std` | +| An order of 500 units for £2 — each number is ordinary, the pair is impossible | `"correlation"` | Only a detector reading the values together can see that the *combination* never occurs | +| A server at 95% CPU with idle memory — normal apart, never seen together on a healthy machine | `"correlation"` | The same reason: no single-column check separates this row | +| You genuinely do not know yet | `"tabular"` | It is the default, and on ordinary tabular data the two are closer than the telemetry figures below suggest. Switching later costs one retrain | + +
+The measured difference, and what it does and does not tell you + +Measured on the Server Machine Dataset: real machine telemetry, 28 machines, 38 metrics, 327 labelled +incidents. Trained on an earlier period and scored on a later one, which is what you do when you train once +and check new data: + +| `profile` | Incidents surfaced, 1% budget | Average precision | +|---|---|---| +| `"tabular"` | 57.5% | 0.277 | +| `"correlation"` | **65.3%** | **0.423** | + +An incident counts as surfaced if the detector flags at least one of its rows, so the first column measures +whether you would have been paged. + +Two limits on reading across from this. It is telemetry, which is exactly what `"correlation"` is for, and +the comparison is between detectors on raw metric matrices — DQX's own feature engineering is not part of it. +Detection quality on DQX's own synthetic fixtures is published separately in +[Benchmarks](/docs/reference/benchmarks). + +
+ +
+How to read contributions on either profile + +Both profiles report contributions the same way: one entry per column you passed, and on an ensemble the mean +across members — the same members, in the same proportions, that produced the score. Only the method differs +(SHAP for `"tabular"`, an exact decomposition for `"correlation"`). + +Read the result as a **ranking of columns**, not as a breakdown of the score's magnitude. Averaging +decomposes the members' mean isolation depth exactly, while the score is the mean of a strictly monotone but +*nonlinear* function of that depth. The two order rows almost identically without being the same quantity. + +
+ +
+Unseen categories: a real difference, but narrower than it looks + +No accuracy number shows this one. A **category that never appeared in training** — a new payment type, an +unrecognised status code — is caught by `"correlation"` and generally not by `"tabular"`, **but only where +the column is one-hot encoded**, meaning its cardinality is at or below the categorical threshold (20 by +default). + +There, every category keeps its own indicator and the indicators sum to one on every training row, so an +unseen value sets none of them. A detector reading features jointly sees a row off the surface all its +training data lay on; Isolation Forest splits one feature at a time, so an all-zero row sits inside every +indicator's own range and scores as ordinary. + +Above that threshold the column is frequency-encoded into a single number and an unseen value becomes `0.0`, +which carries much less. If the training frequencies were near-uniform, that coordinate barely varies, the +correlation-aware detector drops it from the distance as constant, and an unseen value does not move the +score at all — measured with 21 equally frequent categories, a known and an unseen value both scored 0.1837. + +**So neither profile substitutes for a vocabulary check.** If unrecognised values are something you need to +fail on, that is a membership question rather than an anomaly one — use `is_in_list` or `foreign_key` on the +column, under either profile. + +
+ + +DQX does not detect which profile you need. Getting it right cannot be verified without labelled +anomalies, which an unsupervised tool does not have. The one cheap signal — serial correlation between +consecutive rows — was measured and rejected: it is produced just as readily by *sorted storage*, so on +ordinary tabular tables loaded in batches it fires constantly. Rather than guess, DQX defaults to +`"tabular"` and logs the resolved profile on every training run. + + +### What `"correlation"` does and does not need + +- **No timestamp column.** It models correlation *between* metrics, not behaviour over time. Rows may + arrive in any order. +- **No ensemble.** The detector is deterministic, so `ensemble_size` is ignored and `confidence_std` is + unavailable — averaging identical models would cost N times as much and report a spread of zero. +- **Enough rows.** It estimates how the metrics co-vary, which needs many more rows than features. + Training warns when the sample is thin relative to the feature count. + +### Which route fits your problem + +Detector choice and comparison context are separate decisions, so this maps a problem onto both — and says +where DQX has no answer, which matters more than the rows where it does. + +| Your problem | Route | The limit | +|---|---|---| +| One record has unusual values, or an unusual combination of them | `profile="tabular"`, and keep explicit rules for bounds you already know | No guarantee every joint pattern is caught | +| Numeric fields normally agree, and this row breaks that relationship | `profile="correlation"` | Needs no timestamp; does not imply arbitrary non-linear relationships are detected | +| A value is unusual *for its region or product*, though ordinary overall | Add `baseline_by` to either profile | Learns group *levels*, not a separate relationship model per group, and does not refresh itself | +| A value is unusual *for its own expected level at that time* | Add `baseline_over_time` to either profile | Needs representative history; extrapolates only a limited way beyond it, and is not a forecaster | +| Several individually ordinary rows form an unusual **sequence** | **No detector here does this.** Aggregate upstream, then check the aggregates as rows | A row detector may flag some rows of the sequence, but that is not evidence it recognised the sequence | +| A table's freshness, volume or an aggregate metric changed | Table- or metric-level monitoring, not this check | A different unit of observation from a single row | + +A row can be both unusual in itself and unusual for its context — these are not exclusive categories, which +is why context is a separate option rather than a third profile. Note also that a broken relationship +*between columns of one row* is not a sequence anomaly, and neither is grouping rows for a shared AI +explanation. + +### What is handled automatically + +Both profiles get the same feature engineering, so these need no configuration: + +- **Calendar seasonality.** Include a date or timestamp column and DQX derives cyclical hour-of-day, + day-of-week and month features plus a weekend flag — seven features per datetime column, encoded as + sine/cosine pairs so that 23:00 and 00:00 are adjacent rather than maximally far apart. A quiet Sunday + is therefore learned as normal *for a Sunday*, and the same volume on a Wednesday still stands out. +- **Group context.** `baseline_by` judges each metric against its own group's baseline; see + [Group-aware anomaly detection](#group-aware-anomaly-detection) above. +- **Categoricals, booleans and nulls.** One-hot or frequency encoding by cardinality, 0/1 mapping, and an + explicit indicator for columns that contain nulls. + +## AI explanations + +AI explanations are **on by default**: each anomalous row gets a plain-language, LLM-generated explanation in `_dq_info[0].anomaly.ai_explanation` answering *why was this flagged, what's the impact, and what should I do*, without anyone reading raw contribution percentages. Set `enable_ai_explanation=False` to turn it off (or `enable_contributions=False`, which disables explanations too, since they use the contributions as input). The explanation is AI-generated from the anomaly signal (feature names + contributions + severity); it is **not** grounded in your catalog's table/column descriptions, so treat the business-impact and action as a starting point, not authoritative. The call runs inside Spark via the SQL `ai_query` function against a Databricks Model Serving endpoint, so it needs no extra setup and scales with the cluster. This requires **Databricks serverless compute or Databricks Runtime 15.4 LTS or above** (where `ai_query` is available); on older runtimes explanations are skipped with a warning and scoring still completes. Similar anomalous rows are grouped together and the model is called **once per group** rather than once per row, so cost stays predictable on large datasets. -Rows are grouped by their top two contributing features, so occasionally two different kinds of anomaly that share the same top two features land in the same group and get one shared explanation. That's intentional — it keeps the number of AI calls (and the cost) low, at the price of a slightly more general explanation for those rows. +Rows are grouped by their top two contributing features, so occasionally two different kinds of anomaly that share the same top two features land in the same group and get one shared explanation. That's intentional: it keeps the number of AI calls (and the cost) low, at the price of a slightly more general explanation for those rows. ```python from databricks.labs.dqx.rule import DQDatasetRule from databricks.labs.dqx.anomaly.check_funcs import has_no_row_anomalies -# Contributions + AI explanations are on by default — this is all you need: +# Contributions + AI explanations are on by default. This is all you need: checks = [ DQDatasetRule( criticality="error", @@ -375,11 +688,87 @@ checks = [ ``` -* Explanations are **on by default** and call a Model Serving endpoint, so they add per-run LLM cost. `max_groups` (default 500) caps how many groups the model is called for per run. For segmented models the cap is shared across segments (at least one call each), so if you set `max_groups` lower than the number of segments you still get one call per segment — a warning is logged when that happens. Set `enable_ai_explanation=False` to turn explanations off. -* No serving endpoint? If the configured endpoint isn't reachable (e.g. Foundation Model APIs aren't enabled in the workspace), explanations are skipped with a warning and scoring still completes — nothing breaks. -* `redact_columns` keeps the listed feature and segment names out of the prompt. Segment **values** for non-redacted keys are sent verbatim — avoid segmenting on sensitive columns, or list them in `redact_columns`. +* Explanations are **on by default** and call a Model Serving endpoint, so they add per-run LLM cost. `max_groups` (default 500) caps how many anomaly groups the model is called for per run. Set `enable_ai_explanation=False` to turn explanations off. +* No serving endpoint? If the configured endpoint isn't reachable (e.g. Foundation Model APIs aren't enabled in the workspace), explanations are skipped with a warning and scoring still completes, so nothing breaks. +* `redact_columns` keeps the listed feature names out of the prompt. Their contribution entries are **dropped** from what the model is shown, not relabelled, and they are excluded from the group's pattern key too. The remaining shares are renormalised across what is left, so the explanation says how much of the *disclosed* evidence each column carried and states plainly when evidence was withheld — a column shown at 100% can be a small part of what the model actually measured. Note the scope: this governs what is sent to the serving endpoint. Your own `_dq_info[0].anomaly.contributions` map is unchanged and still lists every column, so anyone who can read the scored table can see the full picture; `redact_columns` is not an access control. +## Upgrading and breaking changes + +Row anomaly detection was **Experimental** in earlier releases and is now **Beta**. This release replaces the old per-group model path (`segment_by`) with `baseline_by`, which measured better on detection, reliability, and cost, and removes one parameter that never did what its name promised. + +Two changes need action from you, and the rest is handled. **Every existing model must be retrained** whether or not you used any of this. And if you passed `segment_by` or `expected_anomaly_rate` to `train()`, those calls will now raise. + +- **Replace `segment_by` with `baseline_by`.** They are not the same. `segment_by` trained one model per group; `baseline_by` judges each metric against its own group's baseline on a single model. Your scores will change. `AnomalyParams.max_segment_models` is gone too, since there is only ever one model now. +- **Remove `expected_anomaly_rate`.** It set the estimator's `contamination`, which places only scikit-learn's own `predict` boundary, so it never changed which rows DQX flagged. Nothing replaces it, because nothing needs to: to change how much gets flagged, set `threshold` on the check, which is an alert budget over training severity. If you were relying on `contamination` itself — because you load the registered model and call `predict` yourself — set `params.algorithm_config.contamination` instead. The default is unchanged at 0.02, so a call that did not pass it behaves identically. +- **Retrain your models.** A model trained before this release raises an error at scoring time until you retrain it, rather than scoring against a feature list that no longer matches. +- **Your registry table is handled for you.** Retraining writes the new schema into your existing table in place. The `segmentation` column becomes `grouping`, and a table you never retrain into keeps the old column and will not be read. +- **`_dq_info[].anomaly` changes shape.** The `segment` field is gone (it was always null once per-group models were), and `is_new_baseline`, `new_baseline_key`, `is_stale_baseline` and `stale_baseline_horizon` are added. Named-field queries keep working, but appending to a table that already holds `_dq_info` needs `mergeSchema`. +- **Auto-discovered groupings may score differently** even with no configuration change, because a discovered grouping is now used as `baseline_by` and the policy that picks it is finer than the segmented one it replaced. + +## How it works under the hood + +For full parameter and schema details, see [Row Anomaly Detection in Quality Checks](/docs/reference/quality_checks#row-anomaly-detection). + +### Architecture overview + +1. **Feature engineering** (automatic): DQX detects column types and creates features (numerical standardized, categorical one-hot or frequency-encoded, temporal expanded to hour/day/month/weekend). +2. **Smart sampling and training**: DQX samples your data (default 30%, capped at 1M rows), trains the detector your `profile` selects — an ensemble of Isolation Forest models by default, or a single correlation-aware model for `profile="correlation"` — and captures baseline statistics for drift detection. +3. **Model registry**: Models and metadata live in MLflow and a Delta table; each `train()` call registers one model under its name and archives the previously active version. +4. **Scoring and explanation**: Raw scores are normalized to a 0–100 severity percentile. Feature contributions (which features drove each score) are on by default and computed only for anomalous rows; set `enable_contributions=False` to skip them for the fastest scoring. How they are computed depends on the detector — SHAP for Isolation Forest, an exact decomposition for the correlation-aware detector — but both report **one entry per column you passed**, not per engineered feature, so you read back your own column names. That matters because DQX derives up to three features from one numeric column; scoring each separately splits the column's evidence between them and understates it. Each value is that column's share of the evidence that made the row look unusual. A column that made the row look *more* normal shows `0`, and a flagged row for which nothing pointed towards an anomaly carries an all-null map rather than an invented even split. With an ensemble the contributions are the mean across members — the aggregate the score comes from — but what averaging decomposes exactly is the members' mean isolation depth, not the mean score, which is a nonlinear function of it. Read the shares as a ranking of columns, not as a breakdown of the score. +5. **Auto-discovery of columns and grouping**: When you call `train()` without `columns`, DQX selects numeric columns with enough variance as features. It also looks for a grouping (categorical or low-cardinality columns with 2–50 distinct values, low null rate, and enough rows per group, for example region or product category). A discovered grouping is used as `baseline_by` (see [Group-aware anomaly detection](#group-aware-anomaly-detection)), so it costs one model regardless of how many groups it turns out to have. To compare against the whole table instead, pass `baseline_by=[]`. When you *do* pass `columns`, you have decided what to measure, so DQX leaves the comparison pooled rather than adding a grouping you did not ask for. If your data looks grouped it says so in a warning naming the grouping to pass. + +### Which algorithm, and why + +**Isolation Forest** (`profile="tabular"`, the default) measures how "easy" it is to isolate a data point; anomalies are isolated in few splits, normal points need many. It is fast, handles mixed types, is robust to noise, and explains itself through SHAP. DQX uses it by default because it fits data quality use cases without tuning. + +**A correlation-aware detector** (`profile="correlation"`) measures how far a row sits from normal *once the relationships between metrics are accounted for* — a distance in a space where the metrics have been decorrelated. That is exactly the case Isolation Forest is weakest on, because a broken relationship between two in-range values cannot be separated by splitting either one. It explains itself by leaving each feature out in turn and reporting how much of the anomaly disappears, so it needs no SHAP. See [Choosing a profile](#choosing-a-profile). + +### Output structure and options + +The `_dq_info` column is an array of structs (one element per dataset-level check that produces info; for row anomaly, one per `has_no_row_anomalies` check). Use `severity_percentile` for threshold decisions; raw `score` is for diagnostics only. Enable `drift_threshold` (for example `3.0`) to get warnings when the distribution of the input features shifts away from the training baseline, so you know when to retrain. + +Anomalous records can be identified using the standard reporting columns (`_errors` and `_warnings`). The `_dq_info[0].anomaly.is_anomaly` field provides additional detail for in-depth analysis and is set to `False` for records that are not anomalous. + +### Schema of the info column (_dq_info) + +`_dq_info` has type **array of structs**. Each array element corresponds to one dataset-level check that writes info (for example one `has_no_row_anomalies` check). Element order matches the order of checks; the first anomaly check is at index `0`. + +Each element is a **struct** with a shared “wide” schema. Currently, the only field populated by row anomaly detection is **`anomaly`**. Other check types may add more top-level fields in the future. + +**Nested `anomaly` struct** (when the check is row anomaly detection): + +| Field | Type | Description | +|--------|------|-------------| +| `check_name` | string | Always `"has_no_row_anomalies"` for this check. | +| `score` | double | Raw model score (0–1). Use for diagnostics only. | +| `severity_percentile` | double | Normalized score 0–100. **Use this for thresholds and ordering.** Published so that `severity_percentile >= threshold` gives the same answer as `is_anomaly`: it is floored to the threshold's own precision rather than rounded, so it never reads higher than the value the flag was decided on. | +| `is_anomaly` | boolean | `true` if `severity_percentile` ≥ threshold, and the authoritative decision. | +| `threshold` | double | Severity percentile threshold used (e.g. 95.0). | +| `model` | string | Full model name (e.g. Unity Catalog name). | +| `contributions` | map<string, double> | Each column's share (0–100) of the evidence that made the row look unusual, keyed by **the columns you passed** under either profile — see step 4 of [How it works](#how-it-works-under-the-hood). A column that argued the row was normal shows `0`. On by default (`enable_contributions=True`); populated only for anomalous rows (`null` for non-anomalous rows or if you set it `False`), and all-null on a flagged row where nothing pointed towards an anomaly. | +| `confidence_std` | double | Ensemble score standard deviation. Present when `enable_confidence_std=True`; `null` otherwise. | +| `is_new_baseline` | boolean | `true` when the row's group was absent from training, in which case `score` and `severity_percentile` are `null` and the row is **not** flagged. See [Groups that appear after training](#groups-that-appear-after-training). | +| `new_baseline_key` | string | The unrecognised group key, for rows where `is_new_baseline` is `true`; `null` otherwise. | +| `ai_explanation` | struct | LLM-generated explanation for the row's anomaly group (rows sharing the same top contributing features). On by default (`enable_ai_explanation=True`); `null` for non-anomalous rows, when disabled, or when no serving endpoint is reachable. See [AI explanations](#ai-explanations) below. | +| `is_stale_baseline` | boolean | `true` when the row's timestamp falls beyond the window a `baseline_over_time` baseline was fitted on, so its expected level is an extrapolation. Unlike an unseen group the row **is** still scored and can be flagged — a fitted curve does extrapolate, and near the boundary it stays usable — but accuracy degrades with distance, so treat it as a signal to retrain rather than as a verdict about the row. `null` when no temporal baseline was fitted. | +| `stale_baseline_horizon` | string | How far beyond the fitted window the row sits, so you can tell *just past the edge* from *far outside it*. `null` when no temporal baseline was fitted. | + +The nested `ai_explanation` struct (populated when AI explanations are on, which is the default): + +| Field | Type | Description | +|--------|------|-------------| +| `narrative` | string | Plain-language description of why the group was flagged. | +| `business_impact` | string | Likely downstream impact if the rows are processed unchanged. | +| `top_features` | string | Deterministic top-2 contributing features (e.g. `amount+quantity`), the group's pattern key. | +| `top_drivers` | string | The same drivers rendered for display, as human labels with their shares (e.g. `amount vs its group baseline (74%), quantity (12%)`). | +| `action` | string | What an analyst should investigate. | +| `group_size` | long | Number of anomalous rows in this anomaly group. | +| `group_avg_severity` | double | Mean `severity_percentile` across the group, floored the same way as `severity_percentile` above. | +| `evidence_scope` | string | How much of the contributing evidence the explanation was allowed to show: `complete` when nothing was withheld, `partial` when redacted columns held a minority of the group's evidence, `limited` when they held the majority. Deliberately coarse — a share would disclose by proportion what a redacted name discloses by identity. Filter or escalate on this rather than parsing the narrative. Only `redact_columns` produces anything other than `complete`. | + +**Access in PySpark:** use `F.element_at(F.col("_dq_info"), 1)` for the first element (1-based), then `.getField("anomaly").getField("severity_percentile")` etc. Alternatively `F.col("_dq_info").getItem(0)` for 0-based index (see [Troubleshooting](/docs/guide/row_anomaly_detection/troubleshooting) for Spark Connect friendly patterns). + ## Practical examples (non-technical) - A sudden surge of high-value orders in a region that usually has low spend. @@ -399,17 +788,66 @@ Use row anomaly detection when you want to catch unusual combinations across col ## Frequently Asked Questions +
+Q: I upgraded and my `segment_by` code stopped working. What do I do? + +`segment_by` has been removed. Replace it with `baseline_by`, which compares each metric against its own group's baseline on a single model instead of training one model per group, and retrain (models from earlier releases no longer load). Row anomaly detection was Experimental in earlier releases, which is what allowed this break, and it is now Beta. See [Upgrading and breaking changes](#upgrading-and-breaking-changes). +
+ +
+Q: `train()` says it got an unexpected keyword argument `expected_anomaly_rate`. What replaced it? + +Nothing, deliberately. It supplied the estimator's `contamination`, which places scikit-learn's own `predict` boundary and nothing else — DQX scores by ranking against training-score quantiles, so the parameter never changed which rows were flagged despite its name. + +To control how much gets flagged, set `threshold` on the check: it is an alert budget over training severity, so `threshold=95` flags the top 5%. To set `contamination` because you load the registered model and call `predict` yourself, pass `params=AnomalyParams(algorithm_config=IsolationForestConfig(contamination=...))`. Its default is unchanged, so removing the argument alone changes no behaviour. +
+ +
+Q: Are there anomalies DQX will not find? + +Yes, and they are worth knowing before you rely on it. + +**Trend, unless you ask for it.** DQX learns what normal looks like from a training window, so a metric +that grows steadily eventually sits outside that window and ordinary rows start being flagged. Pass +[`baseline_over_time`](#comparing-against-time) and the trend is fitted and subtracted, which also catches +the opposite case: a value that is ordinary against the whole training range and wrong for where the trend +had got to. Without it, model a quantity that does not trend — `orders_per_customer` rather than +`daily_orders` — and note that `drift_threshold` will not reliably warn you here, because a gradual trend +inflates the very spread it is measured against. + +**Cycles other than hourly, daily, weekly, or monthly.** Those four are handled automatically from a +datetime column, and `baseline_over_time` fits daily and weekly shapes where the window holds enough +complete cycles to identify one. A six-week promotional cycle, or a fiscal quarter that does not align to +calendar months, is not — pass the cycle as a column and use `baseline_by`, which then judges each row +against its own phase. + +**Forecasting.** DQX judges rows against learned normal. It does not predict the next value and compare. + +**A single metric on its own.** With nothing to compare it against, `profile="correlation"` can only ask +"is this value unusually high or low?" — which a simple range check already does, and +[`has_no_outliers`](/docs/reference/quality_checks) does better. Give it something to compare against and +that changes: include the timestamp and it can spot a value that is fine in general but wrong *for a +Saturday*; add `baseline_by` and it can spot one that is wrong *for that region*. Context is what it needs, +and a timestamp or a grouping column both count. + +**Anything you already have labels for.** Train a classifier — it will beat any unsupervised detector on +the pattern it was taught. Anomaly detection is for the problems you cannot describe in advance. + +For volume, freshness and row counts over time, Databricks Data Quality Monitoring is the tool built for +that job, and the two work well together. +
+
Q: How much training data do I really need? -See the **Training data requirements** tip under Quick start. In short: 1,000+ rows preferred; quality and coverage matter more than quantity. For segmented models, use at least 100+ rows per segment. Ensure training data includes all realistic values for categorical columns (regions, types, etc.). +See the **Training data requirements** tip under Quick start. In short: 1,000+ rows preferred; quality and coverage matter more than quantity. If you use `baseline_by`, each group needs enough rows for a stable baseline, and DQX will not auto-pick a grouping that leaves fewer than ~30 rows per group. Ensure training data includes all realistic values for categorical columns (regions, types, etc.); a categorical value absent from training is not scored reliably.
Q: How often should I retrain? **Retrain when**: -- Drift warnings appear (distribution changed). Enable `drift_threshold=3.0` to get warnings when retraining is needed +- Drift warnings appear (the input feature distribution changed). Enable `drift_threshold=3.0` to get warnings when retraining is needed - Business logic changes (new products, pricing, processes) - Seasonality shifts (quarterly/annual patterns) - Major data pipeline changes @@ -421,9 +859,20 @@ See the **Training data requirements** tip under Quick start. In short: 1,000+ r
-Q: Why is my auto-trained model segmented? +Q: Why did auto-training choose a grouping? + +When you let DQX pick the columns as well, it looks for a grouping too: columns that look like good dimensions (for example region or category) with low cardinality (2–50 distinct values), low null rate, and enough rows per group. That is intentional, because behaviour often does differ by group, and the user least likely to pass `baseline_by` explicitly is the one most likely to miss a contextual anomaly. A discovered grouping is used as `baseline_by`: each metric gains its deviation from that group's baseline, on a **single** model, so the discovery cannot turn into an hours-long run however many groups it finds. The grouping it chose is logged. To compare against the whole table instead, pass `baseline_by=[]`. + +Naming `columns` yourself changes this. Then DQX keeps the whole-table comparison and only *tells* you what it found: + +``` +WARNING ['region'] looks like a grouping (3 groups, ~200 rows/group), but metrics are being + compared against the whole table, so a value that is ordinary overall yet wrong for its + own group will not be flagged. Pass baseline_by=['region'] to compare each row against + its own group, or baseline_by=[] to keep the whole-table comparison and silence this. +``` -When you train without specifying `columns` or `segment_by`, DQX auto-discovers both. If your data has columns that look like good segment dimensions (for example region, category) — low cardinality (2–50 distinct values), low null rate, and enough rows per segment — DQX will train **one model per segment**. That is intentional: segmented models often fit better when behavior differs by segment. To get a single global model instead, pass `segment_by=[]` or provide explicit `columns` (and no segment columns) when calling `train()`. +Two reasons it advises rather than acts. Your explicit column list is a decision, and adding engineered features on top of it would change the model you asked for. And a discovered grouping depends on the data, so if a column's cardinality shifts between retrains the feature set would change with it and move every score, drifting the threshold you calibrated. The warning is silent when the data has no usable grouping, so it only speaks when there is something to do.
@@ -433,25 +882,40 @@ When you train without specifying `columns` or `segment_by`, DQX auto-discovers 1. Train model on historical batch data 2. Apply checks to streaming DataFrame -3. **Recommended**: Disable contributions (default): `enable_contributions=False` (SHAP is too slow for real-time processing) +3. **Recommended**: Disable contributions (default): `enable_contributions=False` (computing them is too slow for real-time processing)
Q: Can I use this with PII/sensitive data? -**Yes**, with considerations: +**Yes**, with considerations — and read the AI explanations point, because explanations are **on by +default** and they are the one part of this feature that sends a request anywhere. -**Safe**: -- Training happens in your Databricks environment (data never leaves). +**Your data stays on your compute**: +- Training and scoring both run on your own Databricks compute. No row values are sent anywhere by either. - Models are stored in Unity Catalog that you control. -- There are no external API calls. -**Consider**: -- SHAP contributions may expose sensitive patterns in explanations. +**AI explanations issue a request to a Model Serving endpoint** (`enable_ai_explanation=True` is the +default; set it to `False` to send nothing at all): +- What is sent, per anomaly group: **column names**, each column's contribution share, the group's row + count, its severity range, an ensemble-agreement band, your threshold and a drift summary. +- What is **not** sent: **no row values**. Not the anomalous rows, not the training data, not the group + values a baseline was computed over — `baseline_by` reaches the prompt as its column *names* only. +- The request goes to the endpoint named in `ai_explanation_llm_model_config` — by default a Databricks + Foundation Model endpoint in your own workspace. It is governed like any other endpoint you call, so + point it at one whose governance you accept. +- `redact_columns` keeps named columns out of the prompt entirely. Scope it correctly: it governs the + prompt only. `_dq_info[].anomaly.contributions` still lists every column, so anyone who can read the + scored table sees the full picture — it is not an access control. + +**Also consider**: +- Contributions and explanations can reveal which columns behave unusually together, which is a pattern + even when no value is disclosed. - Model metadata includes column names and statistics. -- Use column-level security for model registry if needed. +- Use column-level security on the model registry if needed. -**Recommendation**: Train on pseudonymized features or aggregate metrics when possible. +**Recommendation**: train on pseudonymized features or aggregate metrics where you can, and decide +deliberately whether AI explanations belong in your environment rather than inheriting the default.
@@ -466,7 +930,7 @@ When you train without specifying `columns` or `segment_by`, DQX auto-discovers **DQX Row Anomaly Detection**: - Row-level anomaly scores - Cross-column pattern detection -- Per-row explanations (SHAP contributions) +- Per-row explanations (feature contributions) - Can be applied together with rule-based checks **Use both**: Data Quality Monitoring for table health + DQX for row-level issues inside the data. diff --git a/docs/dqx/docs/guide/row_anomaly_detection/troubleshooting.mdx b/docs/dqx/docs/guide/row_anomaly_detection/troubleshooting.mdx index fb3710c8e..d7277baca 100644 --- a/docs/dqx/docs/guide/row_anomaly_detection/troubleshooting.mdx +++ b/docs/dqx/docs/guide/row_anomaly_detection/troubleshooting.mdx @@ -191,7 +191,7 @@ Sampling doesn't hurt model quality if the sample is representative! has_no_row_anomalies( model_name=model_name, registry_table=registry_table, - enable_contributions=False, # 10x faster, default is False + enable_contributions=False, # 10x faster; the default is True ) # Investigation: slower, with explanations diff --git a/docs/dqx/docs/installation.mdx b/docs/dqx/docs/installation.mdx index bee319b15..7903977ee 100644 --- a/docs/dqx/docs/installation.mdx +++ b/docs/dqx/docs/installation.mdx @@ -301,6 +301,9 @@ run_configs: # <- list of run configurations, each run co columns: [amount, quantity] # <- optional, omit to use all supported columns model_name: main.iot.orders_monitor # <- required when using anomaly config, fully qualified registry_table: main.iot.dqx_anomaly_models # <- required when using anomaly config, fully qualified + baseline_by: [region, product] # <- optional. Judge each metric against its own group's baseline + profile: correlation # <- optional. "tabular" (default) or "correlation" + baseline_over_time: event_ts # <- optional. Time column each metric's expected level is fitted along # for the full parameter anomaly specification, see the Row Anomaly Detection documentation # if wanting to store checks in lakebase table diff --git a/docs/dqx/docs/reference/feature_lifecycle.mdx b/docs/dqx/docs/reference/feature_lifecycle.mdx index 49ef38c9a..31a0b7525 100644 --- a/docs/dqx/docs/reference/feature_lifecycle.mdx +++ b/docs/dqx/docs/reference/feature_lifecycle.mdx @@ -18,6 +18,7 @@ Features **without** a badge are [generally available](#ga). | Badge | Stage | Summary | |---|---|---| | Experimental | [Experimental](#experimental) | For evaluation only; may change or be removed at any time. | +| Alpha | [Alpha](#alpha) | Taking shape and usable, but not yet feature-complete; expect breaking changes. | | Beta | [Beta](#beta) | Usable and feature-complete; the API may still change before GA. | | Deprecated | [Deprecated](#deprecated) | Still works but slated for removal; migrate to the replacement. | @@ -30,6 +31,18 @@ table formats may change — or the feature may be removed entirely — in any r or a migration path. There are no backward-compatibility or support guarantees. Not recommended for production workloads. +## Alpha {#alpha} + +The feature works and is ready for hands-on use and feedback, but it is **not yet feature-complete** and its +API, behavior, and on-disk or table formats can still change in backward-incompatible ways between releases. +Unlike an [experimental](#experimental) feature it is not expected to disappear, and it is being actively +developed toward [beta](#beta); unlike a beta feature, parts of it are still missing or rough, and some +defaults are still being calibrated on real workloads. + +Use it on non-critical workloads, pin your DQX version if you depend on current behavior, and read the +[changelog](https://github.com/databrickslabs/dqx/blob/v0.16.0/CHANGELOG.md) when upgrading. Feedback at this +stage is especially useful, because the API is still open to change. + ## Beta {#beta} The feature is feature-complete and usable, and is released for broader adoption and feedback ahead diff --git a/docs/dqx/docs/reference/quality_checks.mdx b/docs/dqx/docs/reference/quality_checks.mdx index a385a22c5..819bd4dbc 100644 --- a/docs/dqx/docs/reference/quality_checks.mdx +++ b/docs/dqx/docs/reference/quality_checks.mdx @@ -1978,7 +1978,7 @@ You can also define your own custom dataset-level checks (see [Creating custom c | `has_no_gaps_per_time_window` | Dataset check that flags gaps in a time series, i.e. time windows of a given size that contain no rows between windows that do. The violation is reported on the boundary row before each interior gap. | `column`: timestamp or date column (can be a string column name or a column expression); `window_minutes`: size of the time window in minutes that defines the expected data grain (for example 1440 for daily); `group_by`: optional list of columns or column expressions to detect gaps independently within each group; `trailing_gap`: (optional) if `true`, also flags the last present window (per group) when it ends more than one window before `curr_timestamp`, so missing recent data is caught at the tail of the series (defaults to `false`); `curr_timestamp`: (optional) current timestamp column used to anchor trailing-gap detection, only used when `trailing_gap` is `true` (if not provided, current_timestamp() function is used) | | `has_valid_schema` | Schema check that validates whether the DataFrame schema matches an expected schema. In non-strict mode, validates that all expected columns exist with compatible types (allows extra columns). In strict mode, validates exact schema match (same columns, same order, same types) for all columns by default or for all columns specified in `columns`. This check is applied at the dataset level and reports schema violations for all rows in the DataFrame when incompatibilities are detected. All columns in the `exclude_columns` list will be ignored even if the column is present in the `columns` list. | `expected_schema`: (optional) expected schema as a DDL string (e.g., "id INT, name STRING") or StructType object; `ref_df_name`: (optional) name of the reference DataFrame to load the schema from (dictionary of DataFrames can be passed when applying checks); `ref_table`: (optional) fully qualified reference table name to load the schema from (e.g. "catalog.schema.table"); exactly one of `expected_schema`, `ref_df_name`, or `ref_table` must be provided; `columns`: (optional) list of columns to validate (if not provided, all columns are considered); `strict`: (optional) whether to perform strict schema validation (default: False) - False: validates that all expected columns exist with compatible types, True: validates exact schema match; `exclude_columns`: (optional) list of columns to ignore during validation (if not provided, all columns are considered); | | `has_no_outliers` | Checks whether the values in the input column contain any outliers. This function implements a median absolute deviation (MAD) algorithm to find outliers. | `column`: column of type numeric to check (can be a string column name or a column expression); | -| `has_no_row_anomalies` | Flags rows that are anomalous according to a trained ML model. The model learns "normal" patterns from your training data; at check time each row is scored (severity percentile 0–100) and optionally enriched with SHAP contributions. Requires a model trained with the anomaly engine first. See [Row Anomaly Detection](#row-anomaly-detection) below for training, full parameters, and usage. | `model_name`: fully qualified model name (e.g. catalog.schema.model_name); `registry_table`: fully qualified registry table (e.g. catalog.schema.model_registry); `threshold`: (optional) severity percentile threshold (default 95); `drift_threshold`: (optional) warn when score distribution drifts from training (None = off); `enable_contributions`: (optional) add SHAP per-feature contributions to `_dq_info` (default True; set False to skip the SHAP cost); `enable_confidence_std`: (optional) add ensemble score std to `_dq_info` (default False); `enable_ai_explanation`: (optional) add an LLM-generated explanation to `_dq_info` (default True; degrades to null if contributions are off or no serving endpoint is reachable); `ai_explanation_llm_model_config`: (optional) Databricks Model Serving endpoint config for the explanation; `redact_columns`: (optional) feature/segment names to keep out of the LLM prompt; `max_groups`: (optional) cap on LLM calls per run (default 500). See [Row Anomaly Detection](/docs/reference/quality_checks#row-anomaly-detection) section for full parameter details. | +| `has_no_row_anomalies` | Flags rows that are anomalous according to a trained ML model. The model learns "normal" patterns from your training data; at check time each row is scored (severity percentile 0–100) and optionally enriched with per-column contributions. Requires a model trained with the anomaly engine first. See [Row Anomaly Detection](#row-anomaly-detection) below for training, full parameters, and usage. | `model_name`: fully qualified model name (e.g. catalog.schema.model_name); `registry_table`: fully qualified registry table (e.g. catalog.schema.model_registry); `threshold`: (optional) severity percentile threshold (default 95); `drift_threshold`: (optional) warn when the input feature distribution drifts from the training baseline (None = off, the default); `enable_contributions`: (optional) add per-column contributions to `_dq_info` (default True; set False to skip the attribution cost); `enable_confidence_std`: (optional) add ensemble score std to `_dq_info` (default False); `enable_ai_explanation`: (optional) add an LLM-generated explanation to `_dq_info` (default True; degrades to null if contributions are off or no serving endpoint is reachable); `ai_explanation_llm_model_config`: (optional) Databricks Model Serving endpoint config for the explanation; `redact_columns`: (optional) feature/segment names to keep out of the LLM prompt; `max_groups`: (optional) cap on LLM calls per run (default 500). See [Row Anomaly Detection](/docs/reference/quality_checks#row-anomaly-detection) section for full parameter details. | | `are_polygons_mutually_disjoint` | Checks whether the polygons in a geometry column are mutually disjoint. Polygons sharing an edge or boundary are considered intersecting. Nulls and invalid geometries are excluded from the check. Requires Databricks runtime 17.1 or above. | `column`: column to check (can be a string column name or a column expression), must contain polygon or multipolygon geometries | | `is_geo_contains` | Checks if the reference geometry contains each column geometry using `st_contains` with meter-level precision. A geometry A *contains* B when B lies entirely within the interior of A with no boundary points of B on the boundary of A. Points on the shared boundary are not considered contained — use `is_geo_covers` for boundary-inclusive checks. When a convert flag is set to `True`, `try_to_geometry` is applied to parse the input from any supported format (WKT, WKB, EWKT, EWKB). Null values are skipped. Requires Databricks runtime 17.1 or above. | `column`: column to check (can be a string column name or a column expression); `reference_geometry`: reference geometry as a literal WKT/WKB/EWKT/EWKB string or bytes value, or a `Column` expression (e.g. `F.col('col_name')`) — a plain string is always treated as a literal, not a column name; `convert_column`: when `True`, applies `try_to_geometry` to convert the column values to GEOMETRY (default `False`); `convert_reference_geometry`: when `True`, applies `try_to_geometry` to convert the reference geometry to GEOMETRY (default `False`) | | `is_geo_covers` | Checks if the reference geometry covers each column geometry. When `precise=True`, uses `st_covers` for exact computation — A *covers* B when every point of B lies within A, including boundary points. When `precise=False` (default), approximates coverage using H3 cell indexing: all hexagonal cells of the column geometry must exist in the H3 cells of the reference geometry. Edge membership is not supported by H3 — geometries near boundaries may be misclassified. Higher `resolution` values give finer precision at the cost of more cells. Null values are skipped; in approximate mode invalid (unparseable) geometries are also skipped rather than flagged — use `is_geometry` to flag invalid values. Requires Databricks runtime 17.1 or above. | `column`: column to check (can be a string column name or a column expression); `reference_geometry`: reference geometry as a literal WKT/WKB/EWKT/EWKB string or bytes value, or a `Column` expression (e.g. `F.col('col_name')`) — a plain string is always treated as a literal, not a column name (bytes/WKB only supported when `precise=True`); `precise`: when `True`, uses exact `st_covers`; when `False` (default), uses H3 approximation and requires `resolution`; `resolution`: H3 resolution integer (0–15) or a column — required when `precise=False`; higher values give finer precision at the cost of more cells; `convert_column`: when `True`, applies `try_to_geometry` to the column (only used in precise mode, default `False`); `convert_reference_geometry`: when `True`, applies `try_to_geometry` to the reference geometry (only used in precise mode, default `False`) | @@ -3474,7 +3474,7 @@ Using non-curated aggregate functions is supported with the following limitation Detects unusual rows by learning what "normal" looks like across multiple columns, then flags rows that deviate. Train on recent "good" data so the anomaly model learns typical ranges and combinations (e.g., amount and quantity together), not just single-column thresholds. -**Current algorithm**: IsolationForest (scikit-learn with Spark scoring, requires Spark >= 3.4). Future releases may add additional algorithms behind the same interface. +**Detectors**: two, selected by `profile` (see the parameter table below). `"tabular"` (the default) is IsolationForest (scikit-learn with Spark scoring, requires Spark >= 3.4), which judges a row by its own values. `"correlation"` is a covariance-based detector that judges a row by whether its metrics still relate to each other as they normally do; it needs no timestamp column and trains a single model rather than an ensemble, because it is deterministic. Both share the same feature engineering, registry, check function, contributions and AI explanations. For a conceptual overview (including how this complements Databricks data quality monitoring), see [Row Anomaly Detection](/docs/guide/row_anomaly_detection). @@ -3509,11 +3509,18 @@ run_configs: input_config: location: catalog.schema.orders anomaly_config: - columns: [amount, quantity] # optional; omit to use all supported columns + columns: [amount, quantity] # optional; omit to auto-discover model_name: catalog.schema.orders_monitor registry_table: catalog.schema.dqx_anomaly_models + baseline_by: [region, product] # optional; judge each metric against its own group's baseline + profile: tabular # optional; "tabular" (default) or "correlation" + baseline_over_time: event_ts # optional; time column each metric's level is fitted along ``` +Every key mirrors an argument of `anomaly_engine.train()` below, and omitting one is the same as omitting +that argument, so a scheduled retrain reproduces a hand-trained model exactly. `anomaly_config` is a field of +the run config, beside `input_config` rather than inside it. + ### Training Parameters The `anomaly_engine.train()` method accepts several parameters to tune model behavior, performance, and accuracy. @@ -3525,12 +3532,13 @@ Required arguments: `df`, `model_name`, `registry_table`. Optional parameters: | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `columns` | list[str] | None | Columns to use for row anomaly detection (auto-discovered if omitted) | -| `segment_by` | list[str] | None | Train separate models per segment. When both `columns` and `segment_by` are omitted, DQX may auto-discover segment columns (e.g. categorical, 2–50 distinct values) and train a segmented model. Use `segment_by=[]` to force a single global model. | +| `baseline_by` | list[str] | None | Columns identifying the group a row belongs to, so each metric is judged against its own group's baseline rather than against the whole table. One model whatever the group count. Auto-discovered when both `columns` and `baseline_by` are omitted; pass `baseline_by=[]` to suppress discovery and compare against the whole table. When you name `columns` but not `baseline_by`, the comparison stays pooled and a warning names the grouping to pass if the data looks grouped. | +| `baseline_over_time` | str | None | A timestamp or date column each metric is judged *along*, so a value is compared with what its own history says to expect at that point in time. Composes with `baseline_by`: with both set the expectation is fitted on the group-relative value, so one model still covers every group. Independent of `profile`. Never auto-discovered, and DQX warns rather than acting when the training window shows little structure over time. Not a forecaster, and it never reads the previous row. The named column must not also appear in `columns`. See [Comparing against time](/docs/guide/row_anomaly_detection#comparing-against-time). | +| `profile` | str | `"tabular"` | Which detector to train. `"tabular"` is Isolation Forest — the behaviour that predates this option. `"correlation"` selects a correlation-aware detector for multivariate metrics whose anomalies are broken *correlations* rather than extreme single values; it needs no timestamp column and trains a single model rather than an ensemble. There is no automatic option: DQX never changes the algorithm on your behalf, because the choice cannot be verified without labels. See [Choosing a profile](/docs/guide/row_anomaly_detection#choosing-a-profile). | | `exclude_columns` | list[str] | None | Columns to exclude from training (e.g., IDs, labels, ground truth) | -| `expected_anomaly_rate` | float | 0.02 | Expected fraction of anomalies in your data (0.02 = 2%). Sets model contamination parameter. | | `params` | AnomalyParams | None | Optional. Advanced tuning parameters. See sections below for details. | -**Validation**: Training parameters are validated before model fitting. Invalid values (for example `sample_fraction <= 0`, `train_ratio > 1`, or `expected_anomaly_rate > 0.5`) fail fast with `InvalidParameterError`. +**Validation**: Training parameters are validated before model fitting. Invalid values (for example `sample_fraction <= 0`, `train_ratio > 1`, or `algorithm_config.contamination > 0.5`) fail fast with `InvalidParameterError`. #### AnomalyParams (Advanced Tuning) @@ -3541,7 +3549,8 @@ Pass an `AnomalyParams` object to the `params` argument to customize training be | `sample_fraction` | float | 0.3 | Fraction of data to sample for training (30%). Reduce for faster training on large datasets. | | `max_rows` | int | 1,000,000 | Maximum rows to use for training. Caps memory usage for very large datasets. | | `train_ratio` | float | 0.8 | Train/validation split ratio (80% train, 20% validation). | -| `ensemble_size` | int or None | 3 | Number of models in ensemble. Set to None for single model. More models = more robust but slower training. | +| `ensemble_size` | int or None | 3 | Number of models in ensemble. Set to None for single model. More models = more robust but slower training. Ignored by `profile="correlation"`: that detector is deterministic, so every ensemble member would be an identical model and the reported spread would be exactly zero. `confidence_std` is therefore unavailable for it. | +| `baseline_by` | list[str] or None | None | Columns identifying the group a row belongs to, so each metric is judged against its own group's baseline rather than against the whole table. Adds one feature per metric — its signed log-ratio to that group's median — on a **single** model, so cost does not grow with the group count. Normally set by passing `baseline_by` to `train()`. See [Group-aware anomaly detection](/docs/guide/row_anomaly_detection#group-aware-anomaly-detection). | #### IsolationForestConfig (Algorithm Parameters) @@ -3549,7 +3558,7 @@ Pass an `IsolationForestConfig` object to `params.algorithm_config` to tune the | Parameter | Type | Default | Description | |-----------|------|---------|-------------| -| `contamination` | float | auto | Expected outlier proportion. Auto-set from `expected_anomaly_rate` if not specified. | +| `contamination` | float | 0.02 | Expected outlier proportion. Affects only the estimator's own `predict`/`offset_` boundary, which DQX's scoring path does not use, so it does **not** change which rows DQX flags — tune `threshold` on the check for that. Relevant if you load the registered model and call `predict` yourself. | | `num_trees` | int | 200 | Number of trees in the forest. More trees = better accuracy but slower training. | | `max_depth` | int | None | Maximum tree depth. Auto-calculated as log2(sample_size) if None. | | `subsampling_rate` | float | None | Per-tree subsampling rate. None uses sklearn defaults. | @@ -3596,17 +3605,21 @@ model_name = anomaly_engine.train( df=spark.table("catalog.schema.orders"), model_name="catalog.schema.orders_monitor", registry_table="catalog.schema.dqx_anomaly_models", - expected_anomaly_rate=0.05, # Expect 5% anomalies params=params, ) ``` -- **Too many false positives?** Increase `threshold` (95 → 98) in scoring, or increase `ensemble_size` (3 → 5) -- **Training too slow?** Decrease `sample_fraction`, `max_rows`, or `num_trees`; disable SHAP explainability with `enable_contributions=False` -- **Missing real anomalies?** Decrease `threshold` (95 → 90) in scoring, or increase `expected_anomaly_rate` +These separate the four costs, because they are paid at different times: **training** (sampling, trees), +**scoring** (per row), **contributions** (per anomalous row only) and **AI explanations** (per anomaly group). +A knob that reduces one usually does nothing for the others. + +- **Too many alerts?** Increase `threshold` (95 → 98) on the check. Raising `ensemble_size` (3 → 5) steadies the score by averaging more members, which can move borderline rows either way — it is not a false-positive fix, and it costs proportionally more training time +- **Training too slow?** Decrease `sample_fraction`, `max_rows`, or `num_trees`. Note `enable_contributions` is a *scoring* option and has no effect on training time +- **Scoring too slow?** Set `enable_contributions=False` to skip attribution (this also disables AI explanations), and `enable_ai_explanation=False` to skip the serving calls +- **Missing real anomalies?** Decrease `threshold` (95 → 90) on the check - **Reproducible results needed?** Set `random_seed` in `IsolationForestConfig` -- **High-cardinality categoricals slow?** Increase `categorical_cardinality_threshold` to use Frequency encoding +- **High-cardinality categoricals slow?** *Decrease* `categorical_cardinality_threshold`, so columns above it use Frequency encoding (one feature) instead of OneHot (one feature per category). Increasing it does the opposite and makes training slower. This changes what the model sees, so retrain and re-check quality rather than treating it as a free optimisation ### Define checks @@ -3651,18 +3664,18 @@ checks = [ - `registry_table`: Registry table (required, fully qualified Unity Catalog table name: `catalog.schema.table`). - `threshold`: Severity percentile threshold (0–100, default 95). - `row_filter`: Optional SQL expression to filter rows before scoring. -- `drift_threshold`: Optional float (e.g. 3.0) to enable drift detection; default None (disabled). When set, a warning is emitted if the scoring distribution at check time deviates from training. A value of 3.0 corresponds to roughly 3-sigma deviation from training statistics. See the section below for more details. -- `enable_contributions`: Include per-feature contributions in `_dq_info[0].anomaly.contributions` (default `True`). SHAP is computed only for anomalous rows (severity at or above the threshold), so the cost scales with the number of anomalies rather than the table size; non-anomalous rows get a `null` map. Set `False` to skip the SHAP computation entirely (which also disables AI explanations). See the section below and [Schema of _dq_info](/docs/guide/row_anomaly_detection#schema-of-the-info-column-_dq_info) for field details. +- `drift_threshold`: Optional float (e.g. 3.0) to enable drift detection; default None (disabled). When set, a warning is emitted if the distribution of the *input features* at check time deviates from the baseline statistics recorded at training; the warning names the drifted columns. It does not look at the anomaly scores themselves. Batches smaller than 1,000 rows are skipped, because per-column statistics are too noisy to compare at that size. A value of 3.0 corresponds to roughly 3-sigma deviation from training statistics. See the section below for more details. +- `enable_contributions`: Include per-feature contributions in `_dq_info[0].anomaly.contributions` (default `True`). Attribution runs only for anomalous rows (severity at or above the threshold), so the cost scales with the number of anomalies rather than the table size; non-anomalous rows get a `null` map. How it is computed depends on the detector — SHAP for `profile="tabular"`, an exact leave-one-out decomposition for `profile="correlation"`, which needs no SHAP at all — and the emitted map is identical either way. Set `False` to skip attribution entirely (which also disables AI explanations). See the section below and [Schema of _dq_info](/docs/guide/row_anomaly_detection#schema-of-the-info-column-_dq_info) for field details. - `enable_confidence_std`: Include `confidence_std` for ensembles (default `False`). Useful when using ensemble training. -- `enable_ai_explanation`: Add an LLM-generated plain-language explanation in `_dq_info[0].anomaly.ai_explanation` (default `True`). Uses the SHAP contributions as input — if `enable_contributions=False`, explanations are disabled with a warning (not an error). The LLM call runs inside Spark via the SQL `ai_query` function against a Databricks Model Serving endpoint — no extra dependency, but it requires Databricks serverless compute or Databricks Runtime 15.4 LTS or above (where `ai_query` is available). If `ai_query` is unavailable or the endpoint isn't reachable, explanations are skipped with a warning and scoring still completes. See the **AI Explanations** section below. +- `enable_ai_explanation`: Add an LLM-generated plain-language explanation in `_dq_info[0].anomaly.ai_explanation` (default `True`). Uses the column contributions as input — if `enable_contributions=False`, explanations are disabled with a warning (not an error). The LLM call runs inside Spark via the SQL `ai_query` function against a Databricks Model Serving endpoint — no extra dependency, but it requires Databricks serverless compute or Databricks Runtime 15.4 LTS or above (where `ai_query` is available). If `ai_query` is unavailable or the endpoint isn't reachable, explanations are skipped with a warning and scoring still completes. See the **AI Explanations** section below. - `ai_explanation_llm_model_config`: `LLMModelConfig` (or a dict with keys `model_name`, `api_key`, `api_base`) used by `enable_ai_explanation`. `model_name` must resolve to a Databricks Model Serving endpoint (the `databricks/` prefix, if present, is stripped). Defaults to `databricks/databricks-claude-sonnet-4-5`. -- `redact_columns`: Feature/column names to exclude from the LLM prompt (default `None`). Filters SHAP contribution keys, the top-2 pattern key, and matching segment keys (emitted as `key=`). -- `max_groups`: Maximum number of distinct `(segment, pattern)` groups the LLM is called for per scoring run (default `500`). Anomalous rows are bucketed by `(segment, pattern)` and the LLM is called once per group; groups beyond the cap — ranked by `group_size * group_avg_severity` — get a `null` explanation and a warning is logged. For segmented models the cap is split across eligible segments with a floor of one call each. +- `redact_columns`: Feature/column names to exclude from the LLM prompt (default `None`). Their contribution entries are dropped from what the model is shown, and they are excluded from the top-2 pattern key. Remaining shares are renormalised across what is left, so the explanation reports shares of the *disclosed* evidence and states when evidence was withheld (see `ai_explanation.evidence_scope`). Scope: this governs the prompt only — `_dq_info[].anomaly.contributions` still lists every column, so it is not an access control. +- `max_groups`: Maximum number of distinct contribution-pattern groups the LLM is called for per scoring run (default `500`). Anomalous rows are bucketed by their top-2 contributing columns and the LLM is called once per bucket, so this caps cost directly; groups beyond it — ranked by `group_size * group_avg_severity` — get a `null` explanation and a warning is logged. **Notes** - For workflow training, `model_name` and `registry_table` are required and must be fully qualified names. Both are stored in Unity Catalog. - Scoring uses the columns the model was trained on. -- Rows with nulls in anomaly columns are skipped (not flagged). +- Nulls do not by themselves exclude a row. A null in a modelled column is imputed and accompanied by a null-indicator feature, so the row is scored and the fact that the value was missing is part of what the model sees. Rows genuinely left unscored — `score` and `severity_percentile` both `null`, and never flagged — are those whose `baseline_by` group was absent from training (`is_new_baseline`), and those excluded by `row_filter`. - When `row_filter` is used, all original rows are preserved; non-filtered rows have `null` scores. - Row alignment is handled internally; no row-id parameters are required. @@ -3683,7 +3696,7 @@ ORDER BY training_time DESC **Feature Contributions (Explainability)** -Contributions are disabled by default for performance. Set `enable_contributions=True` to get per-feature SHAP contributions (adds significant cost): +Contributions are **on by default** (`enable_contributions=True`). They are computed only for anomalous rows, so the cost scales with the number of anomalies rather than the size of the table; non-anomalous rows carry a `null` map. Set `enable_contributions=False` to skip them, which also disables AI explanations: ```python from databricks.labs.dqx.rule import DQDatasetRule @@ -3724,9 +3737,9 @@ display( **AI Explanations** -AI explanations are **on by default** — each anomalous row gets an LLM-generated, plain-language explanation in `_dq_info[0].anomaly.ai_explanation`. The LLM call runs entirely inside Spark via the SQL `ai_query` function against a Databricks Model Serving endpoint — no extra Python dependency and it scales with the cluster. Set `enable_ai_explanation=False` to turn it off; setting `enable_contributions=False` also disables explanations (they use SHAP contributions as input). If the serving endpoint isn't reachable, explanations are skipped with a warning and scoring still completes. The explanation is AI-generated from the anomaly signal (feature names + SHAP + severity), not grounded in catalog metadata — treat business impact / action as a starting point. +AI explanations are **on by default** — each anomalous row gets an LLM-generated, plain-language explanation in `_dq_info[0].anomaly.ai_explanation`. The LLM call runs entirely inside Spark via the SQL `ai_query` function against a Databricks Model Serving endpoint — no extra Python dependency and it scales with the cluster. Set `enable_ai_explanation=False` to turn it off; setting `enable_contributions=False` also disables explanations (they use the column contributions as input). If the serving endpoint isn't reachable, explanations are skipped with a warning and scoring still completes. The explanation is AI-generated from the anomaly signal (column names + contribution shares + severity), not grounded in catalog metadata — treat business impact / action as a starting point. -Anomalous rows are bucketed by a deterministic `(segment, pattern)` key, where the pattern is the sorted top-2 contributing SHAP features. The LLM is called **once per group** and every row in the group shares the same narrative, so cost stays predictable on large datasets (bounded by `max_groups`). +Anomalous rows are bucketed by a deterministic pattern key: the sorted top-2 contributing columns. The LLM is called **once per group** and every row in the group shares the same narrative, so cost stays predictable on large datasets (bounded by `max_groups`). ```python from databricks.labs.dqx.rule import DQDatasetRule @@ -3767,8 +3780,8 @@ display( ``` -* The LLM is called once per `(segment, pattern)` group, capped by `max_groups` (default 500). For segmented models the cap is split across eligible segments with a floor of one call each, so when `max_groups` is below the eligible-segment count the effective cap is the segment count (a warning is logged). -* `redact_columns` keeps the listed feature and segment names out of the prompt. Segment **values** for non-redacted keys are sent verbatim — avoid segmenting on sensitive columns, or list them in `redact_columns`. +* The LLM is called once per contribution-pattern group, capped by `max_groups` (default 500), so the cap is the number of calls per scoring run. +* `redact_columns` keeps the listed column names out of the prompt. **No row values are sent at all** — the prompt carries column names, contribution shares, group size, severity range, confidence band, threshold and drift summary, and a `baseline_by` grouping reaches it as column *names* only, never the group values. **Drift Detection** diff --git a/docs/dqx/package.json b/docs/dqx/package.json index 8ab2b6ca5..97c7d0b13 100644 --- a/docs/dqx/package.json +++ b/docs/dqx/package.json @@ -38,6 +38,7 @@ "@docusaurus/module-type-aliases": "^3.8.1", "@docusaurus/tsconfig": "^3.8.1", "@docusaurus/types": "^3.8.1", + "@types/react-dom": "19.2.2", "@tailwindcss/typography": "^0.5.16", "autoprefixer": "^10.4.20", "postcss": "^8.5.1", diff --git a/docs/dqx/src/components/BananaDeck.test.ts b/docs/dqx/src/components/BananaDeck.test.ts new file mode 100644 index 000000000..078211ffa --- /dev/null +++ b/docs/dqx/src/components/BananaDeck.test.ts @@ -0,0 +1,115 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; +import { createElement } from 'react'; +import { renderToStaticMarkup } from 'react-dom/server'; + +import BananaDeck, { BananaSlide, ExplanationExample } from './BananaDeck'; + +const renderSlide = (index: number): string => + renderToStaticMarkup(createElement(BananaSlide, { index })); + +test('the deck renders without browser globals and exposes all 13 slides', () => { + const html = renderToStaticMarkup(createElement(BananaDeck)); + assert.equal((html.match(/