Problem
Report DataFrames are lazy. All silver inputs are read through MeasurementDB._read_table (src/impulse_query_engine/measurement_db.py:89) with a plain spark.read.table(...) (UC) or spark.read.format("delta").load(...) (path), with no version pinned. If an input table changes between when a run starts and when a lazy op materializes, different stages of the same run can read different snapshots, producing inconsistent results.
Why it matters
Within one run, the same input table can be read at several different moments as lazy operations trigger. If the table is updated in between (an append, overwrite, or compaction), earlier and later stages compute against different data, so the run's outputs no longer correspond to a single consistent view of the inputs. This is easy to hit when reports run while upstream ingestion is still writing. Pinning each input to one version at run start makes the whole run deterministic and removes this class of "input changed under a lazy op" bug.
Proposed approach
- At report start, resolve each configured silver table's current Delta version once.
- Have
_read_table read with .option("versionAsOf", <version>) so every lazy op sees the same snapshot.
- Carry the resolved versions on the run context /
MeasurementDBConfig so both the query engine and persistence use them.
Scope
- Covers UC and path modes.
debug mode (in-memory DataFrames) is already stable and exempt.
- Non-Delta inputs / views can't be time-traveled: fall back gracefully (skip pinning, optionally warn).
- Silver inputs only. Self-referential gold reads are out of scope.
Acceptance criteria
- All silver reads in a run observe one snapshot, resolved once at start.
- Works for UC and path tables, with a safe fallback when versioning is unavailable.
- Test: mutate a silver table after a run starts but before a lazy op triggers; assert results reflect the pinned snapshot.
Problem
Report DataFrames are lazy. All silver inputs are read through
MeasurementDB._read_table(src/impulse_query_engine/measurement_db.py:89) with a plainspark.read.table(...)(UC) orspark.read.format("delta").load(...)(path), with no version pinned. If an input table changes between when a run starts and when a lazy op materializes, different stages of the same run can read different snapshots, producing inconsistent results.Why it matters
Within one run, the same input table can be read at several different moments as lazy operations trigger. If the table is updated in between (an append, overwrite, or compaction), earlier and later stages compute against different data, so the run's outputs no longer correspond to a single consistent view of the inputs. This is easy to hit when reports run while upstream ingestion is still writing. Pinning each input to one version at run start makes the whole run deterministic and removes this class of "input changed under a lazy op" bug.
Proposed approach
_read_tableread with.option("versionAsOf", <version>)so every lazy op sees the same snapshot.MeasurementDBConfigso both the query engine and persistence use them.Scope
debugmode (in-memory DataFrames) is already stable and exempt.Acceptance criteria