Skip to content

Pin silver input tables to fixed Delta versions for a report run #87

Description

@tombonfert

Problem

Report DataFrames are lazy. All silver inputs are read through MeasurementDB._read_table (src/impulse_query_engine/measurement_db.py:89) with a plain spark.read.table(...) (UC) or spark.read.format("delta").load(...) (path), with no version pinned. If an input table changes between when a run starts and when a lazy op materializes, different stages of the same run can read different snapshots, producing inconsistent results.

Why it matters

Within one run, the same input table can be read at several different moments as lazy operations trigger. If the table is updated in between (an append, overwrite, or compaction), earlier and later stages compute against different data, so the run's outputs no longer correspond to a single consistent view of the inputs. This is easy to hit when reports run while upstream ingestion is still writing. Pinning each input to one version at run start makes the whole run deterministic and removes this class of "input changed under a lazy op" bug.

Proposed approach

  1. At report start, resolve each configured silver table's current Delta version once.
  2. Have _read_table read with .option("versionAsOf", <version>) so every lazy op sees the same snapshot.
  3. Carry the resolved versions on the run context / MeasurementDBConfig so both the query engine and persistence use them.

Scope

  • Covers UC and path modes. debug mode (in-memory DataFrames) is already stable and exempt.
  • Non-Delta inputs / views can't be time-traveled: fall back gracefully (skip pinning, optionally warn).
  • Silver inputs only. Self-referential gold reads are out of scope.

Acceptance criteria

  • All silver reads in a run observe one snapshot, resolved once at start.
  • Works for UC and path tables, with a safe fallback when versioning is unavailable.
  • Test: mutate a silver table after a run starts but before a lazy op triggers; assert results reflect the pinned snapshot.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions