Skip to content

Convert TIMESTAMP channel timestamps to epoch numbers in Spark (follow-up to #110) #111

Open

Description

@tombonfert

When this matters

This only becomes relevant once #110 (TimeWindowEvent) has landed, and only for reports that use a TimeWindowEvent on a channels table whose timestamp columns are Spark TIMESTAMP (RLE tstart/tend, or RAW timestamp). Reports with numeric epoch timestamps are not affected.

Current behavior

The solver never casts TIMESTAMP channel columns. They reach the pandas solve UDF as datetime64[ns], holding the local clock time of the Spark session time zone. SampleSeries then applies np.array(..., dtype=np.float64), which implicitly turns them into nanoseconds since epoch, as float64.

Verified locally with the basic test data (Engine RPM, container 1), storing tstart/tend as TIMESTAMP:

Session time zone First sample time in the UDF Time-weighted mean Histogram bin duration
µs long (baseline) 1499929242072000.0 900.78599 48,279,000
TIMESTAMP, UTC 1.499929242072e18 900.78599 48,279,000,064
TIMESTAMP, Europe/Berlin 1.499936442072e18 (+2 h) 900.78599 48,279,000,064

Consequences:

  • Implicit unit. Sample times and durations are in nanoseconds without anyone choosing that.
  • Precision loss. Nanosecond epochs around 1.5e18 exceed 2^53, so float64 rounds them to about 256 ns (note the 64 ns error in the durations above).
  • Time-zone dependence. Absolute sample times shift by the session time zone offset. Durations survive, except across a daylight-saving change inside a measurement.

Why it matters for TimeWindowEvent

#110 converts TIMESTAMP container boundaries (container_metrics.start_ts/stop_ts) to epoch numbers in Spark via the opt-in solver_config.epoch_unit, using unix_micros, which gives absolute time regardless of time zone. Channel samples, however, stay in session-local clock time (implicit ns). With a non-UTC session time zone, the windows are therefore offset from the samples by the zone offset, and the per-window statistics cover the wrong slice of data. With a UTC session (the Databricks default) they happen to line up.

Proposal

When solver_config.epoch_unit is set, convert TIMESTAMP channel timestamp columns (tstart/tend in RLE mode, timestamp in RAW mode before encoding) to epoch numbers in that unit in Spark, at the point where the channels table is read. Use the same unix_micros-based conversion as the container boundaries (SolverConfig.normalize_container_boundaries).

  • Opt-in: with epoch_unit unset, behavior stays exactly as today, since existing gold values may rely on the implicit nanoseconds.
  • Unsupported types: reject TIMESTAMP_NTZ / DATE with a clear error, as for the container boundaries.
  • POI channels: consider poi_channels.timestamp too, which feeds tstart via a cast to the tstart type.

Acceptance criteria

  • With epoch_unit set, sample times in the UDF equal the numeric-epoch baseline in that unit, identically for UTC and a non-UTC session time zone.
  • A TimeWindowEvent with TIMESTAMP channel and container timestamps produces per-window statistics identical to the same data stored as numeric epochs.
  • With epoch_unit unset, results are unchanged (regression test against today's behavior).
  • Docs: configuration.md (epoch_unit) and silver_layer_schema.md describe TIMESTAMP channel timestamps.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions