Repository navigation
Convert TIMESTAMP channel timestamps to epoch numbers in Spark (follow-up to #110) #111
Description
When this matters
This only becomes relevant once #110 (TimeWindowEvent) has landed, and only for reports that use a TimeWindowEvent on a channels table whose timestamp columns are Spark TIMESTAMP (RLE tstart/tend, or RAW timestamp). Reports with numeric epoch timestamps are not affected.
Current behavior
The solver never casts TIMESTAMP channel columns. They reach the pandas solve UDF as datetime64[ns], holding the local clock time of the Spark session time zone. SampleSeries then applies np.array(..., dtype=np.float64), which implicitly turns them into nanoseconds since epoch, as float64.
Verified locally with the basic test data (Engine RPM, container 1), storing tstart/tend as TIMESTAMP:
| Session time zone | First sample time in the UDF | Time-weighted mean | Histogram bin duration |
|---|---|---|---|
| µs long (baseline) | 1499929242072000.0 |
900.78599 | 48,279,000 |
TIMESTAMP, UTC |
1.499929242072e18 |
900.78599 | 48,279,000,064 |
TIMESTAMP, Europe/Berlin |
1.499936442072e18 (+2 h) |
900.78599 | 48,279,000,064 |
Consequences:
- Implicit unit. Sample times and durations are in nanoseconds without anyone choosing that.
- Precision loss. Nanosecond epochs around 1.5e18 exceed 2^53, so float64 rounds them to about 256 ns (note the 64 ns error in the durations above).
- Time-zone dependence. Absolute sample times shift by the session time zone offset. Durations survive, except across a daylight-saving change inside a measurement.
Why it matters for TimeWindowEvent
#110 converts TIMESTAMP container boundaries (container_metrics.start_ts/stop_ts) to epoch numbers in Spark via the opt-in solver_config.epoch_unit, using unix_micros, which gives absolute time regardless of time zone. Channel samples, however, stay in session-local clock time (implicit ns). With a non-UTC session time zone, the windows are therefore offset from the samples by the zone offset, and the per-window statistics cover the wrong slice of data. With a UTC session (the Databricks default) they happen to line up.
Proposal
When solver_config.epoch_unit is set, convert TIMESTAMP channel timestamp columns (tstart/tend in RLE mode, timestamp in RAW mode before encoding) to epoch numbers in that unit in Spark, at the point where the channels table is read. Use the same unix_micros-based conversion as the container boundaries (SolverConfig.normalize_container_boundaries).
- Opt-in: with
epoch_unitunset, behavior stays exactly as today, since existing gold values may rely on the implicit nanoseconds. - Unsupported types: reject
TIMESTAMP_NTZ/DATEwith a clear error, as for the container boundaries. - POI channels: consider
poi_channels.timestamptoo, which feedststartvia a cast to thetstarttype.
Acceptance criteria
- With
epoch_unitset, sample times in the UDF equal the numeric-epoch baseline in that unit, identically forUTCand a non-UTC session time zone. - A
TimeWindowEventwithTIMESTAMPchannel and container timestamps produces per-window statistics identical to the same data stored as numeric epochs. - With
epoch_unitunset, results are unchanged (regression test against today's behavior). - Docs:
configuration.md(epoch_unit) andsilver_layer_schema.mddescribeTIMESTAMPchannel timestamps.
Activity