Feat/improve data model conversion

[scripton8]

Splunk backend: configurable data model (tstats) query settings

Branch: feat/improve-data-model-conversion

Problem

The data_model output format emitted a fixed tstats query. Everything after
count min(_time)... max(_time)... was hardcoded, so the accelerated query could not be
adapted to an environment without patching the backend or post-processing its output:

• summariesonly=false was hardcoded — no way to force summariesonly=true
  (accelerated-data-only) per environment.
• Only count was produced — no way to add values(...), dc(...), sum(...), etc.
• No time bucketing — no way to group over _time with a span=.
• Detection logic (Sigma rules) and environment-specific query tuning were entangled.

Fix

Added opt-in settings read from the pipeline processing state (set_state), plus a
summariesonly backend option. Defaults keep the generated query byte-for-byte
identical, so existing behaviour is unchanged. All inputs are validated (allow-listed
functions, regex-checked identifiers/spans) to prevent SPL injection.

Setting	Where	Purpose
summariesonly	backend option or set_state	toggles tstats summariesonly=...
tstats_span	set_state	adds _time to by + span=<value>
tstats_aggregations	set_state	structured extra aggregation functions after count
tstats_aggregation	set_state	raw aggregation string (advanced; wins over the list)

Examples

by-field lists abbreviated with ... for readability.

Default (unchanged)

| tstats summariesonly=false allow_old_summaries=true fillnull_value="null" count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes where Processes.process="test" by Processes.process ... | `drop_dm_object_name(Processes)` | ...

summariesonly

Backend option (also -O summariesonly=true with sigma-cli):

SplunkBackend(summariesonly=True)

Or via pipeline (pipeline state takes precedence over the backend option):

transformations:
  - id: set_summariesonly
    type: set_state
    key: summariesonly
    val: true

Result:

| tstats summariesonly=true allow_old_summaries=true ...

tstats_aggregations (structured, appended after count)

transformations:
  - id: set_aggregations
    type: set_state
    key: tstats_aggregations
    val:
      - func: values
        field: Processes.process_name
        as: process_names
      - func: sum
        field: Processes.count      # `as`/`alias` optional

Result:

| tstats ... count min(_time) as firstTime max(_time) as lastTime values(Processes.process_name) as process_names sum(Processes.count) from datamodel=Endpoint.Processes where ...

tstats_aggregation (raw string, advanced — overrides the structured list)

  - id: set_aggregation
    type: set_state
    key: tstats_aggregation
    val: values(Processes.user) as users

Result:

| tstats ... max(_time) as lastTime values(Processes.user) as users from datamodel=Endpoint.Processes where ...

tstats_span (time bucketing)

  - id: set_span
    type: set_state
    key: tstats_span
    val: 1h

Result (_time added to by, span= appended):

| tstats ... from datamodel=Endpoint.Processes where Processes.process="test" by _time Processes.process ... span=1h | `drop_dm_object_name(Processes)` | ...

Validation & safety

• summariesonly: accepts a bool or a string (true/false/yes/no/1/0);
  anything else raises SigmaFeatureNotSupportedByBackendError.
• tstats_aggregations: func must be in an allow-list of Splunk stats functions
  (avg, dc, sum, values, min, max, latest, per_hour, ...); count is
  intentionally excluded (always present and cannot be redefined); field and alias
  must match ^[\w.]+$; a missing field raises.
• tstats_span: must match ^\d+(?:s|sec|secs|second|seconds|m|min|mins|minute|minutes|h|hr|hrs|hour|hours|d|day|days|mon|mont|months)?$.

Closes #75

@thomaspatzke not sure if this is a solution you like, but I give it a try^^

[thomaspatzke]

Nice one, I totally like this 😉