[medium] Add a streamstats correlation method with a sliding time window

[elhoim]

BLUF

• Problem: correlation rules are converted to | bin _time span=<timespan> | stats ... by _time ..., so the timespan is a fixed bucket. Events that belong together but fall into two adjacent buckets are never correlated.
• Detection impact: a 4625 at 10:04:59 and a 4624 at 10:05:01 with timespan: 5m produce no alert, and neither do 10 events in 10 seconds around 10:45:00 with gte: 10. The Sigma correlation specification describes a sliding window and lists static time boundaries as a restriction that causes false negatives.
• Fix: add a second correlation method, streamstats (| sort 0 _time | streamstats time_window=<timespan> ...), for event_count, value_count, temporal and extended temporal. stats stays the default, so existing output does not change. Users opt in with correlation_method="streamstats" (sigma-cli: --correlation-method streamstats).
• Tests: 8 new conversion tests for the new method; all existing tests are unchanged. Both methods were validated on a live Splunk Enterprise 10.4.3 with a dataset that straddles a bucket boundary.

Priority: medium

Details

The specification (sigma-correlation-rules-specification.md, "Compatibility") lists "Temporal relationships are only recognized within static time boundaries ... This could cause false negatives" as a restriction that backends should warn about. The temporal section says: "The time frame should not be restricted to boundaries if this is not required by the given backend." Splunk does not require it. The backend already has a commented-out second method ("transaction": "... sliding time window"), so this adds that second method with streamstats, which is cheaper than transaction.

Generated SPL (temporal, group-by: [user], timespan: 5m):

| multisearch
[ search EventID=4625 | eval event_type="failed_logon" ]
[ search EventID=4624 | eval event_type="success_logon" ]

| sort 0 _time
| streamstats time_window=5m dc(event_type) as event_type_count by user

| search event_type_count >= 2

Behaviour of the new method:

• Every event gets the aggregate of the events of its group in the preceding timespan, and every event whose window matches the condition is returned. The stats method returns one row per bucket.
• streamstats time_window needs time-ordered input. A plain search returns events in descending order, which would also work, but the order of multisearch output is not guaranteed. The templates therefore always sort with | sort 0 _time, so one template works for all correlation types.
• The window is half-open like bin: with time_window=6m, two events exactly 6 minutes apart are not in the same window.
• streamstats limits a window to max_stream_window events (limits.conf, default 10000).
• Lower-bound conditions (gt, gte) and positive rule references are evaluated exactly. Upper-bound conditions and negated references in extended temporal conditions are evaluated on the window that ends at each event. This is documented in the README.
• The group-by clause is emitted as by <fields> only when group-by fields exist, so a correlation without group-by gives valid SPL.
• timespan: 1M maps to time_window=1mon, which Splunk accepts, as it does 1w, 1y and 1d.

Testing

New tests in tests/test_backend_splunk_correlations.py:

• test_correlation_methods_default_is_stats
• event_count with and without group-by
• value_count
• temporal
• extended temporal
• the fields context (values() output)
• a base rule with a deferred |re expression

Live validation on Splunk Enterprise 10.4.3 (container). The events were ingested through /services/receivers/simple, and the queries were converted by this branch with both methods, timespan: 5m:

Case	Data	stats	streamstats
temporal, user alice	4625 at 10:04:59, 4624 at 10:05:01	no match	match (event_type_count=2)
temporal, user bob	4625 at 10:20:30, 4624 at 10:21:00	match	match
temporal, user carol	4625 at 10:30:00, 4624 at 10:36:00	no match	no match
extended temporal (failed_logon and success_logon)	same	bob only	alice and bob
event_count gte: 10, host h2	10 events from 10:44:55 to 10:45:04	no match (5 + 5)	match (event_count=10)
value_count gte: 3, host h3	users x, y, z at 10:54:59, 10:55:00, 10:55:01	no match (1 + 2)	match (value_count=3)

The steps from .github/workflows/test.yml were run locally with poetry install (the locked pySigma 1.2.0):

• pytest --cov=sigma: 159 passed on Python 3.10, 3.11, 3.12 and 3.13.
• Coverage is 97.14%.

🤖 Generated with Claude Code