Skip to content

Fix: Orca Security Alerts Connector misses alerts that become eligible after creation - #1185

Open
bahdanb-orca wants to merge 8 commits into
chronicle:mainfrom
orcasecurity:pr/orca-connector-last-sync
Open

Fix: Orca Security Alerts Connector misses alerts that become eligible after creation#1185
bahdanb-orca wants to merge 8 commits into
chronicle:mainfrom
orcasecurity:pr/orca-connector-last-sync

Conversation

@bahdanb-orca

Copy link
Copy Markdown

Description

What problem does this PR solve?

The Orca Security Alerts Connector intermittently fails to ingest alerts. Two independent causes were found:

  1. Alerts that become eligible after creation are lost permanently. The connector advanced its watermark over CreatedAt. An alert whose Orca Score is populated after creation (for example by a score override) only starts matching the connector's Lowest Orca Score To Fetch filter later — by which time the watermark has already moved past its CreatedAt, so it never enters the fetch window again.

  2. The date_range filter is day-granular, so the watermark could never advance within a day. The range start is rounded up to the next UTC midnight (verified empirically: a start of 00:00:00.000 matches that day, 00:00:01 matches nothing until the next day). Combined with a server-side limit and no pagination, this made the connector behave like a once-a-day batch that silently dropped the tail of any day with more alerts than Max Alerts To Fetch.

How does this PR solve the problem?

  • connectors/AlertsConnector.py — the fetch position is now a last_sync cursor (the alert row's database write time), so an alert re-enters the window whenever its row becomes visible again. Late score population is handled by construction rather than by a heuristic buffer. A CreatedAt >= now - 3h bound preserves "new alerts only" semantics when resuming, so rewrites of older alerts (rescans, status changes) never re-enter the window. The first run is not restricted by that bound and honours Max Hours Backwards.
  • core/query_builder.py — both time filters now use the datetime-precise range operator with ISO 8601 values instead of the day-granular date_range, and results are ordered by last_sync when that cursor is in use.
  • connectors/AlertsConnector.pycursor pagination: full pages advance the cursor to the page's maximum last_sync; a full page confined to a single last_sync second pages deeper with start_at_index rather than skipping the remainder of that second; and the watermark advances over pages that turned out to be all duplicates, so progress no longer depends on finding new alerts.
  • core/datamodels.py, core/OrcaSecurityParser.py — parse last_sync, falling back to the creation time when the field is absent.
  • core/constants.py — the duplicate-suppression ID cache is raised to 10,000, and the connector logs when it overflows.
  • tests/ — 18 new tests covering the connector loop and the query/response contract.
  • pyproject.toml, uv.lock, release_notes.yaml — integration version bumped 15.0 to 16.0 with a matching release-notes entry.

Any other relevant information (e.g., design choices, tradeoffs, known issues):

  • Why not a "watermark minus one hour" buffer: the query applies its limit server-side before client-side ID de-duplication, so if a buffer window ever contains more already-seen alerts than the page size, every run returns the same first page, all of it is discarded as duplicates, and the watermark never advances — a permanent stall. Advancing the watermark across duplicate-only pages is what avoids this.
  • Why not offset-only pagination: order_by CreatedAt has no deterministic tiebreaker and the range end is re-evaluated per request, so offset pages can silently skip rows under concurrent inserts — the very failure being fixed here. Offsets are therefore used only to page within a single last_sync second, back-to-back inside one run, never carried across runs.
  • Known limitation — ordering within a tied second is not guaranteed (no secondary sort key). A row that shuffles out of view returns on its next last_sync rewrite, unless its CreatedAt has meanwhile aged out of the lookback window. This is documented in a code comment.
  • last_sync is not part of the schema published by /schema. The behaviour relied on here — that the field is filterable, orderable, returned in responses, wrapped in {"value": ...}, and timezone-aware at second resolution — was verified against a live tenant, and the test fixtures pin that shape so a change would fail the test suite rather than fail silently at runtime.
  • Upgrade behaviour: the previous version stored its position as a creation time and this version reads it as a write time, so the first run after upgrading may create a small number of duplicate cases. This is bounded by the 3-hour window and absorbed by the ID cache afterwards; it is called out in the release notes.
  • Second commit ("rename action JSON result examples"): JSON Result Example Validation expects resources/<ActionName>_JsonResult_example.json, matching the action's script file name, while these files used snake_case names. The mismatch is pre-existing, but it makes mp validate exit non-zero for this integration, so the validation job cannot go green without it. It is kept as a separate commit — happy to split it into its own PR if you prefer.
  • Verified locally and end-to-end:
    • mp validate integration orca_security — 23/23
    • mp test --integration orca_security — 18/18
    • mp build --integration orca_security — succeeds
    • Exercised on a Google SecOps SOAR instance against a live Orca tenant: first-run backfill, duplicate suppression with the watermark still advancing, start_at_index paging inside a tied last_sync second, recovery of an alert whose score was raised after the watermark had passed its CreatedAt, rejection of an updated alert created outside the lookback window, and exactly-once delivery across consecutive runs.

Checklist:

General Checks:

  • I have read and followed the project's contributing.md guide.
  • My code follows the project's coding style guidelines.
  • I have performed a self-review of my own code.
  • My changes do not introduce any new warnings.
  • My changes pass all existing tests.
  • I have added new tests where appropriate to cover my changes. (If applicable)
  • I have updated the documentation where necessary (e.g., README, API docs). (If applicable)

Open-Source Specific Checks:

  • My changes do not introduce any Personally Identifiable Information (PII) or sensitive customer data.
  • My changes do not expose any internal-only code examples, configurations, or URLs.
  • All code examples, comments, and messages are generic and suitable for a public repository.
  • I understand that any internal context or sensitive details related to this work are handled separately in internal systems (Buganizer for Google team members).

Screenshots (If Applicable)

Not applicable — no UI or visual changes.


Further Comments / Questions

  1. Should the JSON result example rename stay in this PR, or be split into a separate one?
  2. The 3-hour CreatedAt lookback is the window in which an alert can still be recovered after becoming eligible. Let us know if you would prefer a different default, or the value exposed as a connector parameter.

The JSON result example validation expects resources/<ActionName>_JsonResult_example.json,
matching the action's script file name. These files used snake_case names, so the
validation failed for every action with a JSON result.

Rename the six example files and update the result_example_path in each action
definition accordingly. No content changes.
Both window bounds now appear in the run log, so the fetch window can be
diagnosed from the logs alone instead of being inferred from which alerts
came back.
@bahdanb-orca
bahdanb-orca marked this pull request as ready for review August 26, 2026 10:11
@bahdanb-orca
bahdanb-orca requested a review from a team as a code owner August 26, 2026 10:11
…mary

Drops implementation detail from the release note per review feedback,
leaving a high-level description of the fix.
@bahdanb-orca

Copy link
Copy Markdown
Author

Hello @KrishnaSharma06

I've addressed your comment regarding notes, could you please check it out?

Comment on lines +178 to +179
- description: Orca Security - Alerts Connector - Fixed an issue where alerts becoming
eligible after creation were intermittently missed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Description has to be aligned in single line, please fix it.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, please check it out

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants