diff --git a/AGENTS.md b/AGENTS.md index 288c8de..1edfcce 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,7 +10,15 @@ It should track the code in `main.py`, not stale assumptions from earlier iterat - This is an OpenCTI external-import connector for Double Extortion Platform (DEP) announcements. - The connector authenticates against DEP AWS Cognito, fetches announcement records from the DEP REST API, converts them to STIX 2.1, and sends bundles to OpenCTI with `update=True`. - The connector scope is `report,incident,identity,indicator`. -- The implementation is split across the `dep_connector/` package (`converter_to_stix.py`, `client_api.py`, `config_loader.py`, `connector.py`) with `main.py` as the thin entrypoint. +- The implementation is split across the `dep_connector/` package: + - `connector.py`: run-cycle orchestration and OpenCTI state handling + - `client_api.py`: DEP authentication and fetch HTTP calls + - `api_models.py`: DEP API/auth response validation models + - `datasets.py`: DEP dataset codes, aliases, and validation helpers + - `converter_to_stix.py`: DEP record parsing and STIX object construction + - `stix_objects.py`: shared STIX object protocols and collection helpers + - `config_loader.py`: YAML configuration loading + - `main.py`: thin entrypoint ## Runtime and configuration truths @@ -19,8 +27,8 @@ It should track the code in `main.py`, not stale assumptions from earlier iterat - `DEP_CLIENT_ID` is required at startup even though `config.yml.sample` leaves it blank. Missing it raises `ValueError`. - The runtime loop is infinite: `run()` executes one cycle, then sleeps for `CONNECTOR_RUN_INTERVAL`. - Local Docker Compose mounts `./config.yml` into `/app/config.yml` for the `dep-connector` service. -- The local stack pins OpenCTI services to `6.8.13`; the connector manifest declares support for OpenCTI `>= 6.8.13`. -- The container image runs `python main.py` as the non-root `app` user on Python 3.12. +- The local stack pins OpenCTI services to `6.9.0` (tested against latest); the connector manifest declares support for OpenCTI `>= 6.8.13` (no 6.9.0-only platform API is used). +- The container image runs `python main.py` as the non-root `app` user on Python 3.14.5. ## DEP fetch behavior @@ -48,6 +56,9 @@ It should track the code in `main.py`, not stale assumptions from earlier iterat - `ddos` -> `dds` - `forum` -> `frm` - The DEP API accepts one `dset` value per request, so the connector loops over configured datasets and issues one request per dataset. +- The DEP API caps each response at **1000 items** and exposes no pagination/offset/cursor parameter on this endpoint (empirically: a 1-day window returns ~13 items, 30 days ~654, and both 365-day and 1000-day windows return exactly 1000). `fetch_raw` returns whatever the API sends with no cap detection. +- Consequence: if a single per-dataset run window contains more than 1000 announcements, the excess is silently dropped, and because per-dataset state advances to the window `end` regardless, the dropped items are never re-fetched (permanent loss for that window). Risk scenarios: a large first-run `DEP_LOOKBACK_DAYS`, catch-up after long downtime, or a busy dataset over a wide window. +- Mitigation until window-chunking is implemented: keep each run window comfortably under 1000 items per dataset (small `DEP_LOOKBACK_DAYS`, frequent `CONNECTOR_RUN_INTERVAL`); be cautious with large backfills. A proper fix would split the `ts`/`te` window into smaller sub-windows whenever a response returns exactly 1000 items. ## State management @@ -66,8 +77,9 @@ It should track the code in `main.py`, not stale assumptions from earlier iterat - `annLink` is repaired for a known scrape bug: - `https//...` -> `https://...` - `http//...` -> `http://...` -- `site` and `victimDomain` are stripped; empty strings become `None`. +- `site`, `victimDomain`, and `naics` are stripped; empty strings become `None`. - `sector`, `actor`, and `country` are whitespace-normalized; empty strings, `n/a`, and `none` become `None`. +- `country_code` (DEP `victimCC`) is upper-cased and kept only when it is a 2-letter alpha ISO 3166-1 code; otherwise `None`. - Indicator domain extraction prefers `victimDomain`, then falls back to `site`. - Domain normalization uses `urlsplit`, extracts the hostname, and lowercases it. - `annDescription` is URL-decoded with `urllib.parse.unquote` before the report or incident is created. @@ -127,6 +139,7 @@ It should track the code in `main.py`, not stale assumptions from earlier iterat - Report custom properties (when present): - `dep_actor` - `dep_country` + - `dep_naics` - Report labels always include `DigIntLab`, plus any applicable: - `dep:announcement-type:` - `dep:dataset:` @@ -150,6 +163,7 @@ It should track the code in `main.py`, not stale assumptions from earlier iterat - `first_seen` - `dep_actor` when present - `dep_country` when present + - `dep_naics` when present - Incident labels always include `DigIntLab`, plus any applicable: - `dep:announcement-type:` - `dep:dataset:` @@ -197,9 +211,11 @@ It should track the code in `main.py`, not stale assumptions from earlier iterat - victim identity exists - Deterministic country location ID: - `location--uuid5(NAMESPACE_URL, "dep-country:")` -- Always set both: - - `name=` - - `country=` +- Always set: + - `name=` (human-readable country name) + - `country=`, falling back to the + country name when no valid code is available (STIX 2.1 requires `country`, + region, or lat/long, so the field is never omitted) - Preserve the OpenCTI-specific custom property: - `x_opencti_location_type: Country` @@ -319,8 +335,11 @@ Current automated coverage focuses on: ## File map - Connector entrypoint: `main.py` +- DEP API/auth response validation models: `dep_connector/api_models.py` - Data models and STIX converter: `dep_connector/converter_to_stix.py` - DEP API client (auth + fetch): `dep_connector/client_api.py` +- DEP dataset codes and aliases: `dep_connector/datasets.py` +- STIX object typing helpers: `dep_connector/stix_objects.py` - Configuration loader: `dep_connector/config_loader.py` - Connector orchestration (run cycle): `dep_connector/connector.py` - Package re-export: `dep_connector/__init__.py` diff --git a/Dockerfile b/Dockerfile index 88505c1..8a106f8 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,10 +1,9 @@ -FROM python:3.12-slim AS builder +FROM python:3.14.5-slim AS builder HEALTHCHECK NONE ENV UV_LINK_MODE=copy \ UV_COMPILE_BYTECODE=1 \ UV_PYTHON_DOWNLOADS=never \ - UV_PYTHON=python3.12 \ UV_NO_PROGRESS=1 COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv @@ -22,7 +21,7 @@ COPY dep_connector/ dep_connector/ RUN --mount=type=cache,target=/root/.cache/uv \ uv sync --locked --no-editable --no-dev -FROM python:3.12-slim AS runtime +FROM python:3.14.5-slim AS runtime HEALTHCHECK NONE ENV PATH="/app/.venv/bin:${PATH}" @@ -37,4 +36,4 @@ LABEL org.opencontainers.image.source=https://github.com/DigintLab/opencti-conne LABEL org.opencontainers.image.description="The Double Extortion connector ingests ransomware and data leak announcements published on the DoubleExtortion platform and converts them into STIX entities inside OpenCTI." USER app -ENTRYPOINT ["python", "main.py"] \ No newline at end of file +ENTRYPOINT ["python", "main.py"] diff --git a/README.md b/README.md index 5acabef..d0d3cec 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,8 @@ Dataset aliases are normalized to the short API codes before requests are sent. The connector adds `extended=true` only when `DEP_EXTENDED_RESULTS=true`. +> **Limitation — 1000-item response cap.** The DEP API returns at most **1000 items per request** and offers no pagination on this endpoint. If a single per-dataset run window contains more than 1000 announcements, the surplus is silently dropped and is **not** re-fetched on the next run (state advances regardless). Keep each run window under that cap by using a modest `DEP_LOOKBACK_DAYS` and a frequent enough `CONNECTOR_RUN_INTERVAL`, and take care with large first-run backfills or long downtime on busy datasets. As a rough guide observed from the API: ~13 items/day, ~654 over 30 days, and the cap (1000) is reached somewhere beyond a ~2-month window for the `ext` dataset. + ## Why `IntrusionSet` for DEP actor values DEP `actor` values are modeled as STIX `IntrusionSet` objects instead of `ThreatActor` by default. @@ -130,7 +132,8 @@ docker run --rm \ - The API occasionally URL-encodes announcement descriptions. The connector automatically decodes the description before sending it to OpenCTI. - DEP `annLink` values are repaired for a known scrape bug (`https//...` or `http//...`) before they are used as external references. - DEP actor and country values can be materialized as entities using `DEP_CREATE_INTRUSION_SETS` and `DEP_CREATE_COUNTRY_LOCATIONS`. -- DEP actor and country values are also stored in the primary object custom properties (`dep_actor`, `dep_country`) for source traceability. +- Country locations carry the human-readable country name plus the ISO 3166-1 alpha-2 code from DEP `victimCC` (falling back to the name when no valid code is available), as expected by OpenCTI/STIX. +- DEP actor, country, and NAICS industry code are also stored in the primary object custom properties (`dep_actor`, `dep_country`, `dep_naics`) for source traceability. - Generated indicators are also linked to the victim with `related-to` so those indicator nodes are connected in the Knowledge Graph. - Cross-entity links are automatic: intrusion set -> sector (`targets`), intrusion set -> country (`targets`), and sector -> country (`related-to`) when both entities are present. - Generic low-quality actor values (for example `unknown`, `anonymous`, `ransomware group`) are ignored for intrusion-set creation. diff --git a/config.yml.sample b/config.yml.sample index 06ff0ca..6194adb 100644 --- a/config.yml.sample +++ b/config.yml.sample @@ -25,6 +25,7 @@ dep: api_endpoint: https://api.eu-ep1.doubleextortion.com/v1/dbtr/privlist datasets: - ext + - dds lookback_days: 7 overlap_hours: 72 extended_results: true # Adds extended=true to DEP API requests. diff --git a/dep_connector/__init__.py b/dep_connector/__init__.py index 075b3ea..449df14 100644 --- a/dep_connector/__init__.py +++ b/dep_connector/__init__.py @@ -1,6 +1,6 @@ -from dep_connector.client_api import DepDataset from dep_connector.connector import DepConnector from dep_connector.converter_to_stix import LeakRecord, PrimaryObject, StixBuilder +from dep_connector.datasets import DepDataset __all__ = [ "DepConnector", diff --git a/dep_connector/api_models.py b/dep_connector/api_models.py new file mode 100644 index 0000000..fdc0d52 --- /dev/null +++ b/dep_connector/api_models.py @@ -0,0 +1,41 @@ +from typing import TypeAlias + +from pydantic import ( + BaseModel, + ConfigDict, + Field, + JsonValue, + StrictStr, + TypeAdapter, + field_validator, +) + +DepApiItem: TypeAlias = dict[str, JsonValue] + + +class CognitoAuthenticationResult(BaseModel): + model_config = ConfigDict(extra="ignore", populate_by_name=True) + + id_token: StrictStr = Field(alias="IdToken") + + @field_validator("id_token") + @classmethod + def require_non_empty_token(cls, value: str) -> str: + if not value: + error = "Unable to retrieve IdToken from authentication response" + raise ValueError(error) + return value + + +class CognitoAuthResponse(BaseModel): + model_config = ConfigDict(extra="ignore", populate_by_name=True) + + authentication_result: CognitoAuthenticationResult = Field( + alias="AuthenticationResult" + ) + + +COGNITO_AUTH_RESPONSE_ADAPTER: TypeAdapter[CognitoAuthResponse] = TypeAdapter( + CognitoAuthResponse +) +DEP_API_ITEMS_ADAPTER: TypeAdapter[list[DepApiItem]] = TypeAdapter(list[DepApiItem]) diff --git a/dep_connector/client_api.py b/dep_connector/client_api.py index fcc2a77..14db970 100644 --- a/dep_connector/client_api.py +++ b/dep_connector/client_api.py @@ -1,49 +1,38 @@ import json import logging -from enum import StrEnum -from typing import TypeAlias +from typing import TypeVar import requests +from pydantic import TypeAdapter, ValidationError -logger = logging.getLogger(__name__) - -JsonPrimitive: TypeAlias = str | int | float | bool | None -JsonValue: TypeAlias = JsonPrimitive | list["JsonValue"] | dict[str, "JsonValue"] -DepApiItem: TypeAlias = dict[str, JsonValue] - - -class DepDataset(StrEnum): - EXTORTION = "ext" - PRIVACY = "prv" - OPENNEWS = "nws" - VANDALISM = "vnd" - DDOS = "dds" - FORUM = "frm" +from dep_connector.api_models import ( + COGNITO_AUTH_RESPONSE_ADAPTER, + DEP_API_ITEMS_ADAPTER, + DepApiItem, +) +from dep_connector.datasets import DepDataset - @classmethod - def _missing_(cls, value: object) -> "DepDataset | None": - if not isinstance(value, str): - return None - return DATASET_ALIASES.get(value) +logger = logging.getLogger(__name__) +TValidatedPayload = TypeVar("TValidatedPayload") -DATASET_ALIASES: dict[str, DepDataset] = { - "extortion": DepDataset.EXTORTION, - "privacy": DepDataset.PRIVACY, - "opennews": DepDataset.OPENNEWS, - "news": DepDataset.OPENNEWS, - "vandalism": DepDataset.VANDALISM, - "ddos": DepDataset.DDOS, - "forum": DepDataset.FORUM, -} +def _decode_json_response(response: requests.Response, error_message: str) -> object: + try: + payload: object = response.json() + except (json.JSONDecodeError, requests.exceptions.JSONDecodeError) as exception: + raise ValueError(error_message) from exception + return payload -def dataset_alias_summary() -> str: - aliases_by_dataset: dict[DepDataset, list[str]] = {} - for alias, dataset in DATASET_ALIASES.items(): - aliases_by_dataset.setdefault(dataset, []).append(alias) - groups = ["/".join(aliases_by_dataset[dataset]) for dataset in DepDataset] - return ", ".join(group for group in groups if group) +def _validate_payload( + adapter: TypeAdapter[TValidatedPayload], + payload: object, + error_message: str, +) -> TValidatedPayload: + try: + return adapter.validate_python(payload) + except ValidationError as exception: + raise ValueError(error_message) from exception class DepClient: @@ -83,12 +72,16 @@ def authenticate(self) -> str: timeout=30, ) response.raise_for_status() - auth_payload: dict[str, dict[str, str]] = response.json() - token = auth_payload["AuthenticationResult"]["IdToken"] - if not token: - error = "Unable to retrieve IdToken from authentication response" - raise ValueError(error) - return token + auth_payload = _decode_json_response( + response, + "Unable to decode DEP authentication response", + ) + auth_response = _validate_payload( + COGNITO_AUTH_RESPONSE_ADAPTER, + auth_payload, + "Invalid DEP authentication response", + ) + return auth_response.authentication_result.id_token def fetch_raw( self, @@ -120,9 +113,7 @@ def fetch_raw( timeout=60, ) response.raise_for_status() - try: - payload: list[DepApiItem] = response.json() - except json.JSONDecodeError as exception: - message = "Unable to decode DEP API response" - raise ValueError(message) from exception - return payload + payload = _decode_json_response(response, "Unable to decode DEP API response") + return _validate_payload( + DEP_API_ITEMS_ADAPTER, payload, "Invalid DEP API response" + ) diff --git a/dep_connector/connector.py b/dep_connector/connector.py index e5d240d..1a27ea5 100644 --- a/dep_connector/connector.py +++ b/dep_connector/connector.py @@ -6,9 +6,11 @@ from stix2 import TLP_AMBER # type: ignore[import-untyped] from stix2 import v21 as stix2 -from dep_connector.client_api import DepClient, DepDataset, dataset_alias_summary +from dep_connector.client_api import DepClient from dep_connector.config_loader import load_config from dep_connector.converter_to_stix import LeakRecord, PrimaryObject, StixBuilder +from dep_connector.datasets import DepDataset, dataset_alias_summary +from dep_connector.stix_objects import StixObject, dedupe_by_stix_id class DepConnector: @@ -144,6 +146,12 @@ def _build_client(self, config: dict[str, object]) -> DepClient: if not client_id: error = "DEP client ID must be provided via configuration" raise ValueError(error) + api_key = pycti.get_config_variable( + "DEP_API_KEY", ["dep", "api_key"], config, default="" + ) + if not api_key: + error = "DEP API key must be provided via configuration" + raise ValueError(error) return DepClient( login_endpoint=str( pycti.get_config_variable( @@ -161,9 +169,7 @@ def _build_client(self, config: dict[str, object]) -> DepClient: default="https://api.eu-ep1.doubleextortion.com/v1/dbtr/privlist", ) ), - api_key=pycti.get_config_variable( - "DEP_API_KEY", ["dep", "api_key"], config - ), + api_key=str(api_key), username=pycti.get_config_variable( "DEP_USERNAME", ["dep", "username"], config ), @@ -284,8 +290,8 @@ def _build_cross_entity_relationships( intrusion_set: stix2.IntrusionSet | None, sector_identity: stix2.Identity | None, country_location: stix2.Location | None, - ) -> list[stix2._STIXBase21]: - relationships: list[stix2._STIXBase21] = [] + ) -> list[StixObject]: + relationships: list[StixObject] = [] if intrusion_set and sector_identity: relationships.append( self.stix.build_relationship( @@ -311,8 +317,8 @@ def _build_optional_entities( item: LeakRecord, victim: stix2.Identity | None, incident_id: str | None = None, - ) -> list[stix2._STIXBase21]: - objects: list[stix2._STIXBase21] = [] + ) -> list[StixObject]: + objects: list[StixObject] = [] sector_identity: stix2.Identity | None = None if self.create_sector_identities and item.sector and victim: sector_identity = self.stix.create_sector_identity(item.sector, item) @@ -361,8 +367,8 @@ def _build_content( victim: stix2.Identity | None, indicators: list[stix2.Indicator], incident_id: str | None = None, - ) -> list[stix2._STIXBase21]: - content: list[stix2._STIXBase21] = [self.stix.author_identity] + ) -> list[StixObject]: + content: list[StixObject] = [self.stix.author_identity] if victim: content.append(victim) content.extend(self._build_optional_entities(item, victim, incident_id)) @@ -372,11 +378,10 @@ def _build_content( ) return content - def _send_objects(self, objects: list[stix2._STIXBase21]) -> None: + def _send_objects(self, objects: list[StixObject]) -> None: if not objects: return - deduped = {obj.id: obj for obj in objects if getattr(obj, "id", None)} - bundle = stix2.Bundle(objects=list(deduped.values()), allow_custom=True) + bundle = stix2.Bundle(objects=dedupe_by_stix_id(objects), allow_custom=True) self.helper.send_stix2_bundle( bundle.serialize(), update=True, @@ -426,7 +431,7 @@ def _process_item_as_report( indicators: list[stix2.Indicator], ) -> None: content = self._build_content(item, victim, indicators) - object_refs = [obj.id for obj in content if getattr(obj, "id", None)] + object_refs = [obj.id for obj in content] report = self.stix.create_report(item, object_refs) self._send_objects([*content, report]) @@ -445,6 +450,8 @@ def _run_cycle(self) -> None: self.helper.connect_id, f"DEP connector - {now.strftime('%Y-%m-%d %H:%M:%S')} UTC", ) + work_message = f"DEP connector run completed, last_run: {end.isoformat()}" + in_error = False try: token = self.client.authenticate() for dataset in self.datasets: @@ -460,10 +467,16 @@ def _run_cycle(self) -> None: self._process_cycle_items(items) self._persist_dataset_state(dataset, end) state = self.helper.get_state() or state + except Exception as error: + in_error = True + work_message = f"DEP connector run failed: {error}" + self.helper.log_error(work_message) + raise finally: self.helper.api.work.to_processed( self._current_work_id, - f"DEP connector run completed, last_run: {end.isoformat()}", + work_message, + in_error=in_error, ) self._current_work_id = None diff --git a/dep_connector/converter_to_stix.py b/dep_connector/converter_to_stix.py index 4e23df6..dccdb33 100644 --- a/dep_connector/converter_to_stix.py +++ b/dep_connector/converter_to_stix.py @@ -29,6 +29,9 @@ class PrimaryObject(StrEnum): INCIDENT = "incident" +_ISO_ALPHA2_LENGTH = 2 + + class LeakRecord(BaseModel): model_config = ConfigDict(extra="allow", frozen=True, populate_by_name=True) @@ -40,8 +43,10 @@ class LeakRecord(BaseModel): sector: str | None = None actor: str | None = None country: str | None = None + country_code: str | None = Field(default=None, alias="victimCC") revenue: str | None = None + naics: str | None = None site: str | None = None ann_link: str | None = Field(default=None, alias="annLink") @@ -65,7 +70,7 @@ def annlink_repair_common_scrape_bug(cls, v: str | None) -> str | None: return "http://" + v[len("http//") :] return v - @field_validator("site", "victim_domain") + @field_validator("site", "victim_domain", "naics") @classmethod def strip_optional_text(cls, v: str | None) -> str | None: if v is None: @@ -73,6 +78,16 @@ def strip_optional_text(cls, v: str | None) -> str | None: stripped = v.strip() return stripped or None + @field_validator("country_code") + @classmethod + def normalize_country_code(cls, v: str | None) -> str | None: + if v is None: + return None + code = v.strip().upper() + if len(code) == _ISO_ALPHA2_LENGTH and code.isalpha(): + return code + return None + @staticmethod def _normalize_domain(value: str | None) -> str | None: if not value: @@ -104,6 +119,29 @@ def normalize_named_field(cls, v: str | None) -> str | None: return None return normalized + @field_validator("hashid") + @classmethod + def require_non_empty_hashid(cls, v: str) -> str: + if not v.strip(): + error = "hashid must be a non-empty string" + raise ValueError(error) + return v + + @field_validator("announcement_types", mode="before") + @classmethod + def drop_unknown_announcement_types(cls, v: object) -> object: + if v is None: + return [] + if not isinstance(v, list): + return v + known: list[AnnouncementType] = [] + for entry in v: + try: + known.append(AnnouncementType(entry)) + except ValueError: + continue + return known + GENERIC_ACTOR_VALUES = frozenset( { @@ -157,13 +195,15 @@ def _victim_external_references(item: LeakRecord) -> list[dict[str, str]]: description=item.ann_title, ) ) - if item.site and item.site != item.ann_link: - external_references.append( - _external_reference( - "victim-site", - url=_ensure_scheme(item.site), + if item.site: + site_url = _ensure_scheme(item.site) + if site_url != item.ann_link: + external_references.append( + _external_reference( + "victim-site", + url=site_url, + ) ) - ) return external_references @@ -236,10 +276,13 @@ def create_intrusion_set(self, actor: str, item: LeakRecord) -> stix2.IntrusionS def create_country_location(self, country: str, item: LeakRecord) -> stix2.Location: country_key = country.lower() location_id = f"location--{uuid5(NAMESPACE_URL, f'dep-country:{country_key}')}" + # STIX 2.1 Location requires `country` (or region/lat-long). Prefer DEP's + # ISO 3166-1 alpha-2 `victimCC`; fall back to the country name when the + # code is missing so the object stays valid. return stix2.Location( id=location_id, name=country, - country=country, + country=item.country_code or country, custom_properties={"x_opencti_location_type": "Country"}, allow_custom=True, **self._common_object_kwargs(item), @@ -387,4 +430,6 @@ def build_primary_custom_properties(item: LeakRecord) -> dict[str, str]: properties["dep_actor"] = item.actor if item.country is not None: properties["dep_country"] = item.country + if item.naics is not None: + properties["dep_naics"] = item.naics return properties diff --git a/dep_connector/datasets.py b/dep_connector/datasets.py new file mode 100644 index 0000000..627e601 --- /dev/null +++ b/dep_connector/datasets.py @@ -0,0 +1,35 @@ +from enum import StrEnum + + +class DepDataset(StrEnum): + EXTORTION = "ext" + PRIVACY = "prv" + OPENNEWS = "nws" + VANDALISM = "vnd" + DDOS = "dds" + FORUM = "frm" + + @classmethod + def _missing_(cls, value: object) -> "DepDataset | None": + if not isinstance(value, str): + return None + return DATASET_ALIASES.get(value) + + +DATASET_ALIASES: dict[str, DepDataset] = { + "extortion": DepDataset.EXTORTION, + "privacy": DepDataset.PRIVACY, + "opennews": DepDataset.OPENNEWS, + "news": DepDataset.OPENNEWS, + "vandalism": DepDataset.VANDALISM, + "ddos": DepDataset.DDOS, + "forum": DepDataset.FORUM, +} + + +def dataset_alias_summary() -> str: + aliases_by_dataset: dict[DepDataset, list[str]] = {} + for alias, dataset in DATASET_ALIASES.items(): + aliases_by_dataset.setdefault(dataset, []).append(alias) + groups = ["/".join(aliases_by_dataset[dataset]) for dataset in DepDataset] + return ", ".join(group for group in groups if group) diff --git a/dep_connector/stix_objects.py b/dep_connector/stix_objects.py new file mode 100644 index 0000000..ebfc529 --- /dev/null +++ b/dep_connector/stix_objects.py @@ -0,0 +1,13 @@ +from collections.abc import Iterable +from typing import Protocol, TypeVar + + +class StixObject(Protocol): + id: str + + +TStixObject = TypeVar("TStixObject", bound=StixObject) + + +def dedupe_by_stix_id(objects: Iterable[TStixObject]) -> list[TStixObject]: + return list({stix_object.id: stix_object for stix_object in objects}.values()) diff --git a/docker-compose.yml b/docker-compose.yml index d55b7df..bc1eaa2 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -88,7 +88,7 @@ services: timeout: 30s retries: 3 opencti: - image: opencti/platform:6.8.13 + image: opencti/platform:6.9.0 environment: - NODE_OPTIONS=--max-old-space-size=8096 - APP__PORT=8080 @@ -140,7 +140,7 @@ services: timeout: 5s retries: 20 worker: - image: opencti/worker:6.8.13 + image: opencti/worker:6.9.0 environment: - OPENCTI_URL=http://opencti:8080 - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN} @@ -153,7 +153,7 @@ services: replicas: 3 restart: always connector-export-file-stix: - image: opencti/connector-export-file-stix:6.8.13 + image: opencti/connector-export-file-stix:6.9.0 environment: - OPENCTI_URL=http://opencti:8080 - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN} @@ -167,7 +167,7 @@ services: opencti: condition: service_healthy connector-export-file-csv: - image: opencti/connector-export-file-csv:6.8.13 + image: opencti/connector-export-file-csv:6.9.0 environment: - OPENCTI_URL=http://opencti:8080 - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN} @@ -181,7 +181,7 @@ services: opencti: condition: service_healthy connector-export-file-txt: - image: opencti/connector-export-file-txt:6.8.13 + image: opencti/connector-export-file-txt:6.9.0 environment: - OPENCTI_URL=http://opencti:8080 - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN} @@ -195,7 +195,7 @@ services: opencti: condition: service_healthy connector-import-file-stix: - image: opencti/connector-import-file-stix:6.8.13 + image: opencti/connector-import-file-stix:6.9.0 environment: - OPENCTI_URL=http://opencti:8080 - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN} @@ -211,7 +211,7 @@ services: opencti: condition: service_healthy connector-import-document: - image: opencti/connector-import-document:6.8.13 + image: opencti/connector-import-document:6.9.0 environment: - OPENCTI_URL=http://opencti:8080 - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN} @@ -229,7 +229,7 @@ services: opencti: condition: service_healthy connector-analysis: - image: opencti/connector-import-document:6.8.13 + image: opencti/connector-import-document:6.9.0 environment: - OPENCTI_URL=http://opencti:8080 - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN} diff --git a/pyproject.toml b/pyproject.toml index b8db29b..7d342ee 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -2,11 +2,11 @@ name = "opencti-connector" version = "0.1.0" description = "The Double Extortion connector ingests ransomware and data leak announcements published on the DoubleExtortion platform and converts them into STIX entities inside OpenCTI." -authors = [{ name = "Luca Mella", email = "lm@digintlab.com" }] +authors = [{ name = "notdodo" }, { name = "luca-m" }] requires-python = ">=3.11" readme = "README.md" dependencies = [ - "pycti>=6.8.13", + "pycti>=6.9.0", "pyyaml>=6.0.3", "requests>=2.32.5", "stix2>=3.0.1", diff --git a/tests/test_client_api.py b/tests/test_client_api.py new file mode 100644 index 0000000..d0ed1a1 --- /dev/null +++ b/tests/test_client_api.py @@ -0,0 +1,155 @@ +from unittest.mock import Mock, patch + +import pytest +import requests + +from dep_connector.client_api import DepClient +from dep_connector.datasets import DepDataset + + +def _client(*, extended_results: bool = True) -> DepClient: + return DepClient( + login_endpoint="https://login.example.test", + api_endpoint="https://api.example.test/leaks", + api_key="api-key", + username="user", + password="password", + client_id="client-id", + extended_results=extended_results, + ) + + +def test_authenticate_returns_cognito_id_token() -> None: + response = Mock() + response.json.return_value = {"AuthenticationResult": {"IdToken": "token-123"}} + with patch("dep_connector.client_api.requests.post", return_value=response) as post: + assert _client().authenticate() == "token-123" + response.raise_for_status.assert_called_once() + post.assert_called_once_with( + "https://login.example.test", + headers={ + "Content-Type": "application/x-amz-json-1.1", + "X-Amz-Target": "AWSCognitoIdentityProviderService.InitiateAuth", + }, + json={ + "AuthParameters": {"USERNAME": "user", "PASSWORD": "password"}, + "AuthFlow": "USER_PASSWORD_AUTH", + "ClientId": "client-id", + }, + timeout=30, + ) + + +@pytest.mark.parametrize( + "payload", + [ + {}, + {"AuthenticationResult": {}}, + {"AuthenticationResult": {"IdToken": ""}}, + {"AuthenticationResult": {"IdToken": 42}}, + [], + ], +) +def test_authenticate_rejects_missing_or_invalid_id_token(payload: object) -> None: + response = Mock() + response.json.return_value = payload + + with ( + patch("dep_connector.client_api.requests.post", return_value=response), + pytest.raises(ValueError, match="Invalid DEP authentication response"), + ): + _client().authenticate() + + +def test_authenticate_wraps_invalid_json_response() -> None: + response = Mock() + response.json.side_effect = requests.exceptions.JSONDecodeError("bad", "{}", 0) + + with ( + patch("dep_connector.client_api.requests.post", return_value=response), + pytest.raises(ValueError, match="decode DEP authentication response"), + ): + _client().authenticate() + + +def test_fetch_raw_returns_api_items_and_request_parameters() -> None: + response = Mock() + response.json.return_value = [ + {"date": "2026-03-27", "hashid": "abc", "victim": "Victim"} + ] + with patch("dep_connector.client_api.requests.get", return_value=response) as get: + items = _client(extended_results=False).fetch_raw( + dataset=DepDataset.EXTORTION, + start_date="2026-03-01", + end_date="2026-03-27", + token="token-123", + ) + + assert items == [{"date": "2026-03-27", "hashid": "abc", "victim": "Victim"}] + response.raise_for_status.assert_called_once() + get.assert_called_once_with( + "https://api.example.test/leaks", + headers={"X-Api-Key": "api-key", "Authorization": "token-123"}, + params={ + "ts": "2026-03-01", + "te": "2026-03-27", + "dset": DepDataset.EXTORTION, + "full": "true", + }, + timeout=60, + ) + + +def test_fetch_raw_adds_extended_parameter_when_enabled() -> None: + response = Mock() + response.json.return_value = [] + with patch("dep_connector.client_api.requests.get", return_value=response) as get: + _client().fetch_raw( + dataset=DepDataset.DDOS, + start_date="2026-03-01", + end_date="2026-03-27", + token="token-123", + ) + + assert get.call_args.kwargs["params"]["extended"] == "true" + + +@pytest.mark.parametrize( + "payload", + [ + {}, + {"items": []}, + [42], + ["not an object"], + ], +) +def test_fetch_raw_rejects_invalid_api_payload_shape(payload: object) -> None: + response = Mock() + response.json.return_value = payload + + with ( + patch("dep_connector.client_api.requests.get", return_value=response), + pytest.raises(ValueError, match="DEP API response"), + ): + _client().fetch_raw( + dataset=DepDataset.EXTORTION, + start_date="2026-03-01", + end_date="2026-03-27", + token="token-123", + ) + + +def test_fetch_raw_wraps_invalid_json_response() -> None: + response = Mock() + response.json.side_effect = requests.exceptions.JSONDecodeError("bad", "{}", 0) + + with ( + patch("dep_connector.client_api.requests.get", return_value=response), + pytest.raises(ValueError, match="decode DEP API response"), + ): + _client().fetch_raw( + dataset=DepDataset.EXTORTION, + start_date="2026-03-01", + end_date="2026-03-27", + token="token-123", + ) diff --git a/tests/test_connector_runtime.py b/tests/test_connector_runtime.py index d4a79e6..d4160ef 100644 --- a/tests/test_connector_runtime.py +++ b/tests/test_connector_runtime.py @@ -2,6 +2,7 @@ from unittest.mock import Mock, patch import pycti # type: ignore[import-untyped] +import pytest from stix2 import TLP_AMBER # type: ignore[import-untyped] from stix2 import v21 as stix2 @@ -222,3 +223,40 @@ def test_persist_dataset_state_merges_existing_dataset_entries() -> None: } } ) + + +def test_run_cycle_marks_work_in_error_when_run_fails() -> None: + connector = DepConnector.__new__(DepConnector) + connector.helper = Mock() + connector.helper.get_state.return_value = {} + connector.helper.connect_id = "connector-id" + connector.helper.api.work.initiate_work.return_value = "work-id" + connector.client = Mock() + connector.client.authenticate.side_effect = RuntimeError("auth failed") + connector.datasets = (DepDataset.EXTORTION,) + connector.overlap_hours = 24 + connector.lookback_days = 7 + connector._current_work_id = None + + with pytest.raises(RuntimeError, match="auth failed"): + connector._run_cycle() + + connector.helper.log_error.assert_called_once_with( + "DEP connector run failed: auth failed" + ) + connector.helper.api.work.to_processed.assert_called_once() + assert connector.helper.api.work.to_processed.call_args.args[:2] == ( + "work-id", + "DEP connector run failed: auth failed", + ) + assert connector.helper.api.work.to_processed.call_args.kwargs == {"in_error": True} + assert connector._current_work_id is None + + +def test_build_client_requires_api_key(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv("DEP_API_KEY", raising=False) + monkeypatch.delenv("DEP_CLIENT_ID", raising=False) + connector = DepConnector.__new__(DepConnector) + + with pytest.raises(ValueError, match="API key"): + connector._build_client({"dep": {"client_id": "client-123"}}) diff --git a/tests/test_converter_core.py b/tests/test_converter_core.py index c8e89a1..2ec7fb7 100644 --- a/tests/test_converter_core.py +++ b/tests/test_converter_core.py @@ -1,8 +1,11 @@ import pycti # type: ignore[import-untyped] +import pytest +from pydantic import ValidationError from stix2 import TLP_AMBER # type: ignore[import-untyped] from stix2 import v21 as stix2 from dep_connector import LeakRecord, StixBuilder +from dep_connector.converter_to_stix import AnnouncementType def build_builder() -> StixBuilder: @@ -133,3 +136,154 @@ def test_incident_id_is_deterministic_from_hashid() -> None: assert first_incident.id == second_incident.id assert first_incident.name != second_incident.name + + +def test_leak_record_drops_unknown_announcement_types() -> None: + item = LeakRecord( + date="2026-03-27", + hashid="a" * 64, + victim="Example Victim", + annDataTypes=["PII", "CREDENTIALS", "MEDICAL"], + ) + + assert item.announcement_types == [AnnouncementType.PII, AnnouncementType.MEDICAL] + + +def test_leak_record_coerces_null_announcement_types_to_empty() -> None: + item = LeakRecord( + date="2026-03-27", + hashid="a" * 64, + victim="Example Victim", + annDataTypes=None, + ) + + assert item.announcement_types == [] + + +def test_leak_record_rejects_empty_hashid() -> None: + with pytest.raises(ValidationError): + LeakRecord(date="2026-03-27", hashid=" ", victim="Example Victim") + + +def test_victim_external_references_skips_site_matching_ann_link() -> None: + builder = build_builder() + item = LeakRecord( + date="2026-03-27", + hashid="a" * 64, + victim="Dedup Victim", + annLink="https://example.com", + site="example.com", + ) + + victim = builder.create_victim_identity(item, include_sector_in_description=False) + + assert victim is not None + assert {ref.source_name for ref in victim.external_references} == {"dep"} + + +def test_victim_external_references_keeps_distinct_site() -> None: + builder = build_builder() + item = LeakRecord( + date="2026-03-27", + hashid="b" * 64, + victim="Distinct Victim", + annLink="https://example.com/leak", + site="portal.example.com", + ) + + victim = builder.create_victim_identity(item, include_sector_in_description=False) + + assert victim is not None + refs = {ref.source_name: ref.url for ref in victim.external_references} + assert refs == { + "dep": "https://example.com/leak", + "victim-site": "https://portal.example.com", + } + + +def _full_dep_item() -> dict[str, object]: + # Mirrors the real DEP API item shape (every observed key) with synthetic, + # non-sensitive values, so no real breach-victim data is committed. + return { + "date": "2026-03-27", + "hashid": "a" * 64, + "victim": "Synthetic Corp", + "sector": "Manufacturing", + "actor": "Example Gang", + "country": "Italy", + "victimCC": "it", + "naics": " 541611 ", + "revenue": "$10M-$50M", + "site": "synthetic.example", + "annLink": "https://example.com/leak", + "annTitle": "Synthetic leak", + "annDescription": "Leaked%20data%20available", + "victimDomain": "synthetic.example", + "annDataTypes": ["PII", "FINANCIAL", "FUTURE_TYPE"], + "amount": "over 45 ye", + "victimAddress": None, + "victimCity": "Milan", + "victimState": "25", + } + + +def test_full_real_shaped_item_parses_and_maps() -> None: + item = LeakRecord(**_full_dep_item(), dep_dataset="ext") + + assert item.victim == "Synthetic Corp" + assert item.country_code == "IT" + assert item.naics == "541611" + assert item.country == "Italy" + assert item.indicator_domain == "synthetic.example" + # Unknown enum members dropped; unmapped DEP fields tolerated without crashing. + assert item.announcement_types == [ + AnnouncementType.PII, + AnnouncementType.FINANCIAL, + ] + + +def test_country_location_uses_alpha2_from_country_code() -> None: + builder = build_builder() + item = LeakRecord( + date="2026-03-27", + hashid="a" * 64, + victim="Synthetic Corp", + country="Italy", + victimCC="IT", + ) + + location = builder.create_country_location("Italy", item) + + assert location.name == "Italy" + assert location.country == "IT" + + +def test_country_location_falls_back_to_name_without_code() -> None: + builder = build_builder() + item = LeakRecord( + date="2026-03-27", + hashid="a" * 64, + victim="Synthetic Corp", + country="Italy", + ) + + location = builder.create_country_location("Italy", item) + + assert location.name == "Italy" + assert location.country == "Italy" + + +def test_naics_surfaced_as_custom_property() -> None: + builder = build_builder() + item = LeakRecord( + date="2026-03-27", + hashid="a" * 64, + victim="Synthetic Corp", + country="Italy", + naics="541611", + ) + + properties = builder.build_primary_custom_properties(item) + + assert properties["dep_naics"] == "541611" + assert properties["dep_country"] == "Italy" diff --git a/uv.lock b/uv.lock index 6abd0de..1d15ef3 100644 --- a/uv.lock +++ b/uv.lock @@ -417,7 +417,7 @@ dev = [ [package.metadata] requires-dist = [ - { name = "pycti", specifier = ">=6.8.13" }, + { name = "pycti", specifier = ">=6.9.0" }, { name = "pyyaml", specifier = ">=6.0.3" }, { name = "requests", specifier = ">=2.32.5" }, { name = "stix2", specifier = ">=3.0.1" },