build(deps): bump python from 3.12-slim to 3.14.5-slim in the dep-connector-docker group

AGENTS.md

@@ -10,7 +10,15 @@ It should track the code in `main.py`, not stale assumptions from earlier iterat
 - This is an OpenCTI external-import connector for Double Extortion Platform (DEP) announcements.
 - The connector authenticates against DEP AWS Cognito, fetches announcement records from the DEP REST API, converts them to STIX 2.1, and sends bundles to OpenCTI with `update=True`.
 - The connector scope is `report,incident,identity,indicator`.
-- The implementation is split across the `dep_connector/` package (`converter_to_stix.py`, `client_api.py`, `config_loader.py`, `connector.py`) with `main.py` as the thin entrypoint.
+- The implementation is split across the `dep_connector/` package:
+  - `connector.py`: run-cycle orchestration and OpenCTI state handling
+  - `client_api.py`: DEP authentication and fetch HTTP calls
+  - `api_models.py`: DEP API/auth response validation models
+  - `datasets.py`: DEP dataset codes, aliases, and validation helpers
+  - `converter_to_stix.py`: DEP record parsing and STIX object construction
+  - `stix_objects.py`: shared STIX object protocols and collection helpers
+  - `config_loader.py`: YAML configuration loading
+  - `main.py`: thin entrypoint
 
 ## Runtime and configuration truths
 
@@ -19,8 +27,8 @@ It should track the code in `main.py`, not stale assumptions from earlier iterat
 - `DEP_CLIENT_ID` is required at startup even though `config.yml.sample` leaves it blank. Missing it raises `ValueError`.
 - The runtime loop is infinite: `run()` executes one cycle, then sleeps for `CONNECTOR_RUN_INTERVAL`.
 - Local Docker Compose mounts `./config.yml` into `/app/config.yml` for the `dep-connector` service.
-- The local stack pins OpenCTI services to `6.8.13`; the connector manifest declares support for OpenCTI `>= 6.8.13`.
-- The container image runs `python main.py` as the non-root `app` user on Python 3.12.
+- The local stack pins OpenCTI services to `6.9.0` (tested against latest); the connector manifest declares support for OpenCTI `>= 6.8.13` (no 6.9.0-only platform API is used).
+- The container image runs `python main.py` as the non-root `app` user on Python 3.14.5.
 
 ## DEP fetch behavior
 
@@ -48,6 +56,9 @@ It should track the code in `main.py`, not stale assumptions from earlier iterat
   - `ddos` -> `dds`
   - `forum` -> `frm`
 - The DEP API accepts one `dset` value per request, so the connector loops over configured datasets and issues one request per dataset.
+- The DEP API caps each response at **1000 items** and exposes no pagination/offset/cursor parameter on this endpoint (empirically: a 1-day window returns ~13 items, 30 days ~654, and both 365-day and 1000-day windows return exactly 1000). `fetch_raw` returns whatever the API sends with no cap detection.
+- Consequence: if a single per-dataset run window contains more than 1000 announcements, the excess is silently dropped, and because per-dataset state advances to the window `end` regardless, the dropped items are never re-fetched (permanent loss for that window). Risk scenarios: a large first-run `DEP_LOOKBACK_DAYS`, catch-up after long downtime, or a busy dataset over a wide window.
+- Mitigation until window-chunking is implemented: keep each run window comfortably under 1000 items per dataset (small `DEP_LOOKBACK_DAYS`, frequent `CONNECTOR_RUN_INTERVAL`); be cautious with large backfills. A proper fix would split the `ts`/`te` window into smaller sub-windows whenever a response returns exactly 1000 items.
 
 ## State management
 
@@ -66,8 +77,9 @@ It should track the code in `main.py`, not stale assumptions from earlier iterat
 - `annLink` is repaired for a known scrape bug:
   - `https//...` -> `https://...`
   - `http//...` -> `http://...`
-- `site` and `victimDomain` are stripped; empty strings become `None`.
+- `site`, `victimDomain`, and `naics` are stripped; empty strings become `None`.
 - `sector`, `actor`, and `country` are whitespace-normalized; empty strings, `n/a`, and `none` become `None`.
+- `country_code` (DEP `victimCC`) is upper-cased and kept only when it is a 2-letter alpha ISO 3166-1 code; otherwise `None`.
 - Indicator domain extraction prefers `victimDomain`, then falls back to `site`.
 - Domain normalization uses `urlsplit`, extracts the hostname, and lowercases it.
 - `annDescription` is URL-decoded with `urllib.parse.unquote` before the report or incident is created.
@@ -127,6 +139,7 @@ It should track the code in `main.py`, not stale assumptions from earlier iterat
 - Report custom properties (when present):
   - `dep_actor`
   - `dep_country`
+  - `dep_naics`
 - Report labels always include `DigIntLab`, plus any applicable:
   - `dep:announcement-type:<lowercased enum value>`
   - `dep:dataset:<dataset code>`
@@ -150,6 +163,7 @@ It should track the code in `main.py`, not stale assumptions from earlier iterat
   - `first_seen`
   - `dep_actor` when present
   - `dep_country` when present
+  - `dep_naics` when present
 - Incident labels always include `DigIntLab`, plus any applicable:
   - `dep:announcement-type:<lowercased enum value>`
   - `dep:dataset:<dataset code>`
@@ -197,9 +211,11 @@ It should track the code in `main.py`, not stale assumptions from earlier iterat
   - victim identity exists
 - Deterministic country location ID:
   - `location--uuid5(NAMESPACE_URL, "dep-country:<country>")`
-- Always set both:
-  - `name=<country>`
-  - `country=<country>`
+- Always set:
+  - `name=<country>` (human-readable country name)
+  - `country=<ISO 3166-1 alpha-2 code from DEP victimCC>`, falling back to the
+    country name when no valid code is available (STIX 2.1 requires `country`,
+    region, or lat/long, so the field is never omitted)
 - Preserve the OpenCTI-specific custom property:
   - `x_opencti_location_type: Country`
 
@@ -319,8 +335,11 @@ Current automated coverage focuses on:
 ## File map
 
 - Connector entrypoint: `main.py`
+- DEP API/auth response validation models: `dep_connector/api_models.py`
 - Data models and STIX converter: `dep_connector/converter_to_stix.py`
 - DEP API client (auth + fetch): `dep_connector/client_api.py`
+- DEP dataset codes and aliases: `dep_connector/datasets.py`
+- STIX object typing helpers: `dep_connector/stix_objects.py`
 - Configuration loader: `dep_connector/config_loader.py`
 - Connector orchestration (run cycle): `dep_connector/connector.py`
 - Package re-export: `dep_connector/__init__.py`

Dockerfile

@@ -1,10 +1,9 @@
-FROM python:3.12-slim AS builder
+FROM python:3.14.5-slim AS builder
 HEALTHCHECK NONE
 
 ENV UV_LINK_MODE=copy \
     UV_COMPILE_BYTECODE=1 \
     UV_PYTHON_DOWNLOADS=never \
-    UV_PYTHON=python3.12 \
     UV_NO_PROGRESS=1
 
 COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
@@ -22,7 +21,7 @@ COPY dep_connector/ dep_connector/
 RUN --mount=type=cache,target=/root/.cache/uv \
     uv sync --locked --no-editable --no-dev
 
-FROM python:3.12-slim AS runtime
+FROM python:3.14.5-slim AS runtime
 HEALTHCHECK NONE
 
 ENV PATH="/app/.venv/bin:${PATH}"
@@ -37,4 +36,4 @@ LABEL org.opencontainers.image.source=https://github.com/DigintLab/opencti-conne
 LABEL org.opencontainers.image.description="The Double Extortion connector ingests ransomware and data leak announcements published on the DoubleExtortion platform and converts them into STIX entities inside OpenCTI."
 
 USER app
-ENTRYPOINT ["python", "main.py"]
+ENTRYPOINT ["python", "main.py"]

README.md

@@ -85,6 +85,8 @@ Dataset aliases are normalized to the short API codes before requests are sent.
 
 The connector adds `extended=true` only when `DEP_EXTENDED_RESULTS=true`.
 
+> **Limitation — 1000-item response cap.** The DEP API returns at most **1000 items per request** and offers no pagination on this endpoint. If a single per-dataset run window contains more than 1000 announcements, the surplus is silently dropped and is **not** re-fetched on the next run (state advances regardless). Keep each run window under that cap by using a modest `DEP_LOOKBACK_DAYS` and a frequent enough `CONNECTOR_RUN_INTERVAL`, and take care with large first-run backfills or long downtime on busy datasets. As a rough guide observed from the API: ~13 items/day, ~654 over 30 days, and the cap (1000) is reached somewhere beyond a ~2-month window for the `ext` dataset.
+
 ## Why `IntrusionSet` for DEP actor values
 
 DEP `actor` values are modeled as STIX `IntrusionSet` objects instead of `ThreatActor` by default.
@@ -130,7 +132,8 @@ docker run --rm \
 - The API occasionally URL-encodes announcement descriptions. The connector automatically decodes the description before sending it to OpenCTI.
 - DEP `annLink` values are repaired for a known scrape bug (`https//...` or `http//...`) before they are used as external references.
 - DEP actor and country values can be materialized as entities using `DEP_CREATE_INTRUSION_SETS` and `DEP_CREATE_COUNTRY_LOCATIONS`.
-- DEP actor and country values are also stored in the primary object custom properties (`dep_actor`, `dep_country`) for source traceability.
+- Country locations carry the human-readable country name plus the ISO 3166-1 alpha-2 code from DEP `victimCC` (falling back to the name when no valid code is available), as expected by OpenCTI/STIX.
+- DEP actor, country, and NAICS industry code are also stored in the primary object custom properties (`dep_actor`, `dep_country`, `dep_naics`) for source traceability.
 - Generated indicators are also linked to the victim with `related-to` so those indicator nodes are connected in the Knowledge Graph.
 - Cross-entity links are automatic: intrusion set -> sector (`targets`), intrusion set -> country (`targets`), and sector -> country (`related-to`) when both entities are present.
 - Generic low-quality actor values (for example `unknown`, `anonymous`, `ransomware group`) are ignored for intrusion-set creation.

config.yml.sample

@@ -25,6 +25,7 @@ dep:
   api_endpoint: https://api.eu-ep1.doubleextortion.com/v1/dbtr/privlist
   datasets:
     - ext
+    - dds
   lookback_days: 7
   overlap_hours: 72
   extended_results: true  # Adds extended=true to DEP API requests.

dep_connector/__init__.py

@@ -1,6 +1,6 @@
-from dep_connector.client_api import DepDataset
 from dep_connector.connector import DepConnector
 from dep_connector.converter_to_stix import LeakRecord, PrimaryObject, StixBuilder
+from dep_connector.datasets import DepDataset
 
 __all__ = [
     "DepConnector",

dep_connector/api_models.py

@@ -0,0 +1,41 @@
+from typing import TypeAlias
+
+from pydantic import (
+    BaseModel,
+    ConfigDict,
+    Field,
+    JsonValue,
+    StrictStr,
+    TypeAdapter,
+    field_validator,
+)
+
+DepApiItem: TypeAlias = dict[str, JsonValue]
+
+
+class CognitoAuthenticationResult(BaseModel):
+    model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+    id_token: StrictStr = Field(alias="IdToken")
+
+    @field_validator("id_token")
+    @classmethod
+    def require_non_empty_token(cls, value: str) -> str:
+        if not value:
+            error = "Unable to retrieve IdToken from authentication response"
+            raise ValueError(error)
+        return value
+
+
+class CognitoAuthResponse(BaseModel):
+    model_config = ConfigDict(extra="ignore", populate_by_name=True)
+
+    authentication_result: CognitoAuthenticationResult = Field(
+        alias="AuthenticationResult"
+    )
+
+
+COGNITO_AUTH_RESPONSE_ADAPTER: TypeAdapter[CognitoAuthResponse] = TypeAdapter(
+    CognitoAuthResponse
+)
+DEP_API_ITEMS_ADAPTER: TypeAdapter[list[DepApiItem]] = TypeAdapter(list[DepApiItem])

dep_connector/client_api.py

@@ -1,49 +1,38 @@
 import json
 import logging
-from enum import StrEnum
-from typing import TypeAlias
+from typing import TypeVar
 
 import requests
+from pydantic import TypeAdapter, ValidationError
 
-logger = logging.getLogger(__name__)
-
-JsonPrimitive: TypeAlias = str | int | float | bool | None
-JsonValue: TypeAlias = JsonPrimitive | list["JsonValue"] | dict[str, "JsonValue"]
-DepApiItem: TypeAlias = dict[str, JsonValue]
-
-
-class DepDataset(StrEnum):
-    EXTORTION = "ext"
-    PRIVACY = "prv"
-    OPENNEWS = "nws"
-    VANDALISM = "vnd"
-    DDOS = "dds"
-    FORUM = "frm"
+from dep_connector.api_models import (
+    COGNITO_AUTH_RESPONSE_ADAPTER,
+    DEP_API_ITEMS_ADAPTER,
+    DepApiItem,
+)
+from dep_connector.datasets import DepDataset
 
-    @classmethod
-    def _missing_(cls, value: object) -> "DepDataset | None":
-        if not isinstance(value, str):
-            return None
-        return DATASET_ALIASES.get(value)
+logger = logging.getLogger(__name__)
+TValidatedPayload = TypeVar("TValidatedPayload")
 
 
-DATASET_ALIASES: dict[str, DepDataset] = {
-    "extortion": DepDataset.EXTORTION,
-    "privacy": DepDataset.PRIVACY,
-    "opennews": DepDataset.OPENNEWS,
-    "news": DepDataset.OPENNEWS,
-    "vandalism": DepDataset.VANDALISM,
-    "ddos": DepDataset.DDOS,
-    "forum": DepDataset.FORUM,
-}
+def _decode_json_response(response: requests.Response, error_message: str) -> object:
+    try:
+        payload: object = response.json()
+    except (json.JSONDecodeError, requests.exceptions.JSONDecodeError) as exception:
+        raise ValueError(error_message) from exception
+    return payload
 
 
-def dataset_alias_summary() -> str:
-    aliases_by_dataset: dict[DepDataset, list[str]] = {}
-    for alias, dataset in DATASET_ALIASES.items():
-        aliases_by_dataset.setdefault(dataset, []).append(alias)
-    groups = ["/".join(aliases_by_dataset[dataset]) for dataset in DepDataset]
-    return ", ".join(group for group in groups if group)
+def _validate_payload(
+    adapter: TypeAdapter[TValidatedPayload],
+    payload: object,
+    error_message: str,
+) -> TValidatedPayload:
+    try:
+        return adapter.validate_python(payload)
+    except ValidationError as exception:
+        raise ValueError(error_message) from exception
 
 
 class DepClient:
@@ -83,12 +72,16 @@ def authenticate(self) -> str:
             timeout=30,
         )
         response.raise_for_status()
-        auth_payload: dict[str, dict[str, str]] = response.json()
-        token = auth_payload["AuthenticationResult"]["IdToken"]
-        if not token:
-            error = "Unable to retrieve IdToken from authentication response"
-            raise ValueError(error)
-        return token
+        auth_payload = _decode_json_response(
+            response,
+            "Unable to decode DEP authentication response",
+        )
+        auth_response = _validate_payload(
+            COGNITO_AUTH_RESPONSE_ADAPTER,
+            auth_payload,
+            "Invalid DEP authentication response",
+        )
+        return auth_response.authentication_result.id_token
 
     def fetch_raw(
         self,
@@ -120,9 +113,7 @@ def fetch_raw(
             timeout=60,
         )
         response.raise_for_status()
-        try:
-            payload: list[DepApiItem] = response.json()
-        except json.JSONDecodeError as exception:
-            message = "Unable to decode DEP API response"
-            raise ValueError(message) from exception
-        return payload
+        payload = _decode_json_response(response, "Unable to decode DEP API response")
+        return _validate_payload(
+            DEP_API_ITEMS_ADAPTER, payload, "Invalid DEP API response"
+        )