feat(metras): add Metras external-import connector for EDR telemetry (#7042) - #7039
Open
Khidr6G wants to merge 7 commits into
Open
feat(metras): add Metras external-import connector for EDR telemetry (#7042)#7039Khidr6G wants to merge 7 commits into
Khidr6G wants to merge 7 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new Metras EXTERNAL_IMPORT connector under external-import/metras/ to ingest Metras EDR telemetry into OpenCTI as STIX (Incidents + ATT&CK Attack-Patterns + Url/File observables + System identities), with a shared HTTP client, deterministic IDs, and a basic unit-test suite.
Changes:
- Introduces the Metras feed connector implementation (settings, API client, STIX conversion, import loop with state cursors + Work API lifecycle).
- Adds unit tests for utility functions, settings, API client, converter, and orchestration.
- Adds packaging/runtime artifacts for containerized deployment (Dockerfile, compose, entrypoint) plus connector metadata/docs.
Reviewed changes
Copilot reviewed 25 out of 25 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| external-import/metras/VERSION | Adds connector version marker. |
| external-import/metras/tests/test-requirements.txt | Defines test dependencies for the connector. |
| external-import/metras/tests/test_main.py | Tests pure utility functions (refang, validation, timestamps, severity mapping). |
| external-import/metras/tests/test_connector/test_settings.py | Tests Pydantic settings behavior and defaults. |
| external-import/metras/tests/test_connector/test_orchestration.py | Tests connector import loop behavior with mocked helper/client. |
| external-import/metras/tests/test_connector/test_converter.py | Tests alert/binary/endpoint → STIX conversion outputs. |
| external-import/metras/tests/test_connector/test_client.py | Tests API client request/response handling with mocked HTTP. |
| external-import/metras/tests/conftest.py | Adds src path for test imports. |
| external-import/metras/src/requirements.txt | Adds runtime dependencies (pycti, stix2, requests, connectors-sdk, etc.). |
| external-import/metras/src/metras_client/api_client.py | Implements shared Metras REST API client (retrying session + endpoints). |
| external-import/metras/src/metras_client/init.py | Exposes MetrasClient / MetrasAPIError. |
| external-import/metras/src/main.py | Adds connector entrypoint wiring settings → helper → connector.run(). |
| external-import/metras/src/connector/utils.py | Adds dependency-free helpers (refang, validation, timestamp helpers, severity mapping). |
| external-import/metras/src/connector/settings.py | Adds connector + Metras config models (connectors-sdk + pydantic). |
| external-import/metras/src/connector/converter_to_stix.py | Converts Metras alerts/binaries/endpoints into STIX objects + relationships. |
| external-import/metras/src/connector/connector.py | Implements the feed import loop, cursors/state, and Work API status updates. |
| external-import/metras/src/connector/init.py | Exposes connector and settings entrypoints. |
| external-import/metras/README.md | Documents behavior, mapping, incremental sync, and configuration. |
| external-import/metras/pyproject.toml | Adds coverage configuration for this connector. |
| external-import/metras/entrypoint.sh | Adds container entrypoint script. |
| external-import/metras/Dockerfile | Builds a non-root Alpine-based container image for the connector. |
| external-import/metras/docker-compose.yml | Provides a hardened compose example for deployment. |
| external-import/metras/config.yml.sample | Provides sample YAML configuration. |
| external-import/metras/.dockerignore | Excludes unnecessary files from Docker build context. |
| external-import/metras/metadata/connector_manifest.json | Adds connector catalog metadata manifest. |
Khidr6G
changed the title
Comment on lines
+20
to
+36
| from pycti import ( | ||
| AttackPattern, | ||
| Identity, | ||
| Incident, | ||
| StixCoreRelationship, | ||
| ) | ||
|
|
||
| if TYPE_CHECKING: | ||
| from pycti import OpenCTIConnectorHelper | ||
|
|
||
| _TLP_BY_NAME = { | ||
| "clear": stix2.TLP_WHITE, | ||
| "white": stix2.TLP_WHITE, | ||
| "green": stix2.TLP_GREEN, | ||
| "amber": stix2.TLP_AMBER, | ||
| "red": stix2.TLP_RED, | ||
| } |
Member
There was a problem hiding this comment.
To avoid this kind of issues, it's strongly advised to use the OCTI models from connectors_sdk instead of stix2 objects (see https://github.com/OpenCTI-Platform/connectors/blob/master/connectors-sdk/connectors_sdk/models/tlp_marking.py)
Comment on lines
+181
to
+185
| if len(all_objects) > 1: # more than just the author | ||
| bundle = self.helper.stix2_create_bundle(all_objects) | ||
| self.helper.send_stix2_bundle( | ||
| bundle, work_id=work_id, cleanup_inconsistent_bundle=True | ||
| ) |
Comment on lines
+200
to
+205
| # Advance cursors only for categories that did not hard-fail. | ||
| new_state = dict(state) | ||
| if "alerts" not in "".join(errors) and new_alerts_max is not None: | ||
| new_state["alerts_last_occurrence"] = stix_timestamp(new_alerts_max) | ||
| if "binaries" not in "".join(errors) and new_binaries_max: | ||
| new_state["binaries_last_seen"] = new_binaries_max |
Comment on lines
+45
to
+48
| | `CONNECTOR_SCOPE` | no | `Metras` | Import scope | | ||
| | `CONNECTOR_LOG_LEVEL` | no | `info` | Log level | | ||
| | `CONNECTOR_DURATION_PERIOD` | yes | `PT1H` | ISO-8601 poll interval | | ||
| | `METRAS_API_BASE_URL` | no | `https://api.metras.sa/api` | Metras API base URL | |
Open
5 tasks
Member
|
Also, please rename connector folder so it has a unique name. The rule is: folder name == manifest's slug == docker image's name 👌 |
5 tasks
Powlinett
requested changes
Jul 28, 2026
| "subscription_link": null, | ||
| "source_code": "https://github.com/OpenCTI-Platform/connectors/tree/master/external-import/metras", | ||
| "manager_supported": false, | ||
| "container_version": "1.0.0", |
Member
There was a problem hiding this comment.
Suggested change
| "container_version": "1.0.0", | |
| "container_version": "rolling", |
Member
There was a problem hiding this comment.
To remove - connector's version is declared in connector_manifest.json
Comment on lines
+3
to
+5
| build: | ||
| context: . | ||
| dockerfile: Dockerfile |
Member
There was a problem hiding this comment.
To remove - only pull the image from our Docker hub to stay consistent with the other connectors' docker-compose.Yml
| build: | ||
| context: . | ||
| dockerfile: Dockerfile | ||
| image: opencti/connector-metras-feed:1.0.0 |
Member
There was a problem hiding this comment.
Suggested change
| image: opencti/connector-metras-feed:1.0.0 | |
| image: opencti/connector-metras-feed:latest |
| - OPENCTI_URL=http://opencti:8080 | ||
| - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN} | ||
| - CONNECTOR_ID=${CONNECTOR_METRAS_FEED_ID} | ||
| - CONNECTOR_TYPE=EXTERNAL_IMPORT |
Member
There was a problem hiding this comment.
To remove - connector's type must always be hardcoded, never configurable
Comment on lines
+15
to
+17
| name: str = Field(default="Metras-Feed", examples=["Metras-Feed"]) | ||
| # Poll cadence. ISO-8601 duration (e.g. PT1H). Mandatory for the SDK. | ||
| duration_period: timedelta = Field(default=timedelta(hours=1), examples=["PT1H"]) |
Member
There was a problem hiding this comment.
Suggested change
| name: str = Field(default="Metras-Feed", examples=["Metras-Feed"]) | |
| # Poll cadence. ISO-8601 duration (e.g. PT1H). Mandatory for the SDK. | |
| duration_period: timedelta = Field(default=timedelta(hours=1), examples=["PT1H"]) | |
| id: str = Field( | |
| default="df7d629d-20a6-4a94-aa2b-86c5baacdda6", | |
| description="The unique identifier of the connector.", | |
| ) | |
| name: str = Field( | |
| default="Metras-Feed", | |
| description="The name of the connector.", | |
| examples=["Metras-Feed"], | |
| ) | |
| scope: ListFromString = Field( | |
| default=["Metras"], | |
| description="The scope of the connector.", | |
| examples=["Metras"], | |
| ) | |
| duration_period: timedelta = Field( | |
| default=timedelta(hours=1), | |
| description="ISO-8601 duration (e.g. 'PT1H') representing the period between connector runs.", | |
| examples=["PT1H"], | |
| ) |
Set defaults wherever it's possible/helpful.
Don't omit description for documentation.
|
|
||
| class MetrasConfig(BaseConfigModel): | ||
| api_base_url: HttpUrl = Field( | ||
| default="https://api.metras.sa/api", |
Member
There was a problem hiding this comment.
Suggested change
| default="https://api.metras.sa/api", | |
| default=HttpUrl("https://api.metras.sa/api"), |
nit: typing improvement
Comment on lines
+53
to
+57
| tlp_level: Literal["clear", "white", "green", "amber", "red"] = Field( | ||
| default="amber", | ||
| description="TLP marking applied to imported objects.", | ||
| examples=["amber"], | ||
| ) |
Member
There was a problem hiding this comment.
Suggested change
| tlp_level: Literal["clear", "white", "green", "amber", "red"] = Field( | |
| default="amber", | |
| description="TLP marking applied to imported objects.", | |
| examples=["amber"], | |
| ) | |
| tlp_level: TLPLevel = Field( | |
| default=TLPLevel.AMBER, | |
| description="TLP marking applied to imported objects.", | |
| examples=[TLPLevel.AMBER], | |
| ) |
I strongly encourage the use of TLPLevel enum from connectors_sdk.models.enums to avoid missing values, like amber+strict here.
Comment on lines
+57
to
+87
| # ------------------------------------------------------------------ # | ||
| def run(self) -> None: | ||
| try: | ||
| self.client.ping() | ||
| self.helper.connector_logger.info( | ||
| "[CONNECTOR] Metras API connection verified" | ||
| ) | ||
| except MetrasAPIError as exc: | ||
| self.helper.connector_logger.error( | ||
| "[CONNECTOR] Metras API ping failed at startup", | ||
| meta={"error": str(exc)}, | ||
| ) | ||
| sys.exit(1) | ||
|
|
||
| self.helper.connector_logger.info( | ||
| "[CONNECTOR] Starting Metras Feed import loop", | ||
| meta={"interval_seconds": self._interval}, | ||
| ) | ||
| while True: | ||
| try: | ||
| self._import_data() | ||
| except (KeyboardInterrupt, SystemExit): | ||
| self.helper.connector_logger.info("[CONNECTOR] Stopping") | ||
| break | ||
| except Exception as exc: # noqa: BLE001 - keep the loop alive | ||
| self.helper.connector_logger.error( | ||
| "[CONNECTOR] Import cycle crashed", meta={"error": str(exc)} | ||
| ) | ||
| time.sleep(self._interval) | ||
|
|
||
| # ------------------------------------------------------------------ # |
Member
There was a problem hiding this comment.
Suggested change
| # ------------------------------------------------------------------ # | |
| def run(self) -> None: | |
| try: | |
| self.client.ping() | |
| self.helper.connector_logger.info( | |
| "[CONNECTOR] Metras API connection verified" | |
| ) | |
| except MetrasAPIError as exc: | |
| self.helper.connector_logger.error( | |
| "[CONNECTOR] Metras API ping failed at startup", | |
| meta={"error": str(exc)}, | |
| ) | |
| sys.exit(1) | |
| self.helper.connector_logger.info( | |
| "[CONNECTOR] Starting Metras Feed import loop", | |
| meta={"interval_seconds": self._interval}, | |
| ) | |
| while True: | |
| try: | |
| self._import_data() | |
| except (KeyboardInterrupt, SystemExit): | |
| self.helper.connector_logger.info("[CONNECTOR] Stopping") | |
| break | |
| except Exception as exc: # noqa: BLE001 - keep the loop alive | |
| self.helper.connector_logger.error( | |
| "[CONNECTOR] Import cycle crashed", meta={"error": str(exc)} | |
| ) | |
| time.sleep(self._interval) | |
| # ------------------------------------------------------------------ # | |
| def run(self) -> None: | |
| """Start the connector and schedule its runs.""" | |
| self.helper.schedule_iso( | |
| message_callback=self.process, | |
| duration_period=self.config.connector.duration_period, # type: ignore[arg-type] | |
| ) | |
| def process(self) -> None: | |
| """Process a single import cycle.""" | |
| try: | |
| self.client.ping() | |
| self.helper.connector_logger.info( | |
| "[CONNECTOR] Metras API connection verified" | |
| ) | |
| except MetrasAPIError as exc: | |
| self.helper.connector_logger.error( | |
| "[CONNECTOR] Metras API ping failed at startup", | |
| meta={"error": str(exc)}, | |
| ) | |
| sys.exit(1) | |
| self.helper.connector_logger.info( | |
| "[CONNECTOR] Starting Metras Feed import loop" | |
| ) | |
| try: | |
| self._import_data() | |
| except (KeyboardInterrupt, SystemExit): | |
| self.helper.connector_logger.info( | |
| "[CONNECTOR] Connector stopped by user or system" | |
| ) | |
| sys.exit(0) | |
| except Exception as exc: | |
| self.helper.connector_logger.error( | |
| "[CONNECTOR] An unexpected error occurred, see connector logs for details", | |
| meta={"error": str(exc)}, | |
| ) | |
Please use the official scheduler from pycti instead of custom while loop.
Comment on lines
+37
to
+55
| self._interval = self._duration_seconds(config.connector.duration_period) | ||
|
|
||
| @staticmethod | ||
| def _duration_seconds(duration: timedelta | str) -> int: | ||
| """Resolve CONNECTOR_DURATION_PERIOD (timedelta or ISO8601 string) to seconds.""" | ||
| if hasattr(duration, "total_seconds"): | ||
| return max(60, int(duration.total_seconds())) | ||
| # Minimal ISO8601 duration parse (PT#H#M#S / P#D) as a fallback. | ||
| import re | ||
|
|
||
| text = str(duration or "PT1H").upper() | ||
| match = re.fullmatch( | ||
| r"P(?:(\d+)D)?(?:T(?:(\d+)H)?(?:(\d+)M)?(?:(\d+)S)?)?", text | ||
| ) | ||
| if not match: | ||
| return 3600 | ||
| days, hours, minutes, seconds = (int(g or 0) for g in match.groups()) | ||
| total = days * 86400 + hours * 3600 + minutes * 60 + seconds | ||
| return max(60, total or 3600) |
Member
There was a problem hiding this comment.
Suggested change
| self._interval = self._duration_seconds(config.connector.duration_period) | |
| @staticmethod | |
| def _duration_seconds(duration: timedelta | str) -> int: | |
| """Resolve CONNECTOR_DURATION_PERIOD (timedelta or ISO8601 string) to seconds.""" | |
| if hasattr(duration, "total_seconds"): | |
| return max(60, int(duration.total_seconds())) | |
| # Minimal ISO8601 duration parse (PT#H#M#S / P#D) as a fallback. | |
| import re | |
| text = str(duration or "PT1H").upper() | |
| match = re.fullmatch( | |
| r"P(?:(\d+)D)?(?:T(?:(\d+)H)?(?:(\d+)M)?(?:(\d+)S)?)?", text | |
| ) | |
| if not match: | |
| return 3600 | |
| days, hours, minutes, seconds = (int(g or 0) for g in match.groups()) | |
| total = days * 86400 + hours * 3600 + minutes * 60 + seconds | |
| return max(60, total or 3600) |
By using OpenCTIConnectorHelper.schedule_iso, self._interval is not necessary anymore.
Powlinett
reviewed
Jul 28, 2026
Member
There was a problem hiding this comment.
Please remove this file for now (generated automatically when the connector is "manager supported").
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #7039 +/- ##
===========================================
+ Coverage 35.64% 84.76% +49.12%
===========================================
Files 2048 10 -2038
Lines 125310 847 -124463
===========================================
- Hits 44672 718 -43954
+ Misses 80638 129 -80509
📢 Thoughts on this report? Let us know! 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
EXTERNAL_IMPORTconnector for the Metras EDR platform(
api.metras.sa,X-API-KEY) underexternal-import/metras/.Incident, withAttack-Pattern(uses) from the alertmitre_idsand
Urlobservables for external artifacts.StixFileobservables (banned/unsigned only by default to limit noise).Identity(identity_class="system"). Internal fleet assets aremodeled as System identities (internal IPs/hostnames kept in the description), not as
IOC observables or
Infrastructure, per maintainer guidance (PR [qualys] Add Qualys CVE enrichment connector #6164).last_occurrence_timefor the alerts endpoint(no server
fromTime), server-sidefromTimewindowing for binaries, full listing forendpoints. Deterministic STIX IDs via
pyctigenerate_id. Full Work-API lifecycle.pydantic.SecretStr; startup API ping; structured logging; CIS-hardeneddocker-compose.yml; non-rootpython:3.12-alpineimage withlibmagic.pyproject.tomlcoverage config for the
codecov/patchgate.black/isort --profile black/flake8 --ignore=E,Wclean.7.260529.0(pycti==7.260529.0,connectors-sdk @7.260529.0,support_version ">=7.260529.0").Related issues
Checklist
Further comments
Design notes driven by the Metras API: the EDR alerts endpoint exposes no
fromTimeparameter, so incremental import uses a client-side cursor on
last_occurrence_timeratherthan a server-side window; only the binaries endpoint supports
fromTime. Endpoints areemitted as System identities rather than observables to keep the IOC space clean — this is the
main modeling decision and is consistent with the accompanying Metras enrichment connector.