feat(domaintools): add DomainTools IrisQL and Feeds connector for OpenCTI (#7128) (#7127)

[nnguyen-1]

• Implemented the DomainTools IrisQL connector to fetch data from DomainTools Iris Investigate and import it into OpenCTI.
• Implemented the DomainTools Feeds connector to fetch data from DomainTools and import it into OpenCTI.
• Created necessary files including Dockerfile, entrypoint script, and configuration files.
• Developed core functionality for data retrieval, STIX object conversion, and error handling.
• Added requirements for dependencies including pycti, pydantic, requests, and validators.
• Included README documentation for installation, configuration, and usage instructions.
• Established a client for API interactions and a converter for STIX object creation.
• Implemented logging for better debugging and operational visibility.

Related issues

• Closes feat(domaintools-feeds): create external-import connector #7128
• Closes feat(domaintools-irisql): create external-import connector #7127

Checklist

• [ x] I consider the submitted work as finished
• [ x] I have signed my commits using GPG key.
• [ x] I tested the code for its functionality using different use cases
• [ x] I added/update the relevant documentation (either on github or on notion)
• [ x] Where necessary I refactored code to improve the overall quality

[Copilot]

Pull request overview

This PR introduces new OpenCTI external-import connectors for DomainTools, including an Iris Investigate IrisQL connector (as described) and an additional DomainTools Feeds connector (scope exceeds the PR title/description).

Changes:

• Added a DomainTools IrisQL connector to fetch Iris Investigate results, convert them to STIX, and ingest into OpenCTI.
• Added a DomainTools Feeds connector to ingest DomainTools Threat Feeds and import them as STIX Observables/Indicators.
• Added containerization and deployment assets (Dockerfiles, entrypoints, compose samples) plus end-user documentation for both connectors.

Reviewed changes

Copilot reviewed 32 out of 34 changed files in this pull request and generated 19 comments.

Show a summary per file

File	Description
external-import/domaintools-irisql/src/requirements.txt	Adds Python dependencies for the IrisQL connector runtime.
external-import/domaintools-irisql/src/main.py	Provides the connector entrypoint wiring settings + helper + run loop.
external-import/domaintools-irisql/src/connector/utils.py	Placeholder utilities module for future shared helpers.
external-import/domaintools-irisql/src/connector/settings.py	Defines pydantic/connectors-sdk configuration models for IrisQL connector.
external-import/domaintools-irisql/src/connector/converter_to_stix.py	Implements STIX conversion helpers (author, markings, relationships, observables, notes).
external-import/domaintools-irisql/src/connector/connector.py	Implements the IrisQL ingestion loop, STIX bundling, and OpenCTI work handling.
external-import/domaintools-irisql/src/connector/init.py	Exposes connector + settings for import from main.
external-import/domaintools-irisql/src/client/api_client.py	Implements DomainTools Iris Investigate API client/pagination for IrisQL results.
external-import/domaintools-irisql/src/client/init.py	Exposes the IrisQL API client.
external-import/domaintools-irisql/README.md	Documents install/config/usage for the IrisQL connector.
external-import/domaintools-irisql/entrypoint.sh	Container entrypoint to run the connector.
external-import/domaintools-irisql/Dockerfile	Container build for the IrisQL connector.
external-import/domaintools-irisql/docker-compose.yml	Example docker-compose service definition for IrisQL connector.
external-import/domaintools-irisql/config.yml.sample	Example manual-deployment configuration for IrisQL connector.
external-import/domaintools-irisql/.dockerignore	Excludes local artifacts from Docker build context.
external-import/domaintools-irisql/metadata/connector_manifest.json	Adds connector manifest metadata for IrisQL connector.
external-import/domaintools-feeds/src/requirements.txt	Adds Python dependencies for the Feeds connector runtime.
external-import/domaintools-feeds/src/main.py	Provides the connector entrypoint wiring settings + helper + run loop.
external-import/domaintools-feeds/src/connector/utils.py	Placeholder utilities module for future shared helpers.
external-import/domaintools-feeds/src/connector/settings.py	Defines pydantic/connectors-sdk configuration models for Feeds connector.
external-import/domaintools-feeds/src/connector/converter_to_stix.py	Implements STIX conversion helpers for feed entries and indicators.
external-import/domaintools-feeds/src/connector/connector.py	Implements the Feeds ingestion loop, STIX bundling, and OpenCTI work handling.
external-import/domaintools-feeds/src/connector/init.py	Exposes connector + settings for import from main.
external-import/domaintools-feeds/src/client/api_client.py	Implements DomainTools Threat Feeds API client.
external-import/domaintools-feeds/src/client/init.py	Exposes the Feeds API client.
external-import/domaintools-feeds/README.md	Documents install/config/usage for the Feeds connector.
external-import/domaintools-feeds/entrypoint.sh	Container entrypoint to run the connector.
external-import/domaintools-feeds/Dockerfile	Container build for the Feeds connector.
external-import/domaintools-feeds/docker-compose.yml	Example docker-compose service definition for Feeds connector.
external-import/domaintools-feeds/config.yml.sample	Example manual-deployment configuration for Feeds connector.
external-import/domaintools-feeds/.dockerignore	Excludes local artifacts from Docker build context.
external-import/domaintools-feeds/metadata/connector_manifest.json	Adds connector manifest metadata for Feeds connector.

Comments suppressed due to low confidence (5)

external-import/domaintools-irisql/src/connector/converter_to_stix.py:180

• TLP marking is created but not applied to the generated observables. In this repo, observables/indicators are typically created with object_marking_refs (e.g. external-import/dshield/src/connector/converter_to_stix.py:73-75). Without it, imported entities may be unmarked in OpenCTI.

        elif self._is_ipv4(value) is True:
            stix_ipv4_address = stix2.IPv4Address(
                value=value,
                custom_properties={
                    "x_opencti_created_by_ref": self.author["id"],
                    "x_opencti_labels": labels,
                },
            )

external-import/domaintools-irisql/src/connector/converter_to_stix.py:188

• TLP marking is created but not applied to the generated observables. In this repo, observables/indicators are typically created with object_marking_refs (e.g. external-import/dshield/src/connector/converter_to_stix.py:73-75). Without it, imported entities may be unmarked in OpenCTI.

        elif self._is_email(value) is True:
            stix_email = stix2.EmailAddress(
                value=value,
                custom_properties={
                    "x_opencti_created_by_ref": self.author["id"],
                },
            )

external-import/domaintools-irisql/src/connector/converter_to_stix.py:198

• TLP marking is created but not applied to the generated observables. In this repo, observables/indicators are typically created with object_marking_refs (e.g. external-import/dshield/src/connector/converter_to_stix.py:73-75). Without it, imported entities may be unmarked in OpenCTI.

        elif self._is_domain(value) is True:
            stix_domain_name = stix2.DomainName(
                value=value,
                custom_properties={
                    "x_opencti_created_by_ref": self.author["id"],
                    "x_opencti_score": score,
                    "x_opencti_labels": labels,
                },
            )

external-import/domaintools-feeds/src/connector/converter_to_stix.py:170

• TLP marking is created but not applied to the generated observables. In this repo, observables/indicators are typically created with object_marking_refs (e.g. external-import/dshield/src/connector/converter_to_stix.py:73-75). Without it, imported entities may be unmarked in OpenCTI.

        elif self._is_ipv4(value) is True:
            stix_ipv4_address = stix2.IPv4Address(
                value=value,
                custom_properties={
                    "x_opencti_created_by_ref": self.author["id"],
                },
            )

external-import/domaintools-feeds/src/connector/converter_to_stix.py:179

• TLP marking is created but not applied to the generated observables. In this repo, observables/indicators are typically created with object_marking_refs (e.g. external-import/dshield/src/connector/converter_to_stix.py:73-75). Without it, imported entities may be unmarked in OpenCTI.

        elif self._is_domain(value) is True:
            stix_domain_name = stix2.DomainName(
                value=value,
                custom_properties={
                    "x_opencti_created_by_ref": self.author["id"],
                    "x_opencti_labels": labels,
                },
            )

[nnguyen-1]

done

[Copilot]

Pull request overview

Copilot reviewed 32 out of 34 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (10)

external-import/domaintools-feeds/src/connector/converter_to_stix.py:102

• created_by_ref must be an identifier reference (string), but this sets it to the full Identity object (self.author). This will produce invalid STIX and can break bundle ingestion.

            relationship_type=relationship_type,
            source_ref=source_id,
            target_ref=target_id,
            created_by_ref=self.author,
        )

external-import/domaintools-feeds/src/connector/converter_to_stix.py:210

• create_indicator() accepts a confidence parameter but never sets the STIX Indicator "confidence" field, so the value is silently dropped.

            stix_domain_name = stix2.Indicator(
                name=f"Indicator for {value}",
                id=indicator_id,
                pattern_type="stix",
                pattern=pattern,
                labels=labels,
                valid_from=timestamp,
                object_marking_refs=[self.tlp_marking],
                custom_properties={

external-import/domaintools-irisql/src/client/api_client.py:64

• get_entities() calls response.raise_for_status() before checking whether response is None. If _request_data() returns None (on a RequestException), this raises AttributeError and hides the real request failure.

            while True:
                response = self._request_data(self.base_url, params=params, body=body)
                response.raise_for_status()
                if response is None:
                     raise RuntimeError("Failed to fetch IrisQL results from DomainTools API")

external-import/domaintools-feeds/src/client/api_client.py:70

• If _request_data() returns None, this raise is indented inconsistently and the function later contains unreachable template code. Additionally, returning None on exception will break callers that assume a list (len/iteration).

            response = self._request_data(self.base_url, params=params)
            if response is None:
                 raise RuntimeError("Failed to fetch DomainTools feed entries")
            return response.text.splitlines()

external-import/domaintools-feeds/src/client/api_client.py:81

• This function still includes unreachable template code (raise NotImplementedError) and the except block logs then implicitly returns None. Re-raise so upstream code can fail fast instead of hitting TypeError later.

            raise NotImplementedError

        except Exception as err:
            self.helper.connector_logger.error(err)

external-import/domaintools-irisql/src/connector/converter_to_stix.py:213

• create_note_id() has an unused created_time parameter and a mis-indented closing parenthesis, which makes the signature look accidental and will be reformatted by black. Since you intentionally generate deterministic Note IDs with created=None, dropping the unused parameter keeps the API clearer.

    def create_note_id(
        self, content: str, abstract: str, created_time: str | None = None
     ) -> str:
        return Note.generate_id(created=None, content=content, abstract=abstract)

external-import/domaintools-irisql/src/connector/converter_to_stix.py:68

• TLP:CLEAR is currently mapped to stix2.TLP_WHITE, which makes "clear" indistinguishable from "white" in OpenCTI. OpenCTI uses a custom MarkingDefinition for TLP:CLEAR (different ID/definition) so the connector should build a custom marking for "clear".

    def _create_tlp_marking(level):
        mapping = {
            "white": stix2.TLP_WHITE,
            "clear": stix2.TLP_WHITE,
            "green": stix2.TLP_GREEN,
            "amber": stix2.TLP_AMBER,

external-import/domaintools-feeds/src/connector/converter_to_stix.py:68

• TLP:CLEAR is currently mapped to stix2.TLP_WHITE, which makes "clear" indistinguishable from "white" in OpenCTI. OpenCTI uses a custom MarkingDefinition for TLP:CLEAR (different ID/definition) so the connector should build a custom marking for "clear".

This issue also appears on line 202 of the same file.

    def _create_tlp_marking(level):
        mapping = {
            "white": stix2.TLP_WHITE,
            "clear": stix2.TLP_WHITE,
            "green": stix2.TLP_GREEN,
            "amber": stix2.TLP_AMBER,

external-import/domaintools-irisql/src/client/api_client.py:86

• After successfully returning result_data, the function still contains unreachable template code (raise NotImplementedError). Also, the except block logs the exception but then implicitly returns None, which will later break callers expecting a list (e.g., len(entities)). Re-raise so the caller can handle the failure explicitly.

            raise NotImplementedError

        except Exception as err:
            self.helper.connector_logger.error(err)

external-import/domaintools-irisql/src/connector/connector.py:111

• The continue statement is indented by a non-standard amount (one extra space compared to the surrounding block). This will be reformatted by black/isort checks and is easy to fix now to keep formatting consistent.

            domain_obs = self.converter_to_stix.create_obs(domain, score, labels)
            if not domain_obs:
                 continue
            stix_objects.append(domain_obs)

[romain-filigran]

Thank you @nnguyen-1 for your contribution