feat(whoisfreaks): introduce WhoisFreaks connector and formatting updates - #7108
Open
qayyum-jfreaks wants to merge 30 commits into
Open
feat(whoisfreaks): introduce WhoisFreaks connector and formatting updates#7108qayyum-jfreaks wants to merge 30 commits into
qayyum-jfreaks wants to merge 30 commits into
Conversation
This README provides detailed information about the OpenCTI WhoisFreaks enrichment connector, including its features, configuration variables, and deployment instructions via Docker Compose.
This file is a compiled Python bytecode file for the config_variables module, which contains configuration settings for the WhoisFreaks OpenCTI Connector.
Contributor License Agreement✅ CLA signed 💚 Thank you @qayyum-jfreaks for signing the Contributor License Agreement! Your pull request can now be reviewed and merged. We appreciate your contribution to Filigran's open source projects! ❤️ This is an automated message from the Filigran CLA Bot. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR primarily introduces a new whoisfreaks internal-enrichment connector (implementation, packaging, and tests) and applies a series of formatting-only refactors (wrapping multiline strings in parentheses) across several existing connectors and test files.
Changes:
- Added a full
internal-enrichment/whoisfreaksconnector implementation (client, STIX builder, config loader, Docker artifacts, metadata). - Added initial unit tests for the WhoisFreaks connector (connector flow, client HTTP wrappers, STIX builder).
- Applied consistent multiline-string formatting updates in multiple existing connectors/tests.
Reviewed changes
Copilot reviewed 28 out of 35 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| shared/pylint_plugins/tests/test_check_stix_plugin/test_linter_stix_generator.py | Formatting-only updates to multiline string parsing in plugin tests. |
| internal-import-file/import-document-ai/src/import_doc_ai/client_api.py | Formatting-only update to GraphQL query call chaining. |
| internal-enrichment/whoisfreaks/tests/test_connector.py | Adds connector-level unit tests with mocked OpenCTI helper/config. |
| internal-enrichment/whoisfreaks/tests/test_client.py | Adds unit tests for WhoisFreaks REST client using requests_mock. |
| internal-enrichment/whoisfreaks/tests/test_builder.py | Adds unit tests validating STIX bundle construction. |
| internal-enrichment/whoisfreaks/tests/conftest.py | Adds pytest path setup for importing connector src/ modules. |
| internal-enrichment/whoisfreaks/src/main.py | Implements the WhoisFreaks connector entrypoint and message processing loop. |
| internal-enrichment/whoisfreaks/src/configVariables.py | Adds YAML/env configuration loading and validation. |
| internal-enrichment/whoisfreaks/src/client.py | Implements REST API client wrappers for WhoisFreaks endpoints. |
| internal-enrichment/whoisfreaks/src/builder.py | Implements STIX 2.1 bundle building from WhoisFreaks responses. |
| internal-enrichment/whoisfreaks/src/init.py | Adds package marker file for the connector source directory. |
| internal-enrichment/whoisfreaks/requirements.txt | Declares Python dependencies for the new connector. |
| internal-enrichment/whoisfreaks/README.md | Adds connector documentation and deployment instructions. |
| internal-enrichment/whoisfreaks/entrypoint.sh | Adds container entrypoint script. |
| internal-enrichment/whoisfreaks/Dockerfile | Adds Docker build for the new connector. |
| internal-enrichment/whoisfreaks/docker-compose.yml | Adds a local dev stack compose file including OpenCTI dependencies. |
| internal-enrichment/whoisfreaks/config.yml | Adds example YAML configuration for local usage. |
| internal-enrichment/whoisfreaks/metadata/connector_manifest.json | Adds connector manifest metadata (title/slug/image/etc.). |
| internal-enrichment/whoisfreaks/metadata/connector_config_schema.json | Adds connector config schema metadata. |
| internal-enrichment/reversinglabs-spectra-intel-submission/src/connector/connector.py | Formatting-only updates to multiline string handling. |
| internal-enrichment/reversinglabs-spectra-analyze/src/main.py | Formatting-only updates to multiline string handling. |
| internal-enrichment/reversinglabs-malware-presence/src/connector/connector.py | Formatting-only updates to multiline string handling. |
| internal-enrichment/polyswarm-sandbox/tests/test_e2e_sandbox.py | Formatting-only updates to multiline string query construction. |
| internal-enrichment/polyswarm-enrichment/tests/test_e2e_opencti.py | Formatting-only updates to multiline string query construction. |
| internal-enrichment/polyswarm-enrichment/tests/test_e2e_network_iocs.py | Formatting-only updates to multiline string query construction. |
| internal-enrichment/polyswarm-enrichment/tests/test_e2e_keydoor.py | Formatting-only updates to multiline string query construction. |
| internal-enrichment/import-external-reference/src/import-external-reference.py | Formatting-only updates to multiline JS strings in Playwright usage. |
| external-import/tenable-vuln-management/tests/test_converter_to_stix.py | Formatting-only updates to multiline JSON strings in tests. |
| external-import/comlaude/src/comlaude/init.py | Formatting-only updates to multiline exception string formatting. |
Comments suppressed due to low confidence (1)
internal-enrichment/whoisfreaks/src/client.py:58
- Same issue as
_post():_get()defaults to a hard-coded 30s timeout and does not useself.timeout, so the constructortimeoutparameter has no effect for GET requests.
def _get(
self, endpoint: str, params: Optional[dict] = None, timeout: int = 30
) -> Optional[Dict[str, Any]]:
| @@ -0,0 +1,250 @@ | |||
| import pytest | |||
| import stix2 | |||
| from whoisfreaks.src.builder import WhoisFreaksStixBuilder | |||
| @@ -0,0 +1,319 @@ | |||
| import pytest | |||
| import requests | |||
| import requests_mock | |||
Author
There was a problem hiding this comment.
test-requirements.txt added
Comment on lines
+28
to
+41
| def _post( | ||
| self, | ||
| endpoint: str, | ||
| params: Optional[dict] = None, | ||
| body: Optional[dict] = None, | ||
| timeout: int = 30, | ||
| ) -> Optional[Dict[str, Any]]: | ||
| url = f"{self.BASE_URL}{endpoint}" | ||
| if params is None: | ||
| params = {} | ||
| params["apiKey"] = self.api_key | ||
|
|
||
| try: | ||
| response = self.session.post(url, params=params, json=body, timeout=timeout) |
Comment on lines
+162
to
+169
| def bulk_dns_lookup( | ||
| self, domains: list[str], ipAddresses: list[str], format: Optional[str] = "json" | ||
| ) -> Optional[Dict[str, Any]]: | ||
| return self._post( | ||
| "/v2.0/dns/bulk/live", | ||
| params={"format": format, "type": "all"}, | ||
| body={"domainNames": domains, "ipAddresses": ipAddresses}, | ||
| ) |
Comment on lines
+113
to
+115
| entity_id = msg.get("entity_id") | ||
| observable_type, observable_value = self._get_entity_info(entity_id) | ||
|
|
Comment on lines
+50
to
+52
| self.connector_auto = get_config_variable( | ||
| "CONNECTOR_AUTO", ["connector", "auto"], config | ||
| ) |
Comment on lines
+1
to
+19
| { | ||
| "$schema": "http://json-schema.org/draft-07/schema#", | ||
| "title": "WhoisFreaks connector configuration schema", | ||
| "type": "object", | ||
| "properties": { | ||
| "whoisfreaks_api_key": { | ||
| "type": "string", | ||
| "description": "API key for accessing the WhoisFreaks service", | ||
| "minLength": 1 | ||
| }, | ||
| "whoisfreaks_url": { | ||
| "type": "string", | ||
| "description": "Optional base URL for the WhoisFreaks API", | ||
| "format": "uri" | ||
| } | ||
| }, | ||
| "required": ["whoisfreaks_api_key"], | ||
| "additionalProperties": false | ||
| } |
| "slug": "whoisfreaks", | ||
| "description": "An official internal enrichment connector for OpenCTI powered by WhoisFreaks API. Enriches Domain-Name, IPv4-Addr, and IPv6-Addr observables with live WHOIS registration data, DNS records, SSL/TLS certificate metadata, IP geolocation, and threat context.", | ||
| "short_description": "Enrich domain, IP, and SSL observables with live WHOIS, DNS, Geolocation, and threat intelligence from WhoisFreaks.", | ||
| "logo": "https://whoisfreaks.com/images/logo.png", |
| - MINIO__PORT=9000 | ||
| - MINIO__USE_SSL=false | ||
| - MINIO__ACCESS_KEY=opencti | ||
| - MINIO__SECRET_KEY=opencti_minio_password |
…dling, and test deps
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 36 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (4)
internal-enrichment/whoisfreaks/docker-compose.yml:43
- This docker-compose hardcodes MINIO__SECRET_KEY (opencti_minio_password). Even for local stacks, committing secrets encourages reuse and can leak into real deployments; other OpenCTI compose examples in this repo source it from environment variables instead.
- MINIO__SECRET_KEY=opencti_minio_password
internal-enrichment/whoisfreaks/metadata/connector_manifest.json:6
- Connector manifests in this repo reference a repository-local logo asset (e.g., internal-enrichment/sophoslabs-intelix/metadata/connector_manifest.json:6). This manifest points to an external URL, and there is no logo file in internal-enrichment/whoisfreaks/metadata/ (only the two JSON files are present), which is likely to break manifest validation/offline usage.
"logo": "https://whoisfreaks.com/images/logo.png",
internal-enrichment/whoisfreaks/metadata/connector_config_schema.json:5
- This config schema doesn't follow the repository's connector schema conventions (most use JSON Schema 2020-12 with $id and uppercase env-var keys like OPENCTI_URL / OPENCTI_TOKEN; see internal-enrichment/polyswarm-enrichment/metadata/connector_config_schema.json:1-55). The current draft-07 schema with lowercase keys (whoisfreaks_api_key, whoisfreaks_url) is inconsistent and may fail the manifest/schema validation pipeline.
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "WhoisFreaks connector configuration schema",
"type": "object",
"properties": {
internal-enrichment/whoisfreaks/src/client.py:168
- Parameter name ipAddresses is camelCase, which is inconsistent with Python naming conventions and makes the client API harder to read (especially when used with keyword arguments).
self, domains: list[str], ipAddresses: list[str], format: Optional[str] = "json"
) -> Optional[Dict[str, Any]]:
Comment on lines
+18
to
+26
| # Copy source code and entrypoint | ||
| COPY src ./src | ||
| COPY entrypoint.sh . | ||
|
|
||
| # Ensure entrypoint is executable | ||
| RUN chmod +x entrypoint.sh | ||
|
|
||
| # Run startup script | ||
| ENTRYPOINT ["/entrypoint.sh"] No newline at end of file |
Comment on lines
+210
to
+219
| m.get( | ||
| "https://api.whoisfreaks.com/v3.0/domain/tyops?apiKey=test-api-key&keyword=example", | ||
| json={"data": "typo"}, | ||
| ) | ||
| # Note: in client.py line 188: self._get("/v3.0/domain/typos", params={"keyword": keyword}) | ||
| # Wait, the path is typos, let's fix URL mock: | ||
| m.get( | ||
| "https://api.whoisfreaks.com/v3.0/domain/typos?apiKey=test-api-key&keyword=example", | ||
| json={"data": "typo"}, | ||
| ) |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 38 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
internal-enrichment/whoisfreaks/src/main.py:116
- Lines 115-116 are indented one space too far under
if not entity_id:, which will raise anIndentationErrorand prevent the connector (and tests) from running.
Comment on lines
+5
to
+8
| from builder import WhoisFreaksStixBuilder | ||
| from client import WhoisFreaksClient | ||
| from configVariables import ConfigVariables | ||
| from pycti import OpenCTIConnectorHelper |
Comment on lines
+1
to
+6
| { | ||
| "title": "WhoisFreaks", | ||
| "slug": "whoisfreaks", | ||
| "description": "An official internal enrichment connector for OpenCTI powered by WhoisFreaks API. Enriches Domain-Name, IPv4-Addr, and IPv6-Addr observables with live WHOIS registration data, DNS records, SSL/TLS certificate metadata, IP geolocation, and threat context.", | ||
| "short_description": "Enrich domain, IP, and SSL observables with live WHOIS, DNS, Geolocation, and threat intelligence from WhoisFreaks.", | ||
| "logo": "internal-enrichment/whoisfreaks/__metadata__/logo.png", |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 38 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
internal-enrichment/whoisfreaks/src/builder.py:392
- Using
orto select the reputationscorewill treat a valid score of0as falsy, which can incorrectly skip creating the Note for a legitimate zero-score result.
score = reputation_data.get("reputation_score") or reputation_data.get("score")
if score is not None:
Comment on lines
+43
to
+60
| ```yaml | ||
| connector-whoisfreaks: | ||
| image: opencti-connector-whoisfreaks:latest | ||
| container_name: opencti-connector-whoisfreaks | ||
| environment: | ||
| - OPENCTI_URL=http://opencti:4000 | ||
| - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN} | ||
| - CONNECTOR_ID=${CONNECTOR_KEY} | ||
| - CONNECTOR_TYPE=INTERNAL_ENRICHMENT | ||
| - CONNECTOR_NAME=WhoisFreaks | ||
| - CONNECTOR_SCOPE=Domain-Name,IPv4-Addr,IPv6-Addr | ||
| - CONNECTOR_AUTO=false | ||
| - CONNECTOR_CONFIDENCE_LEVEL=100 | ||
| - CONNECTOR_LOG_LEVEL=INFO | ||
| - WHOISFREAKS_API_KEY=${WHOISFREAKS_API_KEY} | ||
| depends_on: | ||
| - opencti | ||
| restart: always |
Author
There was a problem hiding this comment.
already done, nothing changed
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 38 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
internal-enrichment/whoisfreaks/README.md:43
- The Docker Compose example opens a fenced code block but never closes it, so the remainder of the README renders as YAML.
```yaml
Comment on lines
+231
to
+239
| issuer = cert_info.get("issuer_dn") or "Unknown Issuer" | ||
| subject = cert_info.get("subject_dn") or clean_target | ||
| serial = str(cert_info.get("serial_number", "")) | ||
|
|
||
| cert_stix = stix2.X509Certificate( | ||
| issuer=str(issuer), | ||
| subject=str(subject), | ||
| serial_number=serial, | ||
| ) |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…connectors into fix/whoisfreaks-manifest-tests
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 38 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
internal-enrichment/whoisfreaks/metadata/connector_config_schema.json:74
CONNECTOR_IDshould be included in the schema’srequiredlist to match the connector’s own validation (and README), otherwise the schema implies the connector can run without it even though it will fail at startup.
"required": [
"OPENCTI_URL",
"OPENCTI_TOKEN",
"WHOISFREAKS_API_KEY"
],
Comment on lines
+34
to
+42
| self.connector_type = get_config_variable( | ||
| "CONNECTOR_TYPE", ["connector", "type"], config | ||
| ) | ||
| self.connector_name = get_config_variable( | ||
| "CONNECTOR_NAME", ["connector", "name"], config | ||
| ) | ||
| self.connector_scope = get_config_variable( | ||
| "CONNECTOR_SCOPE", ["connector", "scope"], config | ||
| ) |
Comment on lines
+17
to
+21
| "CONNECTOR_NAME": { | ||
| "default": "WhoisFreaks", | ||
| "description": "The name of the connector.", | ||
| "type": "string" | ||
| }, |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 38 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
internal-enrichment/whoisfreaks/README.md:57
- The Docker Compose example sets
CONNECTOR_LOG_LEVEL=INFO, but the connector config schema enums forCONNECTOR_LOG_LEVELare lowercase (debug/info/warn/warning/error). Changing this example to lowercase avoids schema mismatch for users who copy/paste it.
- CONNECTOR_AUTO=false
- CONNECTOR_CONFIDENCE_LEVEL=100
- CONNECTOR_LOG_LEVEL=INFO
- WHOISFREAKS_API_KEY=${WHOISFREAKS_API_KEY}
Comment on lines
+31
to
+35
| | `CONNECTOR_SCOPE` | No | `Domain-Name,IPv4-Addr,IPv6-Addr` | Supported STIX 2.1 entity types. | | ||
| | `CONNECTOR_AUTO` | No | `false` | Enable automatic enrichment on observable creation (`true`/`false`). | | ||
| | `CONNECTOR_CONFIDENCE_LEVEL` | No | `100` | Confidence score (0-100) assigned to created STIX objects. | | ||
| | `CONNECTOR_LOG_LEVEL` | No | `INFO` | Log verbosity (`DEBUG`, `INFO`, `WARNING`, `ERROR`). | | ||
| | `WHOISFREAKS_API_KEY` | Yes | - | Your active WhoisFreaks API key. | |
Comment on lines
+62
to
+66
| # Force log level to uppercase string (e.g. 'INFO') | ||
| raw_log_level = get_config_variable( | ||
| "CONNECTOR_LOG_LEVEL", ["connector", "log_level"], config, default="INFO" | ||
| ) | ||
| self.connector_log_level = ( |
Comment on lines
+10
to
+13
| auto: false # Manual trigger recommended to manage API quota | ||
| confidence_level: 100 | ||
| log_level: "INFO" | ||
|
|
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 38 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (4)
internal-enrichment/whoisfreaks/README.md:56
- The Docker Compose example sets
CONNECTOR_LOG_LEVEL=INFO(uppercase), but this connector’s config schema enumerates lowercase values (debug/info/warn/warning/error). Copy/pasting the current example can fail schema validation in OpenCTI Manager setups; use lowercase in docs.
- CONNECTOR_LOG_LEVEL=INFO
internal-enrichment/whoisfreaks/config.yml:12
connector.log_levelis set to "INFO" (uppercase) in the sample config, but the connector’s config schema enumerates lowercase values. Using lowercase keeps the examples consistent and avoids schema-validation issues for users who copy/paste config values into env vars.
log_level: "INFO"
internal-enrichment/whoisfreaks/docker-compose.yml:83
- This docker-compose example sets
CONNECTOR_LOG_LEVEL=INFO(uppercase), but the connector’s config schema enumerates lowercase values. Using lowercase here avoids schema-validation issues when users reuse the same value across config mechanisms.
- CONNECTOR_LOG_LEVEL=INFO
internal-enrichment/whoisfreaks/src/config_variables.py:51
CONNECTOR_SCOPEdefaults to a Python list here, while most connectors treat it as a comma-separated string coming from env/config (e.g.internal-enrichment/malbeacon/src/malbeacon_config_variables.py:38), and this connector’s ownconfig.yml/README use the string form. Keeping the default as a string avoids passing an unexpected type intoOpenCTIConnectorHelperwhen the user doesn’t setCONNECTOR_SCOPE.
self.connector_scope = get_config_variable(
"CONNECTOR_SCOPE",
["connector", "scope"],
config,
default=["Domain-Name", "IPv4-Addr", "IPv6-Addr"],
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 38 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
internal-enrichment/whoisfreaks/tests/test_client.py:19
requests_mock_fixtureis not used, and including it as a parameter makes the test signature misleading (it looks like a real pytest fixture dependency). It can be removed to keep the test intent clear.
Author
removed |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #7108 +/- ##
===========================================
+ Coverage 35.64% 59.58% +23.93%
===========================================
Files 2048 143 -1905
Lines 125310 10184 -115126
===========================================
- Hits 44672 6068 -38604
+ Misses 80638 4116 -76522
📢 Thoughts on this report? Let us know! 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Related issues
Checklist
Further comments