feat(xposedornot): add XposedOrNot internal-enrichment connector (#7000) - #6999
Open
DevaOnBreaches wants to merge 6 commits into
Open
feat(xposedornot): add XposedOrNot internal-enrichment connector (#7000)#6999DevaOnBreaches wants to merge 6 commits into
DevaOnBreaches wants to merge 6 commits into
Conversation
Enriches Email-Addr observables with data-breach exposure from the free, keyless XposedOrNot API (optional API key switches to the higher-limit Plus API). Enriches in place to keep graphs clean: risk-derived score, data-breach and plaintext-password-exposure labels, an external reference, and a markdown Note with the full breach table (dates, records, exposed data classes, password-storage risk). TLP-gated via XPOSEDORNOT_MAX_TLP (default TLP:AMBER) since the observable value is PII; clean results modify nothing. Includes __metadata__ manifest, config schema/doc, unit tests with fixtures, and docker-compose deployment.
Contributor License Agreement✅ CLA signed 💚 Thank you @DevaOnBreaches for signing the Contributor License Agreement! Your pull request can now be reviewed and merged. We appreciate your contribution to Filigran's open source projects! ❤️ This is an automated message from the Filigran CLA Bot. |
Filigran-Automation
changed the title
Member
|
🤖 [AI-generated] Hey @DevaOnBreaches! 👋 Thanks so much for opening PR #6999 — a free, no-API-key-required breach enrichment connector is a great contribution, and the amount of detail in your write-up (behavior, privacy/TLP handling, scaffold, and a solid test suite) is genuinely impressive! 🙏 One small thing that could help move things along: I couldn't find a linked GitHub issue referenced anywhere in the description (no "Closes #..." or similar). I haven't changed anything in your description — just a gentle suggestion:
No rush at all — thanks again for contributing XposedOrNot to the OpenCTI ecosystem! 🚀 |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new internal-enrichment/xposedornot connector to enrich Email-Addr observables with XposedOrNot breach exposure data (keyless community API by default, optional Plus API key), updating the observable in place and attaching a markdown summary Note.
Changes:
- Implement XposedOrNot API client with free/Plus paths, normalization, and retry/backoff behavior.
- Implement STIX conversion that generates a deterministic Note with a breach summary table and helper utilities.
- Add connector scaffold (metadata, Docker/compose/config samples) and unit tests with fixtures.
Reviewed changes
Copilot reviewed 25 out of 26 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| internal-enrichment/xposedornot/tests/test-requirements.txt | Adds pytest dependency for connector unit tests. |
| internal-enrichment/xposedornot/tests/test_converter.py | Tests Note rendering helpers and deterministic Note IDs. |
| internal-enrichment/xposedornot/tests/test_connector.py | Tests email validation and observable TLP extraction helpers. |
| internal-enrichment/xposedornot/tests/test_client_api.py | Tests client behavior across free/Plus API paths and error/retry branches. |
| internal-enrichment/xposedornot/tests/fixtures/plus_detailed.json | Adds fixture payload for Plus API detailed response. |
| internal-enrichment/xposedornot/tests/fixtures/breach_analytics.json | Adds fixture payload for community breach-analytics response. |
| internal-enrichment/xposedornot/src/xposedornot/converter_to_stix.py | Builds the markdown breach Note and helper formatting utilities. |
| internal-enrichment/xposedornot/src/xposedornot/connector.py | Implements enrichment flow, TLP gating, observable update-in-place, and Note attachment. |
| internal-enrichment/xposedornot/src/xposedornot/client_api.py | Implements HTTP client, normalization, rate-limit handling, and error logging. |
| internal-enrichment/xposedornot/src/xposedornot/init.py | Exposes connector entrypoint symbol. |
| internal-enrichment/xposedornot/src/requirements.txt | Defines runtime dependencies (pycti, requests, PyYAML, stix2, connectors-sdk). |
| internal-enrichment/xposedornot/src/main.py | Adds runnable module entrypoint for the container command. |
| internal-enrichment/xposedornot/src/init.py | Marks src as a package for python -m src. |
| internal-enrichment/xposedornot/README.md | Documents connector behavior, configuration, privacy notes, and usage. |
| internal-enrichment/xposedornot/pyproject.toml | Configures pytest paths and pythonpath. |
| internal-enrichment/xposedornot/Dockerfile | Adds container build instructions consistent with other connectors. |
| internal-enrichment/xposedornot/docker-compose.yml | Adds local deployment sample service definition. |
| internal-enrichment/xposedornot/config.yml.sample | Adds sample YAML configuration for manual runs. |
| internal-enrichment/xposedornot/.gitignore | Ignores local venv/cache/config artifacts. |
| internal-enrichment/xposedornot/.env.sample | Adds sample env var configuration. |
| internal-enrichment/xposedornot/.dockerignore | Excludes dev/metadata artifacts from Docker build context. |
| internal-enrichment/xposedornot/.build.env | Declares build env defaults for connector runtime. |
| internal-enrichment/xposedornot/metadata/connector_manifest.json | Adds connector manifest metadata for catalog distribution. |
| internal-enrichment/xposedornot/metadata/connector_config_schema.json | Adds JSON schema for connector configuration. |
| internal-enrichment/xposedornot/metadata/CONNECTOR_CONFIG_DOC.md | Adds generated configuration documentation table. |
Comment on lines
+89
to
+94
| tlp = observable_tlp(observable) | ||
| if tlp and not OpenCTIConnectorHelper.check_max_tlp(tlp, self.max_tlp): | ||
| return ( | ||
| "TLP of the observable (%s) is higher than what the connector is" | ||
| " allowed to enrich (%s); skipping." % (tlp, self.max_tlp) | ||
| ) |
Comment on lines
+134
to
+144
| if resp.status_code == 429: | ||
| retry_after = resp.headers.get("Retry-After") | ||
| wait = int(retry_after) if retry_after and retry_after.isdigit() else 15 | ||
| wait = min(wait, 60) | ||
| self.helper.connector_logger.warning( | ||
| "XposedOrNot rate limited (keyless: 2/s, 25/hour); backing off." | ||
| " An optional API key raises limits.", | ||
| meta={"wait_seconds": wait, "attempt": attempt}, | ||
| ) | ||
| time.sleep(wait) | ||
| continue |
handling Call check_max_tlp() unconditionally so unmarked observables are evaluated consistently (pycti allows None; the gate is now future-proof against helper changes). Tailor the 429 warning to the API actually in use (keyless vs Plus) and skip the pointless sleep on the final retry attempt. Adds tests for both.
Comment on lines
+68
to
+72
| lines = [ | ||
| "## XposedOrNot — breach exposure summary", | ||
| "", | ||
| "**Breaches found:** %d " % len(breaches), | ||
| ] |
Comment on lines
+128
to
+137
| if not any( | ||
| ref.get("source_name") == "XposedOrNot" for ref in external_references | ||
| ): | ||
| external_references.append( | ||
| stix2.ExternalReference( | ||
| source_name="XposedOrNot", | ||
| url="https://xposedornot.com", | ||
| description="XposedOrNot breach exposure check", | ||
| ) | ||
| ) |
Comment on lines
+66
to
+69
| cd src | ||
| pip3 install -r requirements.txt | ||
| cp ../config.yml.sample ../config.yml # then edit config.yml | ||
| python3 -m src |
…, dict external ref, drop unused dep, README fix Reviewer fixes: embed a hidden source-id marker so the Note id is unique per observable yet stable across re-enrichments (SDK derives the id from content/abstract only; two observables with identical breaches collided) with a regression test; append the external reference as a plain dict rather than a stix2 object; fix manual-deployment commands to run from the connector root. CI conformance: remove the now-unused stix2 dependency (deptry). Verified locally against the repo's isort/black/flake8/deptry and STIX-id pylint gates.
flow tests Address blocking CI on the new connector: migrate config loading from pycti get_config_variable to typed connectors-sdk BaseConnectorSettings (fixes VC305), keeping the API key optional so the keyless default is preserved. Add StubConnectorSettings and full _process_message flow tests raising patch coverage to ~95% (fixes codecov/patch). Drop unused PyYAML, add pydantic; sync config samples, metadata schema/doc and README, renaming XPOSEDORNOT_BASE_URL to XPOSEDORNOT_API_BASE_URL.
DevaOnBreaches
changed the title
|
…est taxonomy connectors-sdk on master now pins pycti==7.260722.0, so the connector's stale 7.260710.0 pin made `uv pip check` fail in CI before tests ran. Bump pycti and pytest to the repo-wide versions, and add the manifest fields introduced since this branch was cut (solution_categories, license_type, contact) with the current use_cases taxonomy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #7000
Proposed changes
This PR adds an internal-enrichment connector for XposedOrNot — an open, free data-breach search service tracking 760+ breaches. It enriches
Email-Addrobservables with breach exposure.No API key or registration required — the connector is fully functional via the free community API (rate limits: 2 req/s, 25/hour per IP; 429 is retried with backoff honoring
Retry-After, and persistent throttling fails only that enrichment with a log recommendation, never the connector). An optional key (console.xposedornot.com) switches to the Plus API with higher limits.Behavior — enrich in place, keep the graph clean
For a breached email, no new entities are created. The observable itself receives:
data-breach(+plaintext-password-exposurewhen at least one breach stored plaintext passwords),A clean email completes with an explicit "no known breach exposure" message and modifies nothing.
Privacy
The observable's email address is PII and is sent (TLS) to xposedornot.com. Enrichment is gated by
XPOSEDORNOT_MAX_TLP(defaultTLP:AMBER), results carry a configurable TLP marking (defaultamber), and the README/manifest carry a lawful-use (GDPR) statement — following the osint-industries precedent.Scaffold
Mirrors
internal-enrichment/osint-industries:__metadata__/(manifest with XTM Hub use-case taxonomy, config schema, config doc, logo),src/split intoclient_api/connector/converter_to_stix, pycti pinned to7.260710.0,connectors-sdkmodels (OrganizationAuthor, Note, Reference, TLPMarking). The source-observable update follows the fortisandboxstix_entity/stix_objectspattern; the TLP gate usesOpenCTIConnectorHelper.check_max_tlp.Testing
tests/): free + Plus API paths (including URL-encoding ofuser+tag@addresses and thex-api-keyheader), clean 404 and clean 200-empty, rate-limit retry/recovery/give-up, server error, invalid JSON, no-key-leak-in-logs, email validation, TLP extraction, Note content and deterministic IDs.black/isortclean.