diff --git a/DEMO.md b/DEMO.md new file mode 100644 index 00000000..5187d7a3 --- /dev/null +++ b/DEMO.md @@ -0,0 +1,380 @@ +# Compliance demos — PII redaction, secrets, & retention on the memory hook seam (issue #275) + +This is a **presentation branch**: it is `origin/main` plus a small set of demo +files. It shows how to USE the now-merged memory hook seam for the two +enterprise asks in +[issue #275](https://github.com/AgentToolkit/altk-evolve/issues/275), plus a +third, related method (structured secrets): + +1. **PII must never be saved into memory.** → PII redaction plugins on the seam. +2. **Memory must support data-retention policies** (delete/flag by age or + disuse; delete old sessions and the memories derived from them). → the + retention engine + `evolve retention run`. +3. **Credentials/tokens must not be saved either** (adjacent to #275). → the + structured-secrets plugin. + +All three are wired the same way: as **plugins on the memory hook seam** (merged +in [PR #295](https://github.com/AgentToolkit/altk-evolve/pull/295)). The seam is +always live; behavior is decided entirely by which plugins you configure. Hook +points sit at the backend read/write choke points and before every LLM call, so +a redactor fires on `memory_pre_write` (before anything is persisted) and on +`llm_pre_call` (before any completion call site sees the content). + +Three redaction methods ship, and they compose (defence in depth): + +| Method | Plugin / extra | Catches | Misses | +|---|---|---|---| +| **Regex PII** | `PIIFilterMemoryPlugin` · `[pii-regex]` | structured identifiers (email, phone, SSN, card, IP) at precision ~1.00 | **names** and non-Western formats (no NER) | +| **Semantic PII** | `ReadiSemanticPIIPlugin` · `[pii-semantic]` | free-form **names**/locations via IBM READI NER | language-bound; needs a language-matched model; heavy | +| **Structured secrets** | `SecretsFilterMemoryPlugin` · `[secrets]` | credentials/tokens (AWS keys, GitHub/Slack tokens, private keys) | entropy/heuristic secrets (off by default — over-redact) | + +--- + +## Install + +```bash +# from the repo root, on branch feat/275-pii-retention-poc +uv sync --extra hooks --extra pii-regex --extra pii-semantic --extra secrets +# then always run with --no-sync so uv doesn't re-resolve: +uv run --no-sync python examples/.py +``` + +`[pii-semantic]` downloads a spaCy transformer (~460MB) on first use. On +macOS/Apple-Silicon it fails closed on the MPS backend — see +[Limitations](#limitations). If you only want the mac-friendly path, drop +`--extra pii-semantic` and install `--extra hooks --extra pii-regex --extra secrets`. + +--- + +## How to use it — two equivalent ways to configure the seam + +### 1. YAML (`evolve hooks init`) — edit YAML to change behavior + +```bash +evolve hooks init # scaffolds ./evolve.hooks.yaml +``` + +Evolve **auto-discovers** this file (`$EVOLVE_HOOKS_CONFIG` → `./evolve.hooks.yaml` +→ `~/.config/evolve/hooks.yaml`), so every `EvolveClient` in the project picks it +up with no code change. The scaffold ships the **semantic** plugin active and the +regex + secrets plugins commented out. Changing posture is a YAML edit — comment +a plugin out (or set `mode: disabled`) to turn it off; uncomment the regex and +secrets blocks to run all three together. See +[`examples/hooks_plugins.yaml`](examples/hooks_plugins.yaml) for the full, +annotated plugin reference. + +`mode: sequential` + `on_error: fail` are load-bearing on every redactor: +`sequential` (not `transform`) is what lets a redactor **block**, not just +redact (CPEX silently downgrades `continue_processing=False` → `True` in +transform mode), and `on_error: fail` makes a crashing/timing-out redactor halt +the write rather than pass unredacted content through (fail-closed). + +### 2. Code-first (`HooksConfig(plugins=[...])`) — the equivalent + +```python +from altk_evolve.config.evolve import EvolveConfig +from altk_evolve.config.hooks import HookPluginSpec, HooksConfig +from altk_evolve.frontend.client.evolve_client import EvolveClient + +config = EvolveConfig(hooks=HooksConfig(plugins=[ + HookPluginSpec( + name="pii_filter_memory", + kind="altk_evolve.hooks.plugins.pii.PIIFilterMemoryPlugin", + hooks=["memory_pre_write", "llm_pre_call"], + mode="sequential", priority=10, on_error="fail", + config={"detect_email": True, "detect_ssn": True, "detect_phone": True, + "detect_credit_card": True, "detect_ip_address": True, + "redaction_text": "[REDACTED]"}, + ), + HookPluginSpec( + name="readi_semantic_pii", + kind="altk_evolve.hooks.plugins.readi.ReadiSemanticPIIPlugin", + hooks=["memory_pre_write", "llm_pre_call"], + mode="sequential", priority=10, on_error="fail", + config={"readi_extractor": "default", "redaction_text": "[REDACTED]"}, + ), +])) +client = EvolveClient(config) # every update_entities(...) is now redacted before storage +``` + +`HooksConfig(plugins_yaml="path.yaml")` points at a YAML file instead; explicit +`plugins`/`plugins_yaml` override auto-discovery. + +--- + +## Demo 1 — PII never lands in memory (`examples/pii_redaction_demo.py`) + +Writes a fake persona (name + email/phone/SSN/card/IP) with **both** the regex +and semantic PII plugins configured, then reads it back FROM STORAGE. The +headline: the regex method removes every structured identifier, but only the +**semantic** method removes the *name*. + +```bash +uv run --no-sync python examples/pii_redaction_demo.py +``` + +Expected output on a host **without** `[pii-semantic]` (e.g. this darwin/MPS +machine — regex-only path, name deliberately survives to make the point): + +``` +PII redaction on the memory hook seam (memory_pre_write choke point) + regex method ([pii-regex]) : ACTIVE -> structured identifiers + semantic method ([pii-semantic]) : absent -> names will NOT be caught + +What the agent tried to remember -> what actually got stored + + IN : Primary contact is Dana Whitfield, who replies fastest at dana.whitfield@example.com. + OUT: Primary contact is Dana Whitfield, who replies fastest at [REDACTED]. + + IN : Dana Whitfield asked for a callback on 415-555-0199 before noon Friday. + OUT: Dana Whitfield asked for a callback on [REDACTED] before noon Friday. + + IN : For billing we have SSN 123-45-6789 and card 4111 1111 1111 1111 on file. + OUT: For billing we have SSN [REDACTED] and card [REDACTED] on file. + + IN : Last successful login came from IP 192.168.10.42 on the office network. + OUT: Last successful login came from IP [REDACTED] on the office network. + + IN : Remember: the customer prefers metric units and a dark UI theme. + OUT: Remember: the customer prefers metric units and a dark UI theme. + +OK — all 5 structured PII values (email, phone, SSN, card, IP) + were replaced with inert filler before storage [regex method]. +NOTE — the NAME 'Dana Whitfield' is STILL PRESENT — regex has no NER. + Install [pii-semantic] and re-run to catch names too (defence in depth). + + The non-PII memory (units + theme preference) was preserved verbatim. +``` + +With `[pii-semantic]` installed and running (a CPU/CUDA host), the same run +prints `semantic method ([pii-semantic]) : ACTIVE`, the name lines read +`OUT: Primary contact is [REDACTED], who replies fastest at [REDACTED].`, and +the final assertion becomes: + +``` +OK — the free-form NAME 'Dana Whitfield' was also removed [semantic method]; + regex alone has no NER and would have left it in place. +``` + +**Talking points** +- Redaction is at `memory_pre_write`, the backend write choke point — the OUT + lines are the *real persisted content*, read back from the filesystem store. +- Structured identifiers (email, phone, SSN, card, IP) → regex, precision ~1.00, + non-PII sentence survives verbatim (surgical, not lossy). +- The **name** is the semantic method's job; regex has no NER. Running both is + defence in depth. + +--- + +## Demo 2 — Structured secrets never land in memory (`examples/secrets_demo.py`) + +A fake AWS key and GitHub token are scrubbed on write AND on LLM egress; a +non-secret policy string survives. + +```bash +uv run --no-sync python examples/secrets_demo.py +``` + +``` +Structured-secrets redaction on the memory hook seam + + IN (agent tried to remember): + CI notes: the pipeline uses AWS key AKIAIOSFODNN7EXAMPLE and GitHub token ghp_1234567890abcdefghijklmnopqrstuvwxyz; policy is to deploy only on Fridays. # pragma: allowlist secret + + OUT (what actually got stored): + CI notes: the pipeline uses AWS key [REDACTED] and GitHub token [REDACTED]; policy is to deploy only on Fridays. + + LLM egress (dispatch_llm_pre_call): + Use [REDACTED] to list buckets + +OK — the AWS key and GitHub token were redacted on write AND on LLM egress; + the non-secret policy ('deploy only on Fridays') was preserved verbatim. +``` + +**Talking points** +- Orthogonal to the PII methods: catches *credentials/tokens*, not people. +- Same seam, same `memory_pre_write` + `llm_pre_call` wiring — it chains + alongside a PII method. +- Entropy/heuristic detectors (generic API keys, JWT-shaped, hex/base64) are OFF + by default because they over-redact a memory corpus; opt in deliberately. + +--- + +## Demo 3 — All the seam plugins together (`examples/hooks_demo.py`) + +The end-to-end seam demo: a metadata normalizer (stamps `trace_id`/`created_at` +on write), an access stamp (`last_accessed` on read), and the regex PII filter, +all at once. + +```bash +uv run --no-sync python examples/hooks_demo.py +``` + +``` +stored content: Customer Dana Whitfield, email [REDACTED], SSN [REDACTED]. +stored metadata: {'created_at': '2026-07-24T17:34:00.576531+00:00', 'task_id': 'task-0042', 'trace_id': 'task-0042'} +last_accessed: 2026-07-24T17:34:00.577754+00:00 +llm egress: Summarize: call [REDACTED] or mail [REDACTED] +``` + +Note `trace_id` copied from `task_id` (normalizer), `last_accessed` stamped by +the read (access stamp), and the PII redacted on both storage and LLM egress. + +--- + +## Demo 4 — Retention policies & provenance cascade (`examples/retention_demo.py`) + +Seeds a realistic store (stale/fresh/recently-read guidelines, an old session, +and a memory derived from it) and runs a policy from +[`examples/retention.example.yaml`](examples/retention.example.yaml): flag by +disuse, delete by age, and cascade-delete memories derived from a deleted +session. + +```bash +uv run --no-sync python examples/retention_demo.py +# or run a policy against a live namespace: +evolve retention run --policy examples/retention.example.yaml # dry run +evolve retention run --policy examples/retention.example.yaml --apply # mutate +``` + +``` +Seeded memories: + 1 [guideline] STALE: deploy only on Fridays + 2 [guideline] FRESH: prefer uv over pip for installs + 3 [guideline] USED: the flaky test needs a retry (last read ...) + 4 [guideline] DERIVED from the old session: always run ruff + 5 [trajectory] SESSION transcript of an old support chat + +APPLY: + DELETE 5 trajectory reason=age rule=old-sessions + DELETE 4 guideline reason=cascade:T1 rule=old-sessions + FLAG 1 guideline reason=unused rule=unused-guidelines + +Store after apply: + 1 [guideline] STALE: deploy only on Fridays <-- FLAGGED + 2 [guideline] FRESH: prefer uv over pip for installs + 3 [guideline] USED: the flaky test needs a retry +``` + +**Talking points** +- **Flag** is non-destructive (`retention_flagged_at` marker); **delete** is + hard removal. **Dry-run** is the default and mutates nothing. +- **Cascade**: deleting the old *session* also deletes the guideline derived + from it (linked by provenance: `metadata.source_task_id == trace_id`), even + though that guideline is young. +- The "unused" signal is `metadata.last_accessed`, stamped by `AccessStampPlugin` + on `memory_post_read` (or `EvolveClient.record_access`). Absent it, disuse + falls back to `created_at` and the engine warns — the guideline read yesterday + survives *because* it was stamped, the never-read stale one is flagged. + +--- + +## Measured results (retained — these measure detector/model behavior) + +These numbers are independent of how the detectors are wired into the write +path, so they carry forward unchanged from the earlier PoC. Reproduce with +[`examples/pii_benchmark.py`](examples/pii_benchmark.py). + +### English — `ai4privacy/pii-masking-200k`, 43,501 records (both methods) + +| Metric | regex (`[pii-regex]`) | semantic (READI, `en_core_web_trf`) | +|---|---|---| +| Overall recall | 0.12 | **0.47** | +| Precision | 1.00 | 0.99 | +| Leak rate | 0.99 | 0.86 | +| firstname / lastname / middlename | 0.00 / 0.00 / 0.00 | **0.92 / 0.97 / 0.94** | +| city / county / state | 0.00 | 0.88 / 0.90 / 0.86 | +| email / ip | 1.00 / 1.00 | 1.00 / 0.95 | +| phone / ssn / credit_card | 0.46 / 0.22 / 0.04 | 0.60 / 0.43 / 0.03 | + +The regex method is precise on structured identifiers (email/IP = 1.00) but +**0.00 on names** — it catches identifiers, not people. Semantic ~4× the overall +recall, driven entirely by the free-form entities regex has no detector for +(**names 0.00 → ~0.92**), at precision ~1.00 (neither over-redacts). + +### Japanese — `ai4privacy/pii-masking-openpii-1.5m` (`ja`), 2,000 records + +A **language-matched** model is decisive. Same rows, three engines: + +| PII type | regex (`[pii-regex]`) | English NER (`en_core_web_trf`) | Japanese NER (`ja_core_news_trf`) | +|---|---|---|---| +| **Overall recall** | 0.03 | 0.15 | **0.92** | +| **Precision** | 0.99 | 0.99 | 0.93 | +| email | 0.09 | 0.09 | 0.94 | +| phone | 0.13 | 0.47 | 0.99 | +| zipcode | 0.00 | 0.46 | 1.00 | +| credit card | 0.02 | 0.03 | 1.00 | +| social / ID number | 0.46 | 0.46 | 1.00 | +| given / surname | 0.00 / 0.00 | 0.07 / 0.04 | 0.82 / 0.96 | +| city / street | 0.00 / 0.00 | 0.00 / 0.00 | 1.00 / 0.95 | + +```bash +uv run --no-sync python examples/pii_benchmark.py \ + --dataset ai4privacy/pii-masking-openpii-1.5m --language ja --split validation \ + --limit 2000 --mode both --readi-extractor spacy --readi-model ja_core_news_trf --readi-language ja +``` + +- On real Japanese PII, regex is largely ineffective (0.03) — Western-format + patterns don't fit no-space Japanese text (even email → 0.09). +- The English NER barely helps (0.15) — it can't read Japanese names/places. +- A language-matched model catches **~92% at ~93% precision** — a one-line config + change (`readi_extractor="spacy", readi_model="ja_core_news_trf"`). +- Bonus: this benchmark caught a real bug — READI's char offsets vs cpex's byte + offsets; byte offsets broke span scoring on multibyte text (precision looked + like 0.31; fixed to 0.99). Redaction itself was always correct. + +--- + +## Defence in depth — run all three together + +The three methods are complementary, not either/or. The recommended production +posture is to run **regex + semantic + secrets** at once: regex for structured +identifiers, semantic (NER) for names and free-form entities, secrets for +credentials/tokens. They chain on the same `memory_pre_write` + `llm_pre_call` +hooks; each is a high-precision floor, and together they close each other's +gaps. Enable all three by uncommenting the regex and secrets blocks in the +`evolve hooks init` scaffold and leaving READI active — a YAML edit, no code +change. + +--- + +## Limitations + +- **Regex PII is blind to names and non-Western formats.** No NER: names, + addresses, and no-space scripts leak (0.00 recall on names; 0.03 overall on + Japanese). Use it for structured identifiers, and pair it with the semantic + method for coverage. +- **Semantic PII is language-bound and needs a language-matched model.** READI's + default English pipeline barely helps on Japanese (0.15); a matched model + reaches 0.92. It is also heavy (torch + spaCy transformer, ~460MB weights on + first use). +- **macOS / Apple-Silicon: READI fails closed on MPS.** spaCy-curated-transformers + places the model on torch's MPS backend, which binds to the first thread that + touches it; the seam's sync bridge dispatches on a worker thread, so the model + raises *"Placeholder storage has not been allocated on MPS device!"* and, under + `on_error: fail`, blocks the write. **This machine is darwin**, so the semantic + e2e path could not be run here — `examples/pii_redaction_demo.py` catches the + failure and falls back to regex-only with a printed note. **Regex and secrets + are Rust/regex and unaffected — regex is the mac-friendly PII method.** The + caveat does not affect CPU/CUDA hosts. +- **Secrets and regex PII are high-precision floors, not proof of absence.** Both + are pattern-based with no verification; entropy/heuristic secret detectors are + off by default because they over-redact a memory corpus. + +--- + +## Reviewer note + +This branch is **`main` plus demo files only** — a presentation branch, not a +history-preserving merge. The diff vs `main` is exactly: + +- `examples/pii_redaction_demo.py` (new) +- `examples/secrets_demo.py` (new) +- `DEMO.md` (new) + +Everything the demos exercise — the hook seam, the three redaction plugins, the +retention engine, `examples/hooks_demo.py` / `examples/retention_demo.py` / +`examples/pii_benchmark.py` — is already on `main`. Reproduce, don't trust the +prose: run each command above and confirm the output. The semantic e2e path +needs a CPU/CUDA host (see Limitations); everything else runs on this darwin +machine. diff --git a/examples/pii_redaction_demo.py b/examples/pii_redaction_demo.py new file mode 100644 index 00000000..c423281e --- /dev/null +++ b/examples/pii_redaction_demo.py @@ -0,0 +1,199 @@ +#!/usr/bin/env python3 +"""Signature PII demo for the altk_evolve memory hook seam (issue #275, ask 1). + + "PII must never be saved into memory." + +Writes a made-up persona full of fake PII into a throwaway Evolve namespace with +PII redaction configured on the hook seam, then reads it back FROM STORAGE to +prove the PII was scrubbed before it was ever persisted. This is the real +persisted content (a filesystem-backend read), not a cosmetic pass over the +printout — redaction fires at ``memory_pre_write``, the backend write choke +point. + +Two PII methods run TOGETHER (defence in depth), configured as plugins on +``HooksConfig``: + + * REGEX (``PIIFilterMemoryPlugin``, ``[pii-regex]``) — cpex-pii-filter's Rust + engine. High precision on STRUCTURED identifiers (email, phone, SSN, card, + IP). No NER: it cannot catch a name. + * SEMANTIC (``ReadiSemanticPIIPlugin``, ``[pii-semantic]``) — IBM READI NER. + Catches the free-form NAME the regex method structurally cannot. + +HEADLINE: the persona's NAME survives regex alone and is removed only by the +semantic method — that is the case for running both. + +Guards: + * ``[pii-regex]`` absent -> nothing to redact; print a note and exit. + * ``[pii-semantic]`` absent -> run regex-only; print that the name will NOT be + caught without the semantic method. + * macOS / Apple-Silicon: READI's spaCy transformer runs on torch's MPS + backend, which fails closed when the model is first touched off the binding + thread ("Placeholder storage has not been allocated on MPS device!"). With + ``on_error=fail`` that surfaces as a blocked write. We CATCH it and fall + back to regex-only with a printed note, rather than crashing. Regex and + secrets are Rust/regex and unaffected — regex is the mac-friendly method. + +Run: + uv sync --extra hooks --extra pii-regex --extra pii-semantic + uv run --no-sync python examples/pii_redaction_demo.py +""" + +from __future__ import annotations + +import importlib.util +import tempfile + +from altk_evolve.backend.filesystem import FilesystemSettings +from altk_evolve.config.evolve import EvolveConfig +from altk_evolve.config.hooks import HookPluginSpec, HooksConfig +from altk_evolve.frontend.client.evolve_client import EvolveClient +from altk_evolve.hooks.manager import shutdown_hooks +from altk_evolve.schema.core import Entity + +FILLER = "[REDACTED]" + +# All fictional — a made-up persona and obviously-fake identifiers. +PERSONA = "Dana Whitfield" +STRUCTURED_PII = [ + "dana.whitfield@example.com", + "415-555-0199", + "123-45-6789", # SSN-shaped + "4111 1111 1111 1111", # test card number + "192.168.10.42", # private IP +] + +MEMORIES = [ + f"Primary contact is {PERSONA}, who replies fastest at dana.whitfield@example.com.", + f"{PERSONA} asked for a callback on 415-555-0199 before noon Friday.", + "For billing we have SSN 123-45-6789 and card 4111 1111 1111 1111 on file.", + "Last successful login came from IP 192.168.10.42 on the office network.", + "Remember: the customer prefers metric units and a dark UI theme.", # no PII — must survive intact +] + +REGEX_PLUGIN = HookPluginSpec( + name="pii_filter_memory", + kind="altk_evolve.hooks.plugins.pii.PIIFilterMemoryPlugin", + hooks=["memory_pre_write", "llm_pre_call"], + mode="sequential", # sequential (not transform) so it can BLOCK, not just redact + priority=10, + on_error="fail", # fail-closed: never pass unredacted content through + config={ + "detect_email": True, + "detect_ssn": True, + "detect_phone": True, + "detect_credit_card": True, + "detect_ip_address": True, + "default_mask_strategy": "redact", + "redaction_text": FILLER, + }, +) + +SEMANTIC_PLUGIN = HookPluginSpec( + name="readi_semantic_pii", + kind="altk_evolve.hooks.plugins.readi.ReadiSemanticPIIPlugin", + hooks=["memory_pre_write", "llm_pre_call"], + mode="sequential", + priority=10, + on_error="fail", + config={"readi_extractor": "default", "redaction_text": FILLER}, +) + + +def _write_and_read(plugins: list[HookPluginSpec]) -> list[str]: + """Configure the seam with ``plugins``, write the PII memories, read back the + stored content. Returns the stored content strings, sorted to match MEMORIES.""" + config = EvolveConfig( + backend="filesystem", + settings=FilesystemSettings(data_dir=tempfile.mkdtemp(prefix="evolve_pii_demo_")), + hooks=HooksConfig(plugins=plugins), + ) + client = EvolveClient(config) + client.create_namespace("demo") + client.update_entities( + "demo", + [Entity(content=m, type="guideline") for m in MEMORIES], + enable_conflict_resolution=False, + ) + stored = client.search_entities("demo", limit=50) + # search_entities has no guaranteed order; align by matching the surviving + # non-PII tail so IN/OUT pairs line up regardless of storage order. + by_key = {str(e.content): str(e.content) for e in stored} + # Fall back to insertion order when content changed under us; we only need + # the set for assertions, and print in stored order below. + return list(by_key.values()) + + +def main() -> int: + if importlib.util.find_spec("cpex_pii_filter") is None: + print("This demo needs the [pii-regex] extra (cpex-pii-filter). Try:") + print(" uv sync --extra hooks --extra pii-regex --extra pii-semantic") + print(" uv run --no-sync python examples/pii_redaction_demo.py") + return 1 + + has_semantic = importlib.util.find_spec("risk_assessment") is not None + + plugins = [REGEX_PLUGIN] + semantic_active = False + if has_semantic: + plugins = [REGEX_PLUGIN, SEMANTIC_PLUGIN] + + print("PII redaction on the memory hook seam (memory_pre_write choke point)") + print(" regex method ([pii-regex]) : ACTIVE -> structured identifiers") + if has_semantic: + print(" semantic method ([pii-semantic]) : ACTIVE -> free-form names (NER)") + else: + print(" semantic method ([pii-semantic]) : absent -> names will NOT be caught") + print() + + try: + stored = _write_and_read(plugins) + semantic_active = has_semantic + except Exception as exc: # noqa: BLE001 — deliberate: fall back, don't crash + if not has_semantic: + raise + print(" NOTE: the semantic (READI) path failed on this host — falling back to regex-only.") + print(f" ({type(exc).__name__}: {str(exc).splitlines()[0][:120]})") + print(" This is the documented macOS/Apple-Silicon MPS caveat (see DEMO.md);") + print(" regex redaction is unaffected and is the mac-friendly method.") + print() + stored = _write_and_read([REGEX_PLUGIN]) + semantic_active = False + + print("What the agent tried to remember -> what actually got stored\n") + joined = "\n".join(stored) + # Print IN/OUT by matching each stored line back to an input by its redaction- + # invariant tail (everything after the PII), so pairs line up cleanly. + for original in MEMORIES: + # find the stored line whose non-redacted words overlap this input most + match = max(stored, key=lambda s: len(set(s.split()) & set(original.split()))) + print(f" IN : {original}") + print(f" OUT: {match}\n") + + # Assertion 1 (always): every STRUCTURED PII value is gone (regex method). + leaked_structured = [s for s in STRUCTURED_PII if s in joined] + if leaked_structured: + print("FAIL — structured PII leaked into storage:", leaked_structured) + return 1 + print(f"OK — all {len(STRUCTURED_PII)} structured PII values (email, phone, SSN, card, IP)") + print(" were replaced with inert filler before storage [regex method].") + + # Assertion 2 (semantic only): the NAME is gone — the regex method cannot do this. + name_present = PERSONA in joined + if semantic_active: + if name_present: + print(f"FAIL — the name '{PERSONA}' survived despite the semantic method being active.") + return 1 + print(f"OK — the free-form NAME '{PERSONA}' was also removed [semantic method];") + print(" regex alone has no NER and would have left it in place.") + else: + print(f"NOTE — the NAME '{PERSONA}' is {'STILL PRESENT' if name_present else 'absent'} — regex has no NER.") + print(" Install [pii-semantic] and re-run to catch names too (defence in depth).") + + print("\n The non-PII memory (units + theme preference) was preserved verbatim.") + + shutdown_hooks() + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/examples/secrets_demo.py b/examples/secrets_demo.py new file mode 100644 index 00000000..a865bae9 --- /dev/null +++ b/examples/secrets_demo.py @@ -0,0 +1,120 @@ +#!/usr/bin/env python3 +"""Structured-secrets redaction demo for the altk_evolve memory hook seam. + +A THIRD redaction method alongside the two PII ones: it targets +CREDENTIALS/TOKENS (AWS keys, GitHub/Slack tokens, private-key blocks) that +neither PII path aims at. Backed by cpex-secrets-detection's Rust core, wired as +a native plugin on the seam. + +Shows, mirroring examples/hooks_demo.py: + 1. SecretsFilterMemoryPlugin on ``memory_pre_write`` — a fake AWS key and + GitHub token are scrubbed BEFORE the entity is persisted (read back from + the filesystem store to prove it). + 2. The same plugin on ``llm_pre_call`` — ``dispatch_llm_pre_call`` egress is + redacted too, so no completion call site ever sees the raw secret. + 3. A non-secret string ("deploy only on Fridays") survives verbatim. + +Guard: skipped with a note when the ``[secrets]`` extra +(cpex-secrets-detection) is not installed. + +Run: + uv sync --extra hooks --extra secrets + uv run --no-sync python examples/secrets_demo.py +""" + +from __future__ import annotations + +import importlib.util +import tempfile + +from altk_evolve.backend.filesystem import FilesystemSettings +from altk_evolve.config.evolve import EvolveConfig +from altk_evolve.config.hooks import HookPluginSpec, HooksConfig +from altk_evolve.frontend.client.evolve_client import EvolveClient +from altk_evolve.hooks.manager import dispatch_llm_pre_call, shutdown_hooks +from altk_evolve.schema.core import Entity + +FILLER = "[REDACTED]" + +# All fake / non-functional — obviously-invalid demo credentials, never real. +FAKE_AWS_KEY = "AKIAIOSFODNN7EXAMPLE" # pragma: allowlist secret +FAKE_GITHUB_TOKEN = "ghp_1234567890abcdefghijklmnopqrstuvwxyz" # pragma: allowlist secret +NON_SECRET = "deploy only on Fridays" + +MEMORY = ( + f"CI notes: the pipeline uses AWS key {FAKE_AWS_KEY} and GitHub token " # pragma: allowlist secret + f"{FAKE_GITHUB_TOKEN}; policy is to {NON_SECRET}." +) + + +def main() -> int: + if importlib.util.find_spec("cpex_secrets_detection") is None: + print("This demo needs the [secrets] extra (cpex-secrets-detection). Try:") + print(" uv sync --extra hooks --extra secrets") + print(" uv run --no-sync python examples/secrets_demo.py") + return 0 + + plugins = [ + HookPluginSpec( + name="secrets_filter_memory", + kind="altk_evolve.hooks.plugins.secrets.SecretsFilterMemoryPlugin", + hooks=["memory_pre_write", "llm_pre_call"], + mode="sequential", # sequential (not transform) so it can BLOCK, not just redact + priority=10, + on_error="fail", # fail-closed: never pass content with secrets through + config={"redact": True, "redaction_text": FILLER}, + ) + ] + + config = EvolveConfig( + backend="filesystem", + settings=FilesystemSettings(data_dir=tempfile.mkdtemp(prefix="evolve_secrets_demo_")), + hooks=HooksConfig(plugins=plugins), + ) + client = EvolveClient(config) + client.create_namespace("demo") + + print("Structured-secrets redaction on the memory hook seam\n") + print(" IN (agent tried to remember):") + print(f" {MEMORY}\n") + + # 1: write the secret-bearing memory; redaction fires at memory_pre_write. + client.update_entities( + "demo", + [Entity(content=MEMORY, type="guideline")], + enable_conflict_resolution=False, + ) + stored = client.search_entities("demo", limit=10)[0] + print(" OUT (what actually got stored):") + print(f" {stored.content}\n") + + # 2: LLM egress redaction — what any completion call site would see. + messages = dispatch_llm_pre_call( + [{"role": "user", "content": f"Use {FAKE_AWS_KEY} to list buckets"}], # pragma: allowlist secret + purpose="demo", + ) + print(" LLM egress (dispatch_llm_pre_call):") + print(f" {messages[0]['content']}\n") + + # Assertions: secrets gone from BOTH surfaces; the non-secret policy survives. + ok = True + if FAKE_AWS_KEY in stored.content or FAKE_GITHUB_TOKEN in stored.content: + print("FAIL — a secret leaked into storage.") + ok = False + if FAKE_AWS_KEY in messages[0]["content"]: + print("FAIL — a secret leaked to LLM egress.") + ok = False + if NON_SECRET not in stored.content: + print(f"FAIL — the non-secret string '{NON_SECRET}' was over-redacted.") + ok = False + + if ok: + print("OK — the AWS key and GitHub token were redacted on write AND on LLM egress;") + print(f" the non-secret policy ('{NON_SECRET}') was preserved verbatim.") + + shutdown_hooks() + return 0 if ok else 1 + + +if __name__ == "__main__": + raise SystemExit(main())