trust-tools

Live demo (static): barlowa124.github.io/trust-tools. traj-review-ui renders the repo's committed audit artifacts.

Tools for verifying what AI systems did — and checking whether the watchers can be fooled. Each subdirectory is a self-contained package with its own tests and commit history where applicable.

60-second demo

cd agent_trajectory_audit && pip install -e .
trajaudit redteam    # adversarial battery against the detectors; evasions printed

The bundled review UI is deployed: barlowa124.github.io/trust-tools.

Where this sits in the portfolio

trust-tools is the agent security and evals repo: a policy gate on tool calls and emitted answers inside the agent loop (agentmon), span/trace ingestion and audit (agent_observe, trajaudit), deterministic eval batteries and an Inspect bridge (evalh), hash-chained evidence records (inference_receipts, rcptreport), and serving/benchmarking (model_serve). Sibling repos: trust-tools (agent security and evals), bio-qc (lab-data QC pipelines), lab-informatics (lab data plumbing and integrity), llm-posttraining (training-stage behavior work), protein-ml (protein fitness ML), and mol-ml (small-molecule ML).

Packages

Directory	What it does
agent_trajectory_audit/	Parses coding-agent session transcripts into an event schema and flags intent/behavior divergence (unbacked test/deploy claims, abandoned plan items, out-of-scope writes, tool storms). Deterministic heuristics, zero dependencies. trajaudit redteam runs an adversarial battery against the detectors — attacks it evades are printed, not hidden.
agent_monitor/	The audit moved upstream: a policy gate on both sides of the agent loop — tool calls checked before they run (rules), emitted answers checked before they ship (output_rules: credential-shaped strings, JSON format/field contracts, length caps). Each decision (allow/flag/deny) is a hash-chained event record with severity and matched policy — for inline oversight, not post-hoc review. agentmon redteam reports the residual evasions instead of hiding them.
eval_harness/	Task-spec evaluation runner. Deterministic graders, injectable generate_fn, per-task records hash-chained into a verifiable log. sweep runs one battery across staged checkpoints — the training-run-assessment shape. Ships probes/abstention_battery.json plus a real 3-stage SmolLM2 sweep under results/.
inference_receipts/	Hash-bound inference receipts: input, weights, settings, output all sha256-chained per log entry, with replay verification. Any tampering breaks the receipt hash or the chain.
model_serve/	Serving slice: request queue, micro-batching, shadow evaluation of a candidate against production, and a hash-bound record per response. FastAPI wiring optional.
agent_observe/	Span/trace ingest (flat schema + OTLP-lite), policy enrichment through agentmon, trajaudit audit over the same stream, and markdown/HTML trace reports. The observability layer sitting on top of gate + audit.
agent_sandbox/	Containment-layer benchmark: scripted tool-call scenarios run through the agentmon gate and a realpath filesystem jail, measuring each layer separately. The battery documents the one residual (arg aliasing + no kernel isolation without bwrap) instead of hiding it.
receipt_report/	Renders audit documents from the other packages' chains — verifies integrity first, then recomputes every reported number from the records. Broken chains land in the document as findings.
traj_review_ui/	Static React/TS review surface over the audit artifacts: findings panel with severity + evidence, event timeline with call_id-paired tool calls, kind filters and search. Bundles four real example datasets; no backend.
oncology_coscientist/	Verified-numbers LLM reporting on a domain problem. A deterministic TCGA survival pipeline computes every Cox PH and RSF number while a local or remote OpenAI-compatible LLM only writes the prose. A claim verifier binds each number and abstention claim to the recorded computation before a human gate signs off, and every LLM call replays byte-for-byte. The 68-run ledger records the initial Gemma negative result (8 of 9 rejected, the ninth human-rejected) alongside the iterative context fixes and the final 2x3 sweep (Qwen2.5-3B and 14B across LUAD, BRCA, GBM) where all six cells pass and replay PASS under current code.

Running tests

The core packages are stdlib-only:

cd agent_trajectory_audit && PYTHONPATH=src python3 -m pytest tests/ -q
cd ../agent_monitor && PYTHONPATH=src python3 -m pytest tests/ -q
cd ../eval_harness && PYTHONPATH=src python3 -m pytest tests/ -q
cd ../inference_receipts && PYTHONPATH=src python3 -m pytest tests/ -q
cd ../model_serve && PYTHONPATH=src python3 -m pytest tests/ -q
cd ../receipt_report && PYTHONPATH=src python3 -m pytest tests/ -q
cd .. && python3 -m pytest tests/ -q   # monorepo interop

oncology_coscientist has real dependencies (lifelines, scikit-survival, langgraph); its test suite is offline:

cd oncology_coscientist && pip install -e '.[dev]' && python -m pytest tests/ -q

Why one repo

The same question at four points in a system's life: receipts prove what a single computation produced; the eval harness proves what a checkpoint did on a fixed battery; the monitor gates each action before it lands; the audit checks whether a session's recorded behavior matched stated intent — and the redteam battery measures whether those checks can be evaded.

Every package vendors the same canonical-JSON/SHA-256 chain envelope (no runtime dependencies between them). tests/test_interop.py proves the envelopes agree: the same probe object hashes identically under all five implementations, so a record written by one package verifies under any other's primitives.