Skip to content

Commit 5009acf

Browse files
committed
Merge remote-tracking branch 'origin/main' into fix/sequential-envelope-labels
2 parents 9069631 + 620091f commit 5009acf

127 files changed

Lines changed: 8235 additions & 729 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Makefile‎

Lines changed: 12 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
.PHONY: all pre-commit ty unit-test unit-test-junit unit-test-cov-html unit-test-cov-xml diff-cover unit-test-diff-cover
1+
.PHONY: all pre-commit ty docs-api docs-build docs-build-pdf docs-build-all unit-test unit-test-junit unit-test-cov-html unit-test-cov-xml diff-cover unit-test-diff-cover
22

33
CMD:=uv run -m
44
PYMODULE:=pyrit
@@ -19,29 +19,19 @@ pre-commit:
1919
ty:
2020
$(CMD) ty check $(PYMODULE) $(UNIT_TESTS)
2121

22-
# Build the full documentation site:
23-
# 1. Generate API reference JSON from Python source (griffe)
24-
# 2. Convert API JSON to MyST markdown pages
25-
# 3. Build the Jupyter Book site (HTML only — fast, no LaTeX needed)
26-
# 4. Generate RSS feed
27-
docs-build:
28-
uv run python -m build_scripts.pydoc2json pyrit --submodules -o doc/_api/pyrit_all.json
29-
uv run python -m build_scripts.gen_api_md
30-
# --strict validates URLs and cross-refs; skips are configured in doc/myst.yml under error_rules
31-
cd doc && uv run jupyter-book build --all --html --strict
22+
# Build strict HTML and the RSS feed after generating the API reference.
23+
# --all would also select the configured PDF export and require LaTeX.
24+
docs-build: docs-api
25+
cd doc && uv run jupyter-book build --html --strict
3226
uv run python -m build_scripts.generate_rss
3327

34-
# Build the full documentation site including the PDF export.
35-
# Mirrors the ReadTheDocs build (.readthedocs.yaml) so CI catches PDF-only issues
36-
# such as missing images that the HTML-only build silently ignores.
37-
# Requires xelatex / latexmk on PATH (texlive-xetex + texlive-fonts-recommended +
38-
# texlive-plain-generic + latexmk on Ubuntu).
39-
docs-build-all:
40-
uv run python -m build_scripts.pydoc2json pyrit --submodules -o doc/_api/pyrit_all.json
41-
uv run python -m build_scripts.gen_api_md
42-
# --strict validates URLs and cross-refs; skips are configured in doc/myst.yml under error_rules
43-
cd doc && uv run jupyter-book build --all --html --pdf --strict
44-
uv run python -m build_scripts.generate_rss
28+
# PDF is a separate, checked export requiring latexmk and xelatex.
29+
docs-build-pdf: docs-api
30+
uv run python -m build_scripts.build_docs_pdf
31+
32+
# Build HTML first, then check the PDF export without repeating API generation.
33+
docs-build-all: docs-build
34+
uv run python -m build_scripts.build_docs_pdf
4535

4636
# Regenerate only the API reference pages (without building the full site)
4737
docs-api:

‎THIRD_PARTY_NOTICES.txt‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,13 @@ payloads, and report domain payloads. Microsoft Corporation changed the template
2222
markers, separated payload templates from trigger values, and adapted context
2323
generation and sampling for PyRIT.
2424

25+
ProPILE material includes the probe code, PII records, and prompt templates.
26+
Microsoft Corporation converted the JSONL records and TSV templates into seed
27+
datasets, changed the template placeholders to Jinja, and moved prompt
28+
construction and scoring into PyRIT components. Garak extracted the records from
29+
nvidia/Nemotron-CC-v2.1 (https://huggingface.co/datasets/nvidia/Nemotron-CC-v2.1);
30+
each record keeps its source_dataset and source_id.
31+
2532
---------------------------------------------------------
2633

2734
PromptInject source portions - MIT

‎build_scripts/build_docs_pdf.py‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# Copyright (c) Microsoft Corporation.
2+
# Licensed under the MIT license.
3+
4+
"""Build the configured documentation PDF and verify the export, not just the exit code."""
5+
6+
from __future__ import annotations
7+
8+
import re
9+
import shutil
10+
import subprocess
11+
import sys
12+
from pathlib import Path
13+
14+
import yaml
15+
from pypdf import PdfReader
16+
from pypdf.errors import PdfReadError
17+
18+
19+
def _pdf_output(doc_root: Path) -> Path:
20+
config = yaml.safe_load((doc_root / "myst.yml").read_text(encoding="utf-8"))
21+
project = config.get("project") if isinstance(config, dict) else None
22+
exports = project.get("exports") if isinstance(project, dict) else None
23+
if not isinstance(exports, list):
24+
raise ValueError("myst.yml must declare a project PDF export.")
25+
pdf_exports = [export for export in exports if isinstance(export, dict) and export.get("format") == "pdf"]
26+
if len(pdf_exports) != 1:
27+
raise ValueError("Expected exactly one project PDF export in myst.yml.")
28+
export = pdf_exports[0]
29+
if export.get("template") != "plain_latex_book":
30+
raise ValueError("The checked PDF build supports plain_latex_book with xelatex.")
31+
output = export.get("output")
32+
if not isinstance(output, str) or not output or Path(output).suffix != ".pdf":
33+
raise ValueError("The PDF export must declare an output ending in .pdf.")
34+
output_path = (doc_root / output).resolve()
35+
if not output_path.is_relative_to(doc_root.resolve()):
36+
raise ValueError("The PDF output must stay inside the documentation directory.")
37+
return output_path
38+
39+
40+
def _signature(path: Path) -> tuple[int, int, int] | None:
41+
if not path.exists():
42+
return None
43+
stat = path.stat()
44+
return stat.st_mtime_ns, stat.st_ctime_ns, stat.st_size
45+
46+
47+
def _require_fresh_file(*, path: Path, previous: tuple[int, int, int] | None) -> None:
48+
if not path.is_file() or path.stat().st_size == 0:
49+
raise ValueError(f"PDF export did not produce a nonempty file: {path}")
50+
if _signature(path) == previous:
51+
raise ValueError(f"PDF export left a stale file unchanged: {path}")
52+
53+
54+
def _check_engine_log(path: Path) -> None:
55+
fatal = re.compile(
56+
r"^!|(?:LaTeX|Package \S+|Class \S+) Error:|Emergency stop|Fatal error occurred"
57+
r"|^Latexmk: (?:Errors|Failure)|^Collected error summary"
58+
)
59+
for line_number, line in enumerate(path.read_text(encoding="utf-8", errors="replace").splitlines(), start=1):
60+
if fatal.search(line):
61+
raise ValueError(f"Fatal LaTeX diagnostic in {path}:{line_number}: {line}")
62+
63+
64+
def build_pdf(doc_root: Path) -> int:
65+
"""Build the repository's XeLaTeX PDF export and reject missing, stale, or failed output."""
66+
output = _pdf_output(doc_root)
67+
missing = [tool for tool in ("latexmk", "xelatex") if shutil.which(tool) is None]
68+
if missing:
69+
raise ValueError(
70+
f"Missing PDF prerequisites on PATH: {', '.join(missing)}. "
71+
"Provision LaTeX separately; use the HTML-only build if a PDF is not needed."
72+
)
73+
logs_dir = output.parent / f"{output.stem}_pdf_logs"
74+
logs = [logs_dir / f"{output.stem}.log", logs_dir / f"{output.stem}.shell.log"]
75+
previous = {path: _signature(path) for path in [output, *logs]}
76+
result = subprocess.run(
77+
[sys.executable, "-m", "jupyter_book", "build", "--site", "--pdf", "--strict", "--logs"],
78+
cwd=doc_root,
79+
check=False,
80+
)
81+
if result.returncode != 0:
82+
print(f"ERROR: Jupyter Book PDF build exited with code {result.returncode}.", file=sys.stderr)
83+
return result.returncode
84+
for path in [output, *logs]:
85+
_require_fresh_file(path=path, previous=previous[path])
86+
for log in logs:
87+
_check_engine_log(log)
88+
reader = PdfReader(output, strict=True)
89+
if not reader.pages:
90+
raise ValueError(f"PDF export contains no pages: {output}")
91+
print(f"[OK] Verified fresh PDF export and native LaTeX logs: {output}")
92+
return 0
93+
94+
95+
def main() -> int:
96+
"""Run the checked export from this checkout's documentation directory."""
97+
doc_root = Path(__file__).resolve().parents[1] / "doc"
98+
try:
99+
return build_pdf(doc_root)
100+
except (OSError, ValueError, yaml.YAMLError, PdfReadError) as error:
101+
print(f"ERROR: {error}", file=sys.stderr)
102+
return 1
103+
104+
105+
if __name__ == "__main__":
106+
sys.exit(main())

‎doc/bibliography.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,5 +5,5 @@ All academic papers, research blogs, and technical reports referenced throughout
55
:::{dropdown} Citation Keys
66
:class: hidden-citations
77

8-
[@aakanksha2024multilingual; @abughallous2026semguard; @adversaai2023universal; @ahn2025puzzled; @andriushchenko2024tense; @anthropic2024manyshot; @aqrawi2024singleturncrescendo; @atr2026; @banerjee2025safeinfer; @bethany2024mathprompt; @bhardwaj2023harmfulqa; @bhardwaj2024homer; @boucher2023trojan; @brahman2024coconot; @bryan2025agentictaxonomy; @bullwinkel2025airtlessons; @bullwinkel2025repeng; @bullwinkel2026trigger; @chao2023pair; @chao2024jailbreakbench; @choi2026xlsafetybench; @cui2024orbench; @darkbench2025; @derczynski2024garak; @ding2023wolf; @dong2025sata; @embracethered2024unicode; @embracethered2025sneakybits; @evtimov2025wasp; @gehman2020realtoxicityprompts; @ghosh2025aegis; @ghosh2025ailuminate; @gong2025figstep; @gupta2024walledeval; @haider2024phi3safety; @han2024medsafetybench; @han2024wildguard; @hiddenlayer2025policypuppetry; @hines2024spotlighting; @huang2024bijectionlearning; @hughes2024bestofn; @inie2025summon; @ji2023beavertails; @ji2024pkusaferlhf; @jiang2025sosbench; @jones2025computeruse; @kingma2014adam; @knight2025fortress; @li2024drattack; @li2024mossbench; @li2024saladbench; @li2024wmdp; @lin2023toxicchat; @liu2024flipattack; @liu2024mmsafetybench; @lopez2024pyrit; @luo2024jailbreakv; @lutz2026pyrit; @lv2024codechameleon; @mazeika2023tdc; @mazeika2024harmbench; @mckee2024transparency; @mehrotra2023tap; @microsoft2024skeletonkey; @odin2024; @palaskar2025vlsu; @pavlova2024goat; @pfohl2024equitymedqa; @promptfoo2025ccp; @ren2024codeattack; @robustintelligence2024bypass; @roccia2024promptintel; @rottger2023xstest; @rottger2025msts; @russinovich2024crescendo; @russinovich2025cca; @russinovich2025price; @scheuerman2025transphobia; @shaikh2022second; @shayegani2025computeruse; @shen2023donotanything; @sheshadri2024lat; @souly2024strongreject; @stok2023ansi; @tan2026comicjailbreak; @tang2025multilingual; @tedeschi2024alert; @vantaylor2024socialbias; @vidgen2023simplesafetytests; @wang2023decodingtrust; @wang2023donotanswer; @wang2025siuo; @wang2026visualleakbench; @wei2023jailbroken; @xie2024sorrybench; @yu2023gptfuzzer; @yuan2023cipherchat; @zeng2024persuasion; @zeng2024shieldgemma; @zhang2024cbtbench; @ziems2022mic; @zong2024vlguard; @zou2023gcg]
8+
[@aakanksha2024multilingual; @abughallous2026semguard; @adversaai2023universal; @ahn2025puzzled; @andriushchenko2024tense; @anthropic2024manyshot; @aqrawi2024singleturncrescendo; @atr2026; @banerjee2025safeinfer; @bethany2024mathprompt; @bhardwaj2023harmfulqa; @bhardwaj2024homer; @boucher2023trojan; @brahman2024coconot; @bryan2025agentictaxonomy; @bullwinkel2025airtlessons; @bullwinkel2025repeng; @bullwinkel2026trigger; @chao2023pair; @chao2024jailbreakbench; @choi2026xlsafetybench; @cui2024orbench; @darkbench2025; @derczynski2024garak; @ding2023wolf; @dong2025sata; @embracethered2024unicode; @embracethered2025sneakybits; @evtimov2025wasp; @gehman2020realtoxicityprompts; @ghosh2025aegis; @ghosh2025ailuminate; @gong2025figstep; @gupta2024walledeval; @haider2024phi3safety; @han2024medsafetybench; @han2024wildguard; @hiddenlayer2025policypuppetry; @hines2024spotlighting; @huang2024bijectionlearning; @hughes2024bestofn; @inie2025summon; @ji2023beavertails; @ji2024pkusaferlhf; @jiang2025sosbench; @jones2025computeruse; @kim2023propile; @kingma2014adam; @knight2025fortress; @li2024drattack; @li2024mossbench; @li2024saladbench; @li2024wmdp; @lin2023toxicchat; @liu2024flipattack; @liu2024mmsafetybench; @lopez2024pyrit; @luo2024jailbreakv; @lutz2026pyrit; @lv2024codechameleon; @mazeika2023tdc; @mazeika2024harmbench; @mckee2024transparency; @mehrotra2023tap; @microsoft2024skeletonkey; @odin2024; @palaskar2025vlsu; @pavlova2024goat; @pfohl2024equitymedqa; @promptfoo2025ccp; @ren2024codeattack; @robustintelligence2024bypass; @roccia2024promptintel; @rottger2023xstest; @rottger2025msts; @russinovich2024crescendo; @russinovich2025cca; @russinovich2025price; @scheuerman2025transphobia; @shaikh2022second; @shayegani2025computeruse; @shen2023donotanything; @sheshadri2024lat; @souly2024strongreject; @stok2023ansi; @tan2026comicjailbreak; @tang2025multilingual; @tedeschi2024alert; @vantaylor2024socialbias; @vidgen2023simplesafetytests; @wang2023decodingtrust; @wang2023donotanswer; @wang2025siuo; @wang2026visualleakbench; @wei2023jailbroken; @xie2024sorrybench; @yu2023gptfuzzer; @yuan2023cipherchat; @zeng2024persuasion; @zeng2024shieldgemma; @zhang2024cbtbench; @ziems2022mic; @zong2024vlguard; @zou2023gcg]
99
:::

‎doc/code/datasets/1_loading_datasets.ipynb‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@
7272
"(`prompt_inject_contexts`, `prompt_inject_techniques`), API-key probe corpora (`garak_api_key_services`,\n",
7373
"`garak_api_key_templates`, `garak_api_key_partial_keys`, `garak_api_key_safe_placeholders`),\n",
7474
"the API-key service-to-pattern map (`garak_api_key_service_patterns`),\n",
75+
"ProPILE PII records and prompt templates (`garak_propile_pii`, `garak_propile_templates`),\n",
7576
"an audio jailbreak set\n",
7677
"(`garak_audio_achilles_heel`), and visual jailbreak sets (`figstep`, `figstep_pro`)."
7778
]
@@ -152,6 +153,8 @@
152153
" 'garak_package_hallucination_stubs',\n",
153154
" 'garak_package_hallucination_unreal_tasks',\n",
154155
" 'garak_perl_packages',\n",
156+
" 'garak_propile_pii',\n",
157+
" 'garak_propile_templates',\n",
155158
" 'garak_pypi_packages',\n",
156159
" 'garak_raku_packages',\n",
157160
" 'garak_rubygems_packages',\n",

‎doc/code/datasets/1_loading_datasets.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,7 @@
7676
# (`prompt_inject_contexts`, `prompt_inject_techniques`), API-key probe corpora (`garak_api_key_services`,
7777
# `garak_api_key_templates`, `garak_api_key_partial_keys`, `garak_api_key_safe_placeholders`),
7878
# the API-key service-to-pattern map (`garak_api_key_service_patterns`),
79+
# ProPILE PII records and prompt templates (`garak_propile_pii`, `garak_propile_templates`),
7980
# an audio jailbreak set
8081
# (`garak_audio_achilles_heel`), and visual jailbreak sets (`figstep`, `figstep_pro`).
8182

‎doc/code/executor/8_modality_feedback.ipynb‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -172,8 +172,8 @@
172172
"- `roakey.png` is loaded from the docs root.\n",
173173
"- A modern color photo of a three-masted ship is loaded from a checked-in asset.\n",
174174
"- Ship photo source: [Gorch Fock unter Segeln Kieler Foerde 2006](\n",
175-
" https://en.wikipedia.org/wiki/German_training_ship_Gorch_Fock_%281958%29#/media/File:Gorch_Fock_unter_Segeln_Kieler_Foerde_2006.jpg\n",
176-
" ) (Wikimedia Commons), licensed under CC BY-SA 2.5."
175+
" https://commons.wikimedia.org/wiki/File:Gorch_Fock_unter_Segeln_Kieler_Foerde_2006.jpg\n",
176+
" ) (Wikimedia Commons). Photo by Felix Koenig, cropped by Ibn Battuta, licensed under CC BY-SA 2.5."
177177
]
178178
},
179179
{

‎doc/code/executor/8_modality_feedback.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -113,8 +113,8 @@
113113
# - `roakey.png` is loaded from the docs root.
114114
# - A modern color photo of a three-masted ship is loaded from a checked-in asset.
115115
# - Ship photo source: [Gorch Fock unter Segeln Kieler Foerde 2006](
116-
# https://en.wikipedia.org/wiki/German_training_ship_Gorch_Fock_%281958%29#/media/File:Gorch_Fock_unter_Segeln_Kieler_Foerde_2006.jpg
117-
# ) (Wikimedia Commons), licensed under CC BY-SA 2.5.
116+
# https://commons.wikimedia.org/wiki/File:Gorch_Fock_unter_Segeln_Kieler_Foerde_2006.jpg
117+
# ) (Wikimedia Commons). Photo by Felix Koenig, cropped by Ibn Battuta, licensed under CC BY-SA 2.5.
118118

119119
# %%
120120
roakey_seed_path = (Path(".") / ".." / ".." / "roakey.png").resolve()

‎doc/code/framework.md‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,6 @@
33
Learn how to use PyRIT's components to build red teaming workflows.
44

55
:::::{grid} 1 1 2 3
6-
:gutter: 3
76

87
::::{card} 📦 Datasets
98
:link: ./datasets/0_dataset
@@ -248,7 +247,7 @@ If you are contributing to PyRIT, that work will most likely land in one of the
248247
- A target may observe an internal, caller-owned send context at the provider-invocation boundary,
249248
after target-side waits and immediately before irreversible provider I/O, but the caller owns any
250249
bootstrap-history identity, replay, or branching state.
251-
- HTTP targets and `LiteLLMChatTarget` can propagate a separate trace context for each send. `TargetTraceConfig` controls this
250+
- HTTP targets, `LiteLLMChatTarget`, and `WebSocketCopilotTarget` can propagate a separate trace context for each send. `TargetTraceConfig` controls this
252251
behavior, is off by default, and can use a caller-owned tracer. Enable it only for an endpoint that
253252
is known to accept W3C trace context. Targets record request trace metadata; the prompt normalizer
254253
persists it. Outbound request metadata is separate from chat role, so missing trace

‎doc/code/registry/0_registry.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -135,8 +135,8 @@ attack = AttackRegistry.get_registry_singleton().create_instance(
135135
Simple scalar inputs use the shared resolver. Pass live Python configuration
136136
objects, such as `AttackAdversarialConfig`, `AttackConverterConfig`, and
137137
`AttackScoringConfig`, for nested components. Advanced Python values, such as a
138-
prompt normalizer or a parameter class, pass through unchanged. Nested JSON
139-
attack recipes are not supported.
138+
prompt normalizer or a parameter class, pass through unchanged. The shared
139+
resolver also constructs explicitly declared structured configuration variants.
140140

141141
An attack class implements the conversation algorithm. An attack technique
142142
factory selects and configures that class, converters, scorers, and seeds.

0 commit comments

Comments
 (0)