MAWILE (Multi-Axis Workbench for Inspecting LLM Evaluators) is a black-box reliability audit for configured LLM judges. It treats the judge prompt, rubric, model, decoding settings, and output schema as one measurement instrument, then estimates where the resulting verdicts are noisy, fragile, or worth human review.
python -m pip install -e '.[dev]'
mawile run data/demo/configs/mock_judge.yaml
mawile uiGenerated artifacts are written under runs/<timestamp>/.
The new local HTML UI starts with mawile ui and serves
http://127.0.0.1:8000 by default.
Optional agent overrides use one rule in the UI: blank means inherit from the
perturbation generator; {} means explicitly empty decoding parameters.
The command line supports the same audit spine as the UI:
mawile configs
mawile catalog --validation
mawile inspect data/demo/configs/mock_judge.yaml --previews
mawile run data/demo/configs/mock_judge.yaml --progress
mawile runs
mawile reportLLM-backed planning helpers are available when a non-mock suggester model and
the API key required by its selected provider are configured:
mawile plan path/to/config.yaml --output planned.yaml
mawile suggest-custom path/to/config.yaml --target item.output --format yamlMost inspection commands also support --format json for scripts.
Create or update the conda environment from the repo root:
conda env create -f environment.yml
conda activate mawile
python -m pip install -e .Python dependencies are declared in pyproject.toml; requirements.txt installs the development extra from that single source. Copy .env.example to
.env and fill in the key for each provider you use (OPENAI_API_KEY,
FIREWORKS_API_KEY, or GEMINI_API_KEY). MAWILE loads .env automatically
from the directory where you start the command; if you prefer to source it in
your shell, use export OPENAI_API_KEY=... or
set -a; source .env; set +a so child processes inherit the value.
cp .env.example .env
# Fill the provider keys used by your config in .env
mawile run data/demo/configs/openai_judge.example.yamldata/ Dataset bundles with raw files, items, samples, and configs
runs/ Timestamped audit run directories
scripts/ Data conversion and maintenance scripts
src/mawile/ Reusable audit package
Dataset bundles keep the inputs for an audit together:
data/<dataset>/
raw_data/ Original source files, when available
items.jsonl Canonical audit items
samples/ Optional smaller slices
configs/ Dataset-specific judge configs
README.md Optional dataset notes
Each audit run is self-contained:
runs/<timestamp>/
run.json
report.md
item_risks.csv
config.resolved.yaml
suite.json
perturbations.jsonl
judge_results.jsonl
# run.json also includes attempted_perturbations, operator_manifest,
# and perturbation_ledger for rejected/skipped perturbation transparency.
traces/
llm_calls.jsonl
generation_calls.jsonl
validation_calls.jsonl
judge_calls.jsonl
logs/
events.jsonl
The src/mawile package is split by responsibility:
config.pyloads YAML audit configsschemas.pydefines canonical items, results, perturbations, and risksjudges/contains black-box judge runnersperturbations/declares every perturbation operator inregistry.py; deterministic operators build offline, while LLM-backed operators run through the perturbation agent underperturbations/generated/validation/runs independent, dimension-aware admissibility checks on semantic perturbations (including rule-generated proposals) and gates them viavalidity_statusmetrics/computes noise, flip-risk, signed-shift attribution, judge correctness against gold labels, directional degradation detection, bootstrap confidence intervals, review ranking, and costreporting/writes self-contained run directoriessuites.pyvalidates and loads exact saved probe suites for judge-only replaypipeline/orchestrates an end-to-end audit runweb/serves the local FastAPI/Jinja HTML UI
Choose explicit operator names in audit.perturbation_operators and
audit.directional_perturbation_operators. An empty list means no operators in
that category. With both empty and no enabled custom probes, the run measures
only the repeated baseline. Unknown or unsupported selections are errors.
mawile plan config.yaml --output planned.yaml produces an editable selection.
The suggested built-in lists replace the previous lists; manual edits made
afterward are authoritative. audit.plan_operators: true performs planning when
execution starts. Planning never occurs merely by constructing an audit object.
Planner examples use a recorded seed; saving a plan freezes the operator choices,
but only saved variants preserve the actual LLM rewrites.
There are two current expected relations:
- Invariant (
same_verdict): preserve the semantic judgment. Numeric judges use the configured score tolerance; binary judges compare classes; pairwise judges compare stable candidate identities. - Directional (
worse_verdict): introduce a validated degradation that the judge should detect. Directional metrics are separate from invariant flip risk.
Pairwise audits support invariant probes only. Directional built-in and custom probes are unavailable. Position swap is an ordinary invariant: a candidate chosen at A before the swap should still be chosen when displayed at B. Changing the display label does not itself constitute a verdict flip. Historical reports may retain their old equivariant measurements; these are not comparable by simple relabeling to the new repeated-verdict invariant estimator.
Directional probes are opt-in for pointwise judges and can be selected manually or by the Perturbation Suggester:
agent_output_partial_completion(Rule proposal, semantic validation): remove a final output chunk, then check that a judged requirement or quality property was degraded.agent_output_format_violation(LLM,format_degradation): break exactly one explicit output-format requirement that the current output satisfies, without changing substantive content.agent_input_unmet_requirement_injection(LLM,unmet_requirement): add one plausible requirement to the input that the unchanged output does not satisfy.agent_output_unsupported_claim_insertion(LLM,directional_degradation): add one plausible unsupported claim.agent_output_factual_inconsistency_insertion(LLM,directional_degradation): introduce one small factual or reasoning inconsistency.agent_output_requirement_omission(LLM,directional_degradation): remove or weaken one requirement-satisfying part of the output.agent_output_over_refusal(LLM,refusal_degradation): turn an answerable response into an unnecessary refusal; inapplicable items are recorded as generation skips.
The registered invariant operators, grouped by component. Type is Rule (offline
string transform with structural or semantic validation) or LLM (perturbation-agent rewrite). Check is
the admissibility check applied before a variant counts (— for
deterministic ones). registry.py is the
single source of truth.
| Operator | Type | Check | What it does |
|---|---|---|---|
judge_prompt_instruction_order |
Rule | — | Reorder the assembled prompt/rubric/output blocks without rewriting any of them. |
judge_prompt_reasoning_style |
Rule | judge_prompt_preserve |
Append a reasoning-style directive: direct, step-by-step, or criterion-by-criterion. |
judge_prompt_role_framing |
LLM | judge_prompt_preserve |
Change the evaluator persona, leaving the task fixed. |
judge_prompt_paraphrase |
LLM | judge_prompt_preserve |
Reword the whole prompt, instructions preserved. |
| Operator | Type | Check | What it does |
|---|---|---|---|
judge_rubric_criterion_reorder |
Rule | judge_rubric_preserve |
Reorder intact explicitly delimited rubric blocks; ambiguous prose is skipped. Flat numeric score rows may use structural validation. |
judge_rubric_paraphrase |
LLM | judge_rubric_preserve |
Reword each criterion; meaning, ordering, and thresholds preserved. |
| Operator | Type | Touches | Check | What it does |
|---|---|---|---|---|
agent_input_minor_typo_noise |
LLM | surface | typo_preserve |
Inject a few natural prose typos while preserving exact-syntax content. |
agent_input_paraphrase |
LLM | lexical | meaning_preserve |
Reword the input; meaning and requirements preserved. |
agent_input_distractor_insertion |
LLM | lexical | sample_plausibility |
Add plausible but irrelevant sentences. |
agent_input_context_compression |
LLM | facts | facts_preserve |
Drop only redundant context, keeping every required fact. |
| Operator | Type | Touches | Check | What it does |
|---|---|---|---|---|
agent_output_format_conversion |
LLM | format | same_answer |
Convert between prose, bullets, numbered list, or table. |
agent_output_politeness_tone_shift |
LLM | tone | same_answer |
Change politeness or tone; substance unchanged. |
agent_output_uncertainty_calibration |
LLM | confidence | same_answer |
Make the same answer sound more or less certain. |
agent_output_verbosity_shift |
LLM | length | same_answer |
Make the output shorter or longer; same answer. |
| Operator | Type | Touches | Check | What it does |
|---|---|---|---|---|
agent_paired_position_swap |
Rule | candidate_position | — | Swap A/B display positions while preserving candidate IDs and content; compare the same candidate identity as an invariant verdict. |
agent_paired_entity_rename |
LLM | identity | pair_coherence |
Rename named entities consistently across every section — the bias keystone. |
A run config is a YAML file with nine top-level blocks (providers, judge,
data, audit, output, perturbation_agent, validator_agent,
planner_agent, summary_agent). Only judge and data are required; the rest
fall back to the defaults below.
The canonical definitions live in schemas.py.
Every LLM role uses the OpenAI SDK's Chat Completions interface. openai,
fireworks, and gemini are built in with their standard base URLs and the
environment variables shown above. The map is optional and can add a provider
or replace a built-in profile without changing MAWILE code:
providers:
local_gateway:
api_key_env: LOCAL_GATEWAY_API_KEY
base_url: http://localhost:8000/v1
validator_agent:
provider: local_gateway
model: independent-validator-modelOnly the environment-variable name is serialized in run artifacts; API-key
values are read at runtime. See
cross_provider_validation.example.yaml
for an OpenAI judge, Fireworks perturbation generator, and Gemini validator.
| Key | Type | Default | Meaning |
|---|---|---|---|
provider |
str | openai |
Built-in or config-defined provider profile used for the judge. |
model |
str | required | Model identifier passed to the runner. |
prompt_template |
str | required | Prompt template the judge is given. |
rubric |
str | required | Rubric text injected into the prompt. |
decoding_params |
map | {} |
Runner decoding options (e.g. temperature, top_p). |
output_schema |
map / str / null | null |
Expected structured-output schema; surfaced in the prompt when set. |
threshold |
float / null | null |
Optional pass/fail cutoff for scalar and ordinal verdicts; the passing side follows score_direction. |
score_min, score_max |
float / null | null |
Inclusive valid score range; set both or neither. |
score_direction |
enum | higher_is_better |
Whether larger or smaller scores represent better quality. |
invariant_tolerance |
float ≥ 0 | 0 |
Maximum scalar/ordinal change still counted as invariant. |
gold_tolerance |
float ≥ 0 / null | null |
Optional scalar gold-error tolerance used to define tolerance accuracy. |
output_type |
enum | binary |
Verdict shape: binary, ordinal, scalar, pairwise, or structured. |
Pointwise items contain exactly two judged transcript elements: the complete
input and complete output. Any instructions belong in input; any visible
reasoning or execution details belong in output. Pairwise items replace the
pointwise value inside output with stable candidates plus an explicit
position_to_candidate mapping for A and B.
| Key | Type | Default | Meaning |
|---|---|---|---|
items_path |
path | required | JSONL file of items to audit. |
context_path |
path / null | null |
Optional explicit text, Markdown, or JSON description included in dataset-aware suggestion prompts. Documentation is not auto-discovered. |
input_field |
str | input |
Column mapped to the complete agent input, including instructions. |
output_field |
str | output |
Column mapped to the complete agent output; pairwise structure is nested here. |
gold_field |
str / null | gold_label |
Column with the gold label; set null if items have none. |
Binary gold is a normalized semantic class; pairwise gold is a stable preferred candidate ID; scalar gold is numeric. Reports use class/pairwise accuracy, or scalar MAE, Spearman rank correlation, and optional tolerance accuracy.
| Key | Type | Default | Meaning |
|---|---|---|---|
perturbation_operators |
list | [] (none) |
Explicit operator names to build; empty means no built-in invariant probes. See Perturbations. |
directional_perturbation_operators |
list | [] (none) |
Explicit allow-list of directional degradation operators. These probes intentionally make items worse and are reported separately from invariant flip-risk. |
custom_perturbations |
list | [] |
User-defined LLM-backed perturbations. Enabled custom perturbations always run, regardless of perturbation_operators. Each entry has name, target, instruction, optional expected_effect (same_verdict or worse_verdict), and optional enabled; targets are judge.prompt_template, judge.rubric, item.input, or item.output. |
plan_operators |
bool | false |
Auto-route the invariant and directional allow-lists with the Perturbation Suggester instead of hand-picking them. The suggester uses planner_agent when configured and otherwise defaults to perturbation_agent, independent of the judge. The resulting built-in lists replace the previous selections and remain editable. Needs a non-mock suggester model and its provider key; a failed suggestion call aborts the run. |
repeats |
int ≥ 1 | 10 |
Times every judge call is repeated — the original (which sets the baseline and observed repeat noise) and each perturbation variant — so a variant's flip rate is measured on equal footing with the baseline noise. |
num_workers |
int ≥ 1 | 1 |
Worker threads issuing perturbation-generation and judge calls concurrently. 1 runs each phase sequentially; higher values overlap per-call API latency. Results are order-stable regardless of this value. |
scalar_delta |
float > 0 | 1.0 |
Scalar margin for the boundary-proximity signal: an item's proximity rises from 0 at 2·scalar_delta away from judge.threshold to 1 at the threshold. Used only as review context; item risk is ranked by raw invariant flip risk. |
top_k_routes |
int ≥ 1 | 10 |
Number of highest flip-risk items listed in the review section of the report. |
bootstrap_samples |
int ≥ 100 | 1000 |
Resamples used for bootstrap confidence intervals. |
bootstrap_seed |
int | 42 |
RNG seed for reproducible bootstrap intervals. |
Example custom perturbation:
audit:
custom_perturbations:
- name: Legal wording shift
target: item.input
instruction: Rewrite the input in dense legal language while preserving every fact and requirement.
expected_effect: same_verdict
enabled: true| Key | Type | Default | Meaning |
|---|---|---|---|
runs_dir |
path | runs |
Directory under which each run's self-contained report is written. |
| Key | Type | Default | Meaning |
|---|---|---|---|
provider |
str | openai |
Built-in or config-defined provider profile for generation. |
model |
str | mock |
Model that generates LLM-backed dimensions. Must be independent of the judge under test. mock supports offline demos; explicitly selected LLM probes require a configured generator. |
decoding_params |
map | {} |
Decoding options forwarded to the perturbation-agent runner. |
When omitted, the validator inherits the perturbation agent's provider, model, and decoding parameters. Semantic probes require an available semantic validator; structural rule checks do not substitute for it. Freeze the validator explicitly when comparing generation models.
| Key | Type | Default | Meaning |
|---|---|---|---|
provider |
str / null | null |
Provider override. Setting it also requires an explicit validator model. |
model |
str / null | null |
Model used only for LLM admissibility validation. |
decoding_params |
map / null | null |
Decoding options forwarded to the validator; null inherits the perturbation-agent settings. |
When omitted, the suggester inherits the perturbation agent's provider, model, and decoding parameters.
| Key | Type | Default | Meaning |
|---|---|---|---|
provider |
str / null | null |
Provider override. Setting it also requires an explicit suggester model. |
model |
str / null | null |
Model used only for perturbation suggestions when audit.plan_operators is true. Set this to route with the suggester even when perturbation_agent.model is mock. |
decoding_params |
map / null | null |
Decoding options forwarded to the suggester; null inherits the perturbation-agent settings. |
sampling_seed |
int | 42 |
Seed for the planner's sampled dataset examples. |
dataset_context_max_chars |
int ≥ 1000 | 40000 |
Maximum characters of dataset statistics, explicit context, and sampled items sent to either suggestion feature. Small datasets are included fully; larger ones use a seeded sample recorded in planner metadata. |
When omitted, the summary agent inherits the perturbation agent's provider,
model, and decoding parameters. If the resolved model is mock or its provider
key is absent, the report uses the deterministic summary.
| Key | Type | Default | Meaning |
|---|---|---|---|
provider |
str / null | null |
Provider override. Setting it also requires an explicit summary model. |
model |
str / null | null |
Model used only for the report interpretation summary. |
decoding_params |
map / null | null |
Decoding options forwarded to the summary agent; null inherits the perturbation-agent settings. |
Pricing is not a config field. The run report prices recorded token usage
from every LLM-backed phase: judge calls, operator planning, perturbation
generation, equivalence validation, and report generation. The cost artifact
contains aggregate totals and a by_category breakdown with calls, input
tokens, cached-read tokens, cache-write tokens, output tokens, and USD for each
active phase.
Costs use the central provider-and-model table in
pricing.py (USD per 1M input, cached-input, and output
tokens, plus cache writes where the provider charges separately). Edit or extend
that table for the models you run. When a call lacks a
price entry or token usage, the report marks pricing as incomplete and presents
the known USD subtotal as a lower bound. Models without price entries are also
listed under unpriced_models. The mock runner makes no paid model calls and
reports no token usage.
All OSS components used within the product are listed below (including their copyright holders and the license information).
For OSS components having different portions released under different licenses, please refer to the included Upstream link(s) specified for each of the respective OSS components for identifications of code files released under the identified licenses.
| ID | OSS Component Name | Modified | Copyright Holder | Upstream Link | License |
|---|---|---|---|---|---|
| 1 | fastapi | No | Sebastián Ramírez | link | MIT License |
| 2 | hatchling | No | Ofek Lev | link | MIT License |
| 3 | httpx | No | Encode OSS Ltd | link | BSD 3-Clause License |
| 4 | jinja2 | No | Pallets | link | BSD 3-Clause License |
| 5 | openai | No | OpenAI | link | Apache License 2.0 |
| 6 | pydantic | No | Pydantic Services Inc. and individual contributors | link | MIT License |
| 7 | python-dotenv | No | Saurabh Kumar, Ted Tieken, and Jacob Kaplan-Moss | link | BSD 3-Clause License |
| 8 | python-multipart | No | Andrew Dunham and contributors | link | Apache License 2.0 |
| 9 | PyYAML | No | Ingy döt Net and Kirill Simonov | link | MIT License |
| 10 | tqdm | No | Casper da Costa-Luis, Noam Yorav-Raphael, Google Inc., and other contributors | link | MIT License and Mozilla Public License 2.0 (different portions) |
| 11 | uvicorn | No | Encode OSS Ltd | link | BSD 3-Clause License |