Mech Reasoner

Mech Reasoner is a benchmark toolkit for qualitative mechanistic reasoning. It validates electrical, mechanical, and thermal mechanism catalogs; generates open-answer JSON tasks; evaluates structured model answers; and plots accuracy by task complexity.

The benchmark includes necessity, plausibility, state_consistency, transition, and functional_recovery. Generated tasks are divided into four approximately equal-count complexity levels (D1 through D4) within each task family. All generation requires simulation-validated artifacts in the current strict catalog schema. The transition family audits bounded subproblems of the episode relation derived by the simulator; small devices may materialize that relation, while larger devices derive the required portion on demand. There is no synthetic graph or declared-transition fallback.

The repository is organized around:

• catalog/: mechanism definitions used to generate benchmark tasks.
• src/catalog_generation/: catalog schema and loading support.
• src/simulator/: qualitative sign solver.
• src/task_generation/: benchmark task generation.
• src/evaluation/: model evaluation.
• src/analysis/: aggregate tables and plots.
• benchmark/: versioned generated tasks, solutions, evaluation records, and plots.

Requirements

• Python 3.13 or newer.
• uv for dependency and environment management.
• Docker and Modelica Standard Library 4.1.0 only for physical validation.

Installation

Install dependencies with uv:

uv sync

Install development tools:

uv sync --group dev

Quickstart

Validate the mechanism catalog:

uv run python -m src.main validate-catalog --catalog-dir catalog

Generate benchmark tasks:

uv run python -m src.main generate --catalog-dir catalog

Unless an output option is supplied, commands use the versioned benchmark directory selected by DEFAULT_OUTPUT_VERSION in src/task_generation/build_benchmark.py.

Evaluate ten generated tasks with Azure OpenAI. Evaluation concurrency is 20 by default and should remain 20 for benchmark runs:

export AZURE_BASE_URL="https://<resource>.openai.azure.com"
export AZURE_KEY="<api-key>"
export DEPLOYMENT="<deployment-name>"

uv run python -m src.main evaluate \
  --evaluator azure \
  --model-profile gpt-5.5 \
  --concurrency 20 \
  --limit 10

Plot evaluation accuracy:

uv run python -m src.main plot

To generate, evaluate, and plot in one command:

uv run python -m src.main run --catalog-dir catalog --concurrency 20

Pass --task-family <name> to generate, evaluate, or plot one task family. To store generated tasks separately from evaluations and plots, use --task-output-dir and --analysis-output-dir:

uv run python -m src.main generate \
  --catalog-dir catalog \
  --task-output-dir task_outputs
uv run python -m src.main evaluate \
  --task-output-dir task_outputs \
  --analysis-output-dir analysis_outputs \
  --concurrency 20
uv run python -m src.main plot --analysis-output-dir analysis_outputs

Generation writes task JSON files, solutions.jsonl, manifest.json, and selection_summary.csv under benchmark/<version>/. Evaluation records are stored by model and task family under benchmark/<version>/analysis/evaluations/; plotting writes per-family CSV, LaTeX, and PNG summaries under the same analysis/ directory.

All evaluator providers use one capability-adaptive benchmark protocol: low reasoning effort when that control is supported, temperature zero when available, JSON mode when available, a 600s request timeout, and concurrency 20 by default. The output budget is 20,000 tokens or the evaluator's declared maximum, whichever is lower. No separate reasoning-token budget is sent. Eligible models must support at least 80,000 input tokens. Rate-limit responses wait and retry automatically; every other infrastructure failure is recorded and retried only when --retry-errors is passed explicitly.

If an Azure deployment has an arbitrary name, identify its backing model with --model-profile so its capabilities can be validated without changing the deployment name sent to Azure:

uv run python -m src.main evaluate \
  --evaluator azure \
  --model production-reasoner \
  --model-profile gpt-5.5

For transition calibration, generate at most one hardest exact task per eligible catalog mechanism, or exactly ten sparse tasks from each globally supported equal-width interval:

uv run python -m src.task_generation.transition.generate \
  --catalog-dir catalog \
  --output-dir transition-hard \
  --selection-mode hard_distinct \
  --task-count 10

uv run python -m src.task_generation.transition.generate \
  --catalog-dir catalog \
  --output-dir transition-levels \
  --selection-mode global_levels \
  --tasks-per-level 10

Hard-distinct generation fails if the catalog cannot supply the requested number of different mechanisms. Pass --allow-shortfall only when an explicitly recorded, smaller capability sample is acceptable. Transition complexity is the number of exact simulator-derived source-target-cause triples audited. The calibrated default per-task cap is 128; equal-width selection and fixed per-mechanism quotas prevent one very large mechanism from monopolizing the global intervals.

Evaluation providers

Environment variables may be exported in the shell or placed in .env.

Azure OpenAI

Set AZURE_BASE_URL and AZURE_KEY. Set DEPLOYMENT, or pass a deployment name with --model:

uv run python -m src.main evaluate \
  --evaluator azure \
  --model gpt-5.5 \
  --concurrency 20

The llama evaluator uses the same Azure credentials and defaults to the Llama-4-Maverick-17B-128E-Instruct-FP8 deployment:

uv run python -m src.main evaluate --evaluator llama --concurrency 20

OpenAI

Set OPENAI_API_KEY. The default model is gpt-5.5; override it with --model:

uv run python -m src.main evaluate --evaluator openai --concurrency 20

DeepSeek on Azure

Set AZURE_RESOURCE_BASE_URL and AZURE_RESOURCE_KEY. The default deployment is DeepSeek-V3-0324. Models such as DeepSeek-R1 that do not expose reasoning effort control run with their native reasoning behavior:

uv run python -m src.main evaluate --evaluator deepseek --concurrency 20

OpenRouter

Set OPENROUTER_API_KEY (or OPENROUTER_TOKEN). The default model is qwen/qwen3-30b-a3b-thinking-2507:

uv run python -m src.main evaluate \
  --evaluator openrouter \
  --model qwen/qwen3-30b-a3b-thinking-2507 \
  --concurrency 20

Compiler-driven conversion and physical validation

The catalog is derived from compiler structure rather than regular expressions over Modelica source. The pinned toolchain is OpenModelica 1.26.9 with MSL 4.1.0. A shared catalog/component-library.yaml defines reusable generic types from Modelica class, inheritance, component, and equation APIs. Each type holds only local variables, local state partitions, function-free local laws, terminals, dynamic properties, and typed class-wide assumptions. Numeric parameters remain instance bindings.

Each model.yaml therefore contains component instances, operating contexts, and connection sets rather than a second set of authored system laws. Assembly alpha-renames the generic laws and derives physical compatibility and conservation laws from the declared material, conduits, terminals, and topology; directed signals derive their one-driver compatibility laws. Source time profiles live in operating contexts and never become structural component functions. Unsupported connector profiles, component functions, modes, or cross-component residues fail closed and leave the existing catalog untouched.

Every item retains compiler model-instance.json, flattened XML DAE, deterministic structural.json, compiler-resolved simulation options, a validation manifest, and exactly one nominal trace. structural.json is the sole deterministic structural-extraction artifact. The certificate hashes the shared library and all item artifacts and requires complete structural accounting and passed trace soundness. It records honestly whether interpretation realizability or a fully materialized episode graph was proven.

The solver, causal account, trace validator, and task generators consume the assembled device. Exact episode graphs are materialized only for small devices; larger devices use the same topology-local transition rules on demand to keep conversion within the reviewed work budget.

Validation uses the model's declared start/stop time, tolerance, and output interval by default. --start-time, --stop-time, --tolerance, and either --number-of-intervals or --interval provide explicit audited overrides.

Convert and validate the eighteen admitted catalog targets:

uv run python -m src.modelica_to_qualitative \
  --selected all \
  --modelica-root ModelicaStandardLibrary_v4.1.0 \
  --catalog-dir catalog

Twelve targets are stock MSL 4.1 examples: one electrical (Resistor), seven translational (SignConvention, Accelerate, Damper, InitialConditions, Oscillator, WhyArrows, and CompareBrakingForce), three rotational (FirstGrounded, First, and CompareBrakingTorque), and one thermal (TwoMasses). Six repository-authored Modelica compositions reuse only already-supported MSL component classes: an electrothermal resistor ladder, an electrothermal parallel network, a three-node thermal chain, a branched thermal star, a branched three-mass spring--damper system, and a grounded geared dual-inertia rotational system.

The electrothermal models and Resistor use the audited operating window 0.01--0.49 s, so their single nominal traces remain within one continuous source regime. The two braking comparisons use 0--0.4 s with 1,000 intervals; this audited context ends before the stop-event discontinuities that require multiple operating contexts. Command-line time overrides take precedence over these per-job defaults. The configured profile targets a complete conversion and validation run within three minutes per item. Nine original targets are explicit converter limitations and are skipped rather than approximated:

• ParallelResonance and SeriesResonance: variable-frequency sinusoidal sources need phase-region states; a sign envelope admits histories outside the prescribed input.

• ShowVariableResistor: its mixed variable-resistance network exhausts the bounded intrastate solver before realizability can be established.

• EddyCurrentBrake: its nonlinear torque depends jointly on excitation, speed, and temperature; those physical coordinates cannot be replaced by arbitrary source histories.

• DemoPowerSupply: algorithmic initialization and piecewise source modes are outside the function-free local polynomial profile.

• RollingWheel: IdealRollingWheel has internal physical topology; recursive composite expansion is outside the bounded converter.

• ElasticBearing: exact episode construction exceeded the 175-second watchdog.

• translational Sensors: its power output needs stable second derivatives of a derived product, which the nominal trace cannot certify without numerical differentiation artifacts.

• thermal Motor: piecewise table-source jumps require multiple operating contexts and violate the single-context zero-time derivative check.

These limitations are inventory metadata, not model-name behavior templates; the generic converter reaches them through unsupported-structure or work-budget checks. Previously generated benchmark tasks, evaluations, and plots were invalidated by the schema and solver replacement and were not regenerated in this migration.

The non-catalog regression model Modelica.Thermal.HeatTransfer.Examples.ControlledTemperature, is expected to return inconclusive: it exposes a Boolean connector outside the reviewed real-signal connector profile. It remains a fail-closed integration-test fixture only and is never written to catalog/.

Convert one custom model:

uv run python -m src.modelica_to_qualitative \
  --source path/to/Model.mo \
  --model-name Package.Model \
  --catalog-id my_model \
  --modelica-root ModelicaStandardLibrary_v4.1.0 \
  --catalog-dir catalog

Exit status 0 means validation passed, 1 means a trace or compiler failure, and 2 means conversion was inconclusive (for example, an unavailable compiler or unsupported bounded-profile construct). Existing catalog artifacts are not replaced on failed or inconclusive runs.

Requirements

Simulation runs in Docker. Pull the OpenModelica image once:

docker pull openmodelica/openmodelica:v1.26.9-minimal

Download and unpack Modelica Standard Library 4.1.0 so its source tree has this layout (the library is not tracked by this repository):

ModelicaStandardLibrary_v4.1.0/
└── Modelica 4.1.0/
    ├── Electrical/
    ├── Mechanics/
    └── Thermal/

validate-catalog is the strict task-readiness gate: it loads the shared component library, assembles every model, verifies each sibling validation.json and its bound artifact hashes, and requires complete structural accounting and passed trace soundness.

uv run python -m src.main validate-catalog --catalog-dir catalog

Batch certificate validation performs the same strict check for each catalog item and verifies that nested provenance.source_modelica_path resolves to an available source file under the supplied Modelica root:

uv run python scripts/validate_catalog_all.py \
  --catalog-dir catalog \
  --modelica-root ModelicaStandardLibrary_v4.1.0

Development

Format Python code with Black:

uv run black src tests scripts

Run Ruff checks:

uv run ruff check src tests scripts

Run the default test suite:

uv run pytest

Run the complete release checks with pre-commit. Gitleaks must be installed separately; on macOS it is available through Homebrew:

brew install gitleaks
uv run pre-commit install
uv run pre-commit run --all-files

License

Mech Reasoner is distributed under the GNU General Public License, version 3 only (GPL-3.0-only). See LICENSES/GPL-3.0-only.txt.