aratest checks the claims of an
Agent-Native Research Artifact
(ARA) against the numbers the ARA reports and against fresh experiment results.
You declare, for each claim, the properties that must be true if the claim is,
and aratest decides each one: it holds, it is violated with a witness that
names the failing case, or it stays undecided. Every violation is saved as a
self-contained record that aratest replay can recheck later without the ARA.
Install from GitHub with uv:
uv venv --python 3.12
source .venv/bin/activate
uv pip install "git+https://github.com/ARA-Labs/aratest.git"aratest needs at least Python 3.12.
Given an ARA whose evidence/tables/accuracy.md reports Ours at 95.1 and
Baseline at 95.9 on SVHN, this aratest/spec.yaml tests the claim
"Ours beats Baseline on every benchmark":
format_version: 7
evidence:
evidence/tables/accuracy.md:
metric: {name: accuracy}
axes: {Method: method, column: benchmark}
claims:
- claim_id: C01
properties:
- kind: consistency
metric: accuracy
comparator: gt
treatment: {method: Ours}
baseline: {method: Baseline}
over: benchmark
in_scope: [CIFAR-10, CIFAR-100, SVHN]$ aratest test my-ara
ARA: my-ara
C01: violated
consistency@1: violated (witnesses: [["benchmark","SVHN"]])
refinement: inconsistency
note: 2 of 3 in-scope values pass (min_pass=all); hard fails at benchmark in [SVHN]
record: .aratest/records/75b492966d92696af181f73c1bed4fc8660a6e48141546b7860131690fa015bb.json
0 hold, 1 violated, 0 undecided; exit code 1This repository ships two distributions that share one version:
-
aratest: the library and thearatestcommand. It reads an ARA, checks declared properties with a catalog of relations, and saves and replays check records. It needs no model provider and no Docker. -
dissclaimer: an optional plugin with theopenhandsexperiment agent, thearatest continuouscommand, and an agent skill. Install it with both packages from GitHub:uv pip install \ "git+https://github.com/ARA-Labs/aratest.git" \ "git+https://github.com/ARA-Labs/aratest.git#subdirectory=plugins/dissclaimer"
- Getting started
- Concepts
- The spec file
- Relations and families
- Command line
- Extending aratest
- dissclaimer
The repository is a uv workspace:
uv sync --all-packages
uv run pytest
uv run ruff check .
uv run mypy src plugins/dissclaimer/src packagingMIT. See LICENSE.