Skip to content
ARA-LabsPublic

Repository files navigation

aratest: property-based testing for research claims

aratest checks the claims of an Agent-Native Research Artifact (ARA) against the numbers the ARA reports and against fresh experiment results. You declare, for each claim, the properties that must be true if the claim is, and aratest decides each one: it holds, it is violated with a witness that names the failing case, or it stays undecided. Every violation is saved as a self-contained record that aratest replay can recheck later without the ARA.

Install

Install from GitHub with uv:

uv venv --python 3.12
source .venv/bin/activate
uv pip install "git+https://github.com/ARA-Labs/aratest.git"

aratest needs at least Python 3.12.

Quick example

Given an ARA whose evidence/tables/accuracy.md reports Ours at 95.1 and Baseline at 95.9 on SVHN, this aratest/spec.yaml tests the claim "Ours beats Baseline on every benchmark":

format_version: 7
evidence:
  evidence/tables/accuracy.md:
    metric: {name: accuracy}
    axes: {Method: method, column: benchmark}
claims:
  - claim_id: C01
    properties:
      - kind: consistency
        metric: accuracy
        comparator: gt
        treatment: {method: Ours}
        baseline: {method: Baseline}
        over: benchmark
        in_scope: [CIFAR-10, CIFAR-100, SVHN]
$ aratest test my-ara
ARA: my-ara

C01: violated
  consistency@1: violated (witnesses: [["benchmark","SVHN"]])
    refinement: inconsistency
    note: 2 of 3 in-scope values pass (min_pass=all); hard fails at benchmark in [SVHN]
  record: .aratest/records/75b492966d92696af181f73c1bed4fc8660a6e48141546b7860131690fa015bb.json

0 hold, 1 violated, 0 undecided; exit code 1

Packages

This repository ships two distributions that share one version:

  • aratest: the library and the aratest command. It reads an ARA, checks declared properties with a catalog of relations, and saves and replays check records. It needs no model provider and no Docker.

  • dissclaimer: an optional plugin with the openhands experiment agent, the aratest continuous command, and an agent skill. Install it with both packages from GitHub:

    uv pip install \
      "git+https://github.com/ARA-Labs/aratest.git" \
      "git+https://github.com/ARA-Labs/aratest.git#subdirectory=plugins/dissclaimer"

Documentation

Development

The repository is a uv workspace:

uv sync --all-packages
uv run pytest
uv run ruff check .
uv run mypy src plugins/dissclaimer/src packaging

License

MIT. See LICENSE.

Releases

Packages

Contributors

Languages