[PAI-1739] feat: add an eval harness for the MCP tool surface - #200

Open

dheeru0198 wants to merge 94 commits into

mainfrom

feat/mcp-eval-harness

Commits on Aug 12, 2026

Add the MCP tool-surface eval harness

dheeru0198
and
claude
committed
1607b9d
Split the driver monolith into a package

dheeru0198
and
claude
committed
936c072
Replace the Anthropic-only SDK path with a generic API driver

dheeru0198
and
claude
committed
6af196c
Report response-token cost for every driver, and label estimates

dheeru0198
and
claude
committed
4afe655
Split the task catalog into one module per task class

dheeru0198
and
claude
committed
c735b1c
Separate the CLI from the run loop

dheeru0198
and
claude
committed
19cd92f
Reconcile the eval docs with the code

dheeru0198
and
claude
committed
adf6534

Commits on Aug 13, 2026

Make the API driver provider-neutral and the model tiers vendor-free

dheeru0198
and
claude
committed
ea50b74
Collapse the four CLI driver bodies into one template

dheeru0198
and
claude
committed
b8f4617
Declare the result schema instead of assembling it by hand

dheeru0198
and
claude
committed
3ddebae
Grade answers by contract, not by prose style

dheeru0198
and
claude
committed
de9190c
Aggregate repetitions instead of showing the last one

dheeru0198
and
claude
committed
9d7c759
Split the fixture builders by Plane object

dheeru0198
and
claude
committed
87fb16e
Split the report by job and read rows only through the schema

dheeru0198
and
claude
committed
38e1991
Group the run loop into a package and place helpers with their users

dheeru0198
and
claude
committed
5dbdbff
Drop the abandoned v2 surface vocabulary from the eval harness
dheeru0198
committed
ba60db7
Give the CLI drivers the same package shape as the API driver
dheeru0198
committed
0700a96
Stop treating a local Plane instance as an eval prerequisite
dheeru0198
committed
22ddca4
Put the two drivers at the top of the drivers package
dheeru0198
committed
128a877
Split the task package by concern
dheeru0198
committed
694ae76
Give the report summary a type instead of a magic key
dheeru0198
committed
6feffb7
Break the task repetition into named stages, behind a pinned taxonomy
dheeru0198
committed
f35b498
Group the tests the way the harness is grouped
dheeru0198
committed
6ad1b1c
Report progress while a battery runs
dheeru0198
committed
b082b06

Commits on Aug 14, 2026

Make a feature exclusion mean the feature is off

dheeru0198
and
claude
committed
1a1a07a
Require a plan refusal to say so before skipping a task

dheeru0198
and
claude
committed
79ac46d
Skip a task whose capability the plan excludes, instead of erroring

dheeru0198
and
claude
committed
6e6bacf
Add W11: log work against a project with time tracking switched off

dheeru0198
and
claude
committed
cd30fda
Stop documenting the flag server as a prerequisite

dheeru0198
and
claude
committed
7064dd9
Halve the eval test suite by folding duplicated cases together

dheeru0198
and
claude
committed
8d37e00
Tighten the long docstrings without dropping what they know

dheeru0198
and
claude
committed
acd6e3d

Commits on Aug 15, 2026

Make fixture seeding and teardown own what they touch

dheeru0198
and
claude
committed
ef82281
Stop verifiers passing tasks they could not verify

dheeru0198
and
claude
committed
b4378c0
Report what a run actually measured, not what it attempted

dheeru0198
and
claude
committed
05debe9
Keep the call trace honest about what it dropped

dheeru0198
and
claude
committed
7fe80ec
Restore per-case test reporting, and document what changed

dheeru0198
and
claude
committed
cd900bc
Bind read provenance to the target entity, and stop shipping the secret

dheeru0198
and
claude
committed
12933c7
Count what the proxy lost, and identify the surface it recorded

dheeru0198
and
claude
committed
8177441
Refuse comparisons the persisted identity cannot establish

dheeru0198
and
claude
committed
0f7392f
Give each repetition its own fixture seed, and record it

dheeru0198
and
claude
committed
031037f
Make trace loss visible in the run verdict, and validate the exact key set

dheeru0198
and
claude
committed
da12a35
Record a trace the CLI can actually produce

dheeru0198
and
claude
committed
b9eb079

Commits on Aug 16, 2026

Stop leaking the answers, and give each proxy its own file

dheeru0198
and
claude
committed
619e937
Measure off-surface work instead of asserting it cannot happen

dheeru0198
and
claude
committed
c232656
Close the reporting gaps, and say only what the tests establish

dheeru0198
and
claude
committed
9feaf78
Count the calls that failed, not just the calls

dheeru0198
and
claude
committed
99166e5
Say what the fixture seed actually guarantees

dheeru0198
and
claude
committed
dbcc6a5
Let the isolated Codex home actually make tool calls

dheeru0198
and
claude
committed
054cf72
Stop demanding a tool manifest from rows that never reached the surface

dheeru0198
and
claude
committed
974c097

Commits on Aug 17, 2026

Merge remote-tracking branch 'origin/main' into feat/mcp-eval-harness
dheeru0198
committed
70d3c4b
Read the attachment list the shape the SDK returns it in

dheeru0198
and
claude
committed
74b4372
Ask L2 for evidence Plane can actually give

dheeru0198
and
claude
committed
5dbffca
Let a count be evidence at the seed gate too

dheeru0198
and
claude
committed
f190185
Bind R1 and I2 evidence to the state that carries the answer

dheeru0198
and
claude
committed
8da1d6c
Let R6 be proven by a count per project

dheeru0198
and
claude
committed
181840c
Make provenance a property instead of a list of accepted routes

dheeru0198
and
claude
committed
275683a
Stop charging infrastructure failures to the model

dheeru0198
and
claude
committed
e980b69
Close the holes the regrade found in the new provenance rule

dheeru0198
and
claude
committed
12f9538
Skip the opencode config readback instead of failing on it

dheeru0198
and
claude
committed
0e0a355
Fix two fixtures the smoke run caught

dheeru0198
and
claude
committed
a508129
Keep the refusal that caused a skip

dheeru0198
and
claude
committed
2f65d15
Give the plan-gate policy and the server env their own modules

dheeru0198
and
claude
committed
e34a877
Report the run that was executed, not the checkout

dheeru0198
and
claude
committed
55d9842
Give each driver surface a base its vendors implement
dheeru0198
committed
27d589f
Load only the driver surface the run asked for
dheeru0198
committed
95b2010

Commits on Aug 18, 2026

Give the shared floor of the package a name
dheeru0198
committed
fdd9714
Test the five verifiers that could pass a wrong answer
dheeru0198
committed
637d2a7

Commits on Aug 19, 2026

Stop a parked stdin read from discarding a whole trace
dheeru0198
committed
bd4b7cd
Stop handing the agent a shell to route around the surface
dheeru0198
committed
c3d3a71
Score the surface, not the database
dheeru0198
committed
4ddc39c
Give seeded projects names with nothing id-shaped in them
dheeru0198
committed
5c88274
Make the Antigravity driver actually reach agy
dheeru0198
committed
56372c8

Commits on Aug 20, 2026

Split the errored-call count by what kind of "no" a call received
dheeru0198
committed
ff9e4b3
Document the errored-call split in DESIGN and README
dheeru0198
committed
ce2158a
Count a refusal the server reports as a successful result
dheeru0198
committed
85708d9
Survive a project-name collision, and stop reporting a dirty workspace as clean

dheeru0198
and
claude
committed
8646581
Record the request beside a recorded result

dheeru0198
and
claude
committed
d5cb995

Commits on Aug 21, 2026

Adopt the project an ambiguous create leaves behind

dheeru0198
and
claude
committed
8978f2a
Document the success-flagged refusal count and recorded request args

dheeru0198
and
claude
committed
824ef67
Declare the OpenAI eval provider's package

dheeru0198
and
claude
committed
8e1681e

Commits on Aug 24, 2026

Move the OpenAI eval backend to /v1/responses

dheeru0198
and
claude
committed
16b0e80
Record the cause inside an ExceptionGroup, not its sub-exception count

dheeru0198
and
claude
committed
75da834
Ask Anthropic for prompt caching

dheeru0198
and
claude
committed
fd09347
Make the agent-loop iteration cap settable

dheeru0198
and
claude
committed
3c786b6
feat(evals): normalise token accounting and price runs

dheeru0198
and
claude
committed
f2a41b6
feat(evals): report cost, input, result volume and latency in the A/B view

dheeru0198
and
claude
committed
ac2fcfc
feat(evals): classify failures by kind, not just pass/fail

dheeru0198
and
claude
committed
27e3d13
feat(evals): warn when per-task verdicts are underpowered

dheeru0198
and
claude
committed
e610d15
fix(evals): eight defects from the adversarial review of the pricing change

dheeru0198
and
claude
committed
979b0cd
feat(evals): measure redundant lookups, and record call args on every driver

dheeru0198
and
claude
committed
7352003
docs(evals): describe cost, failure kinds, redundant lookups and power

dheeru0198
and
claude
committed
ed7c8a6
docs(evals): README no longer ties request args to payload recording

dheeru0198
and
claude
committed
dec701e
fix(evals): ten defects from the second adversarial review

dheeru0198
and
claude
committed
3d7f0e8

Commits on Aug 25, 2026

fix(tests): bind SDK signatures that Python 3.14 will not evaluate

dheeru0198
and
claude
committed
35e91b9