Summary
In the test_case_evaluation stage, run_validation assigns each case a confidence score (a ValidationResult with predicted_label, verdict, confidence). By design, an ARES case is a jailbreak-transformed variant of a parent test case, so it should carry the same confidence score as its parent — it should not be independently re-validated the way Promptfoo cases are.
Fix
After the parent cases are validated in run_validation, propagate each parent's confidence score (and label/verdict) onto its derived ARES cases, keying off the parent record the ARES case was expanded from — rather than re-scoring the ARES cases independently. This requires preserving a parent reference through merge_with_ares (convert_test_case.py:67-84) so the ARES case can be matched back to its parent's ValidationResult.
Summary
In the test_case_evaluation stage, run_validation assigns each case a confidence score (a ValidationResult with
predicted_label, verdict, confidence). By design, an ARES case is a jailbreak-transformed variant of a parent test case, so it should carry the same confidence score as its parent — it should not be independently re-validated the way Promptfoo cases are.Fix
After the parent cases are validated in run_validation, propagate each parent's confidence score (and label/verdict) onto its derived ARES cases, keying off the parent record the ARES case was expanded from — rather than re-scoring the ARES cases independently. This requires preserving a parent reference through merge_with_ares (convert_test_case.py:67-84) so the ARES case can be matched back to its parent's ValidationResult.