feat: add --model-provider flag for keyless Bedrock LLM judge evaluators by stone-coding · Pull Request #1828 · aws/agentcore-cli

stone-coding · 2026-07-24T16:42:20Z

Description

Adds --model-provider <openai|bedrock> flag to the add evaluator command for third-party evaluation
libraries. When bedrock is selected, the CLI generates Lambda handlers that use Amazon Bedrock as the LLM
judge instead of OpenAI — fully keyless, IAM-only auth (SigV4 via the Lambda execution role; the existing
bedrock:InvokeModel policy suffices).

DeepEval (bedrock): Uses AmazonBedrockModel from deepeval.models for native Bedrock access. Judge model
from BEDROCK_MODEL_ID env var (default anthropic.claude-3-haiku-20240307-v1:0).

Autoevals (bedrock): Uses LiteLLMClient from autoevals.litellm routing to the Bedrock Converse API.
Judge model from BEDROCK_MODEL_ID env var (default bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0, a
cross-region inference profile).

Default remains openai — the default path generates byte-identical output to before this change; no
breaking change. Templates use Handlebars {{#if ModelProviderBedrock}} conditionals. Dependencies are
dynamic: aiobotocore>=2.13.0 (DeepEval) / litellm>=1.60,<1.85 (Autoevals) only when bedrock is selected;
openai>=1.0.0 only on the Autoevals default path.

Usage:

agentcore add evaluator --name fact --level TRACE --3p-library autoevals --metric Factuality --model-provider
bedrock --param threshold=0.5

E2E tested on Lambda (us-west-2), scaffolded with the built CLI and deployed via agentcore deploy:

DeepEval + Bedrock (AnswerRelevancyMetric, Claude 3 Haiku judge): {"value": 1.0, "label": "Pass"} with
rationale
Autoevals + Bedrock (Factuality, Claude Sonnet 4 judge): {"value": 0.6, "label": "Pass"} with rationale
Deployed Lambdas have no environment variables — no OPENAI_API_KEY anywhere
Regression: evaluator scaffolded without the flag is byte-identical to pre-change output
Bundle size with litellm: 212–221 MB unzipped, under Lambda's 250 MB limit (needs platform-targeted
dependency install, e.g. pip --platform manylinux2014_aarch64 --only-binary=:all:)

Related Issue

Depends on #1779 (third-party eval library support). Runtime E2E requires the SDK third_party adapters from
bedrock-agentcore-sdk-python PR #568 (not yet on PyPI; tested against the locally built wheel).

Documentation PR

N/A — flag documented via --help text.

Type of Change

Testing

How have you tested the change?

I ran npm run test:unit and npm run test:integ
I ran npm run typecheck
I ran npm run lint
If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Additionally:

3 new unit tests for the render-context behavior (bedrock set / openai explicit / flag omitted)
Verified all 8 template variants (2 libraries × 2 providers × 2 files) render correctly; all Python outputs
pass py_compile
Deployed both Bedrock handlers to Lambda and confirmed valid Claude-judged scores with no OpenAI API key
Regression tested: OpenAI path scaffolds identical output to before

Checklist

I have read the CONTRIBUTING document
I have added any necessary tests that prove my fix is effective or my feature works
I have updated the documentation accordingly (N/A — no separate docs; flag is self-documented in --help)
I have added an appropriate example to the documentation to outline the feature, or no new docs are
needed
My changes generate no new warnings
Any dependent changes have been merged and published

---By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this
contribution, under the terms of your choice.

…t tests

…Eval + Autoevals)

…support - DeepEval template: add /tmp workaround + DEEPEVAL_TELEMETRY_OPT_OUT for Lambda - Autoevals template: fix scorer= → metric= to match SDK API - Rename --from-3p-library → --3p-library - Add --param key=value (repeatable) for metric constructor kwargs - Add --parameters-file for JSON file of kwargs - Add parseParamFlags() utility + unit tests - --param and --parameters-file are mutually exclusive

- Add e2e-tests/third-party-eval-lifecycle.test.ts: full lifecycle test for DeepEval + Autoevals 3P evaluators (create → add → deploy → invoke → run eval) - Fix autoevals template: params go to AutoevalsAdapter (not metric constructor) - Add openai>=1.0.0 to autoevals pyproject.toml (required by LLM-based scorers)

…K rename)

Add validation guard so that explicitly passing --type llm-as-a-judge with --3p-library fails with a clear error message instead of silently proceeding into an invalid state.

Adds --model-provider <openai|bedrock> flag to `add evaluator` command. When bedrock is selected: - DeepEval: uses AmazonBedrockModel (native Bedrock via aiobotocore) - Autoevals: uses LiteLLMClient routing to Bedrock Converse API Templates are conditional via Handlebars {{#if ModelProviderBedrock}}. Dependencies are dynamic: aiobotocore for DeepEval, litellm for Autoevals (only included when bedrock is selected). Default remains openai for backward compatibility. Tested end-to-end on Lambda: - DeepEval + Bedrock: 1.0, Pass (AnswerRelevancy, Claude Haiku) - Autoevals + Bedrock: 0.6, Pass (Factuality, Claude Sonnet 4) - No OpenAI API key, IAM-only auth

Copilot

Pull request overview

This PR extends the CLI’s third-party evaluator scaffolding to support Bedrock-backed “LLM-as-judge” execution via a new --model-provider <openai|bedrock> flag on agentcore add evaluator, generating provider-specific Python Lambda handlers and template dependencies.

Changes:

Add third-party evaluator library registry + CLI flags (--3p-library, --metric, --param/--parameters-file, --model-provider, --memory) and render Bedrock-aware templates.
Introduce new evaluator template assets for DeepEval and Autoevals (handler, pyproject, execution-role policy) with Handlebars conditionals for Bedrock paths.
Extend evaluator managed code config schema/type to support memorySizeMb.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/schema/schemas/primitives/evaluator.ts	Adds optional `memorySizeMb` to managed evaluator code config schema.
src/schema/llm-compacted/agentcore.ts	Mirrors `memorySizeMb` in the compacted schema interface.
src/cli/templates/EvaluatorRenderer.ts	Adds a renderer and template data shape for third-party evaluator templates.
src/cli/primitives/EvaluatorPrimitive.ts	Implements third-party evaluator registry/flag parsing and Bedrock/OpenAI template rendering.
src/cli/primitives/tests/EvaluatorPrimitive.test.ts	Adds unit tests covering third-party render context and param parsing helpers.
src/assets/evaluators/deepeval-lambda/pyproject.toml	New DeepEval Lambda pyproject with conditional Bedrock dependency.
src/assets/evaluators/deepeval-lambda/lambda_function.py	New DeepEval Lambda handler template with conditional Bedrock judge model wiring.
src/assets/evaluators/deepeval-lambda/execution-role-policy.json	New execution role policy template including Bedrock invoke permission.
src/assets/evaluators/autoevals-lambda/pyproject.toml	New Autoevals Lambda pyproject with conditional Bedrock (litellm) vs OpenAI deps.
src/assets/evaluators/autoevals-lambda/lambda_function.py	New Autoevals Lambda handler template with conditional Bedrock judge wiring.
src/assets/evaluators/autoevals-lambda/execution-role-policy.json	New execution role policy template including Bedrock invoke permission.
src/assets/tests/snapshots/assets.snapshot.test.ts.snap	Updates asset snapshot listing to include new evaluator template files.
e2e-tests/third-party-eval-lifecycle.test.ts	Adds an E2E lifecycle test for third-party evaluators (skipped unless AWS-capable).
.gitignore	Ignores a new E2E output directory.

Comments suppressed due to low confidence (1)

src/assets/evaluators/autoevals-lambda/lambda_function.py:34

When EvaluatorParams is empty (default when no --param/--parameters-file is provided), this renders AutoEvalsAdapter(..., ), which is invalid Python syntax due to the dangling comma. Remove the unconditional comma so the no-params case renders valid code.

adapter = AutoEvalsAdapter(metric={{ EvaluatorClass }}(), {{{ EvaluatorParams }}})

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+REGION = os.environ.get("AWS_REGION", "us-west-2")
+
+model = AmazonBedrockModel(model=MODEL_ID, region=REGION)
+adapter = DeepEvalAdapter(metric={{ EvaluatorClass }}(model=model, {{{ EvaluatorParams }}}))


+JUDGE_MODEL = os.environ.get("BEDROCK_MODEL_ID", "bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0")
+init(client=LiteLLMClient(), default_model=JUDGE_MODEL)
+
+adapter = AutoEvalsAdapter(metric={{ EvaluatorClass }}(client=LiteLLMClient(), model=JUDGE_MODEL), {{{ EvaluatorParams }}})


+export function jsonToKwargs(json: string): string {
+  const obj = JSON.parse(json) as Record<string, unknown>;
+  return Object.entries(obj)
+    .map(([key, value]) => `${key}=${jsonToPythonValue(value)}`)
+    .join(', ');
+}


+export function parseParamFlags(params: string[]): string {
+  return params
+    .map(param => {
+      const eqIndex = param.indexOf('=');
+      if (eqIndex === -1) {
+        throw new Error(`"${param}" is not in key=value format`);
+      }
+      const key = param.slice(0, eqIndex);
+      const rawValue = param.slice(eqIndex + 1);
+      let value: unknown;
+      try {
+        value = JSON.parse(rawValue);
+      } catch {
+        value = rawValue;
+      }
+      return `${key}=${jsonToPythonValue(value)}`;
+    })
+    .join(', ');
+}


chianhw · 2026-07-24T22:45:58Z

+              if (cliOptions.memory) {
+                const memVal = parseInt(cliOptions.memory, 10);
+                if (isNaN(memVal) || memVal < 128 || memVal > 10240) {
+                  fail('--memory must be an integer between 128 and 10240');
+                }
+              }


can we fix this?

chianhw · 2026-07-24T19:15:16Z

+# Autoevals grades via an OpenAI-compatible client. LiteLLMClient routes to Bedrock;
+# litellm auto-routes Anthropic Claude models through the Converse API. Cross-region
+# inference profiles (us.*/eu.*/apac.*) are required for on-demand invocation.
+JUDGE_MODEL = os.environ.get("BEDROCK_MODEL_ID", "bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0")


Do we need to use bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0 as the default model? This model is not available in all aws regions. Can we instead add a validation check to ensure --model is provided when the user selects Bedrock as the model provider?

chianhw · 2026-07-24T19:15:21Z

+from bedrock_agentcore.evaluation.custom_code_based_evaluators.third_party.deepeval import DeepEvalAdapter
+
+{{#if ModelProviderBedrock}}
+MODEL_ID = os.environ.get("BEDROCK_MODEL_ID", "anthropic.claude-3-haiku-20240307-v1:0")


we can modify and reuse the existing cli input option('--model <model>', '[LLM] Bedrock model ID for LLM-as-a-Judge').
Where possible, avoid relying on environment variables. The goal of the intern project is to reduce the customer effort required to set up environment configuration as much as possible.

Copilot

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

src/cli/primitives/EvaluatorPrimitive.ts:154

jsonToKwargs assumes the parsed JSON is an object and emits keys verbatim as Python kwarg names. If the JSON is an array/null/non-object or contains keys that are not valid Python identifiers, the generated handler will be invalid Python. Validate the top-level type and kwarg names and fail with a clear error.

export function jsonToKwargs(json: string): string {
  const obj = JSON.parse(json) as Record<string, unknown>;
  return Object.entries(obj)
    .map(([key, value]) => `${key}=${jsonToPythonValue(value)}`)
    .join(', ');
}

src/cli/primitives/EvaluatorPrimitive.ts:173

parseParamFlags emits the left-hand side of key=value directly as a Python kwarg name. If the key contains spaces/dashes/etc, the generated handler will not compile. Validate the key as a Python identifier and surface a user-friendly error.

export function parseParamFlags(params: string[]): string {
  return params
    .map(param => {
      const eqIndex = param.indexOf('=');
      if (eqIndex === -1) {
        throw new Error(`"${param}" is not in key=value format`);
      }
      const key = param.slice(0, eqIndex);
      const rawValue = param.slice(eqIndex + 1);
      let value: unknown;
      try {
        value = JSON.parse(rawValue);
      } catch {
        value = rawValue;
      }
      return `${key}=${jsonToPythonValue(value)}`;
    })
    .join(', ');

+function isSupportedLibrary(value: string): value is ThirdPartyLibrary {
+  return value in THIRD_PARTY_EVALUATOR_LIBRARIES;
+}


+init(client=LiteLLMClient(), default_model="{{ Model }}")
+
+adapter = AutoEvalsAdapter(metric={{ EvaluatorClass }}(client=LiteLLMClient(), model="{{ Model }}"), {{{ EvaluatorParams }}})


+              if (cliOptions.modelProvider === 'bedrock' && !cliOptions.model) {
+                fail('--model is required when using --model-provider bedrock');
+              }


…faults Per Irene's review feedback: - Remove hardcoded default model IDs (not available in all regions) - Require --model flag when --model-provider bedrock is selected - Bake model ID into generated template at scaffold time (no env vars) - Reuse existing --model CLI flag for Bedrock model selection

Copilot