Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .agents/gateways/RULES.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S

3a. **Agent-side env vars are SDK placeholders, set by the eval image.** Agents inside the eval image consult provider-native env vars to satisfy their SDK's startup requirements (`OPENAI_API_KEY=sk-proxy`, `ANTHROPIC_API_KEY=sk-proxy`, `GEMINI_API_KEY=sk-proxy`). The value is a placeholder because the agent talks to the local gateway (which accepts any auth header), not the real provider. Eval images MUST set these placeholders so each SDK boots; the gateway's upstream credentials are read from the *gateway* container's env, populated separately at deploy time (k8s Secret, compose `env_file`, etc.).

4. **Configuration as template.** Where the gateway requires a static config file (`config.json`, `config.yaml`, etc.), the image MUST ship the file as a `.template` with `${VAR}` placeholders. The `start` script MUST render the live config via `envsubst` (or equivalent) at container startup, before launching the gateway process. Hardcoded provider blocks, model names, or URLs in the committed template are forbidden. Credential refs MUST point to provider-native env vars (per rule 3), not framework aliases.
4. **Configuration as template.** Where the gateway requires a static config file (`config.json`, `config.yaml`, etc.), the image MUST ship the file as a `.template` with `${VAR}` placeholders. The `start` script MUST render the live config from the template at container startup, before launching the gateway process. Hardcoded provider blocks, model names, or URLs in the committed template are forbidden. Credential refs MUST point to provider-native env vars (per rule 3), not framework aliases.

### Path-Prefix Protocol Namespace

Expand Down Expand Up @@ -123,3 +123,4 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S
|------|--------|
| 2026-05-17 | Initial version. Defines provider-agnostic gateway images, the `/<protocol>/<path>` URL namespace, the `EVAL_MODEL=<provider>/<model>` env contract, and the gateway↔model separation (gateways/ holds implementations, models/ holds pre-built combos). |
| 2026-05-18 | Rule 2a added: gateways MUST work with any model out of the box (one-env-var swap, no rebuild). Rule 16 rewritten: `models/<model>--<gateway>` combo images are OPTIONAL convenience wrappers, MUST equal bare-gateway-plus-mounted-template behavior. |
| 2026-07-06 | Rule 4 reworded: the `start` script renders config "from the template" at startup; the rule no longer names a specific render tool (gateways render with POSIX `sed`). |
4 changes: 2 additions & 2 deletions containers/benchmarks/_chart/values.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Defaults for the shared eval Job. The benchmark is selected with
# `--set benchmark=<x>` (required); a benchmark with bespoke topology overrides
# what differs via an optional `presets/<x>.yaml` in this chart. The per-run axes
# (agent, task, model, …) come from `--set` (or the eval-containers CLI). envsubst
# is gone: Helm fills these and gets the types right (task stays a quoted string).
# (agent, task, model, …) come from `--set` (or the eval-containers CLI). Helm
# fills these and gets the types right (task stays a quoted string).

# ── Per-run axes (overridden at deploy via --set / the eval-containers CLI) ──
agent: claude-code
Expand Down
2 changes: 1 addition & 1 deletion containers/core/benchmark-base-duckdb/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ ARG DUCKDB_VERSION=1.5.3
RUN set -e; ok=0; \
for i in 1 2 3 4 5; do \
if apt-get update --fix-missing && apt-get install -y --no-install-recommends \
curl ca-certificates unzip jq gettext-base; then ok=1; break; fi; \
curl ca-certificates unzip jq; then ok=1; break; fi; \
echo "apt attempt $i failed; retry" >&2; sleep $((i*5)); \
done; \
[ "$ok" = 1 ] || { echo "apt install failed after 5 tries" >&2; exit 1; }; \
Expand Down
2 changes: 1 addition & 1 deletion containers/core/benchmark-base-external/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ RUN set -e; ok=0; \
if apt-get update --fix-missing \
&& apt-get upgrade -y \
&& apt-get install -y --no-install-recommends \
curl ca-certificates git jq gettext-base; then ok=1; break; fi; \
curl ca-certificates git jq; then ok=1; break; fi; \
echo "apt attempt $i failed; retry" >&2; sleep $((i*5)); \
done; \
[ "$ok" = 1 ] || { echo "apt install failed after 5 tries" >&2; exit 1; }; \
Expand Down
2 changes: 1 addition & 1 deletion containers/core/benchmark-base-github/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ RUN set -e; ok=0; \
if apt-get update --fix-missing \
&& apt-get upgrade -y \
&& apt-get install -y --no-install-recommends \
curl ca-certificates git jq gettext-base; then ok=1; break; fi; \
curl ca-certificates git jq; then ok=1; break; fi; \
echo "apt attempt $i failed; retry" >&2; sleep $((i*5)); \
done; \
[ "$ok" = 1 ] || { echo "apt install failed after 5 tries" >&2; exit 1; }; \
Expand Down
2 changes: 1 addition & 1 deletion containers/core/benchmark-base-hf/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ RUN set -e; ok=0; \
if apt-get update --fix-missing \
&& apt-get upgrade -y \
&& apt-get install -y --no-install-recommends \
curl ca-certificates git jq gettext-base; then ok=1; break; fi; \
curl ca-certificates git jq; then ok=1; break; fi; \
echo "apt attempt $i failed; retry" >&2; sleep $((i*5)); \
done; \
[ "$ok" = 1 ] || { echo "apt install failed after 5 tries" >&2; exit 1; }; \
Expand Down
2 changes: 1 addition & 1 deletion containers/core/benchmark-base-python-slim/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ LABEL eval.base.kind="python-slim"
RUN set -e; ok=0; \
for i in 1 2 3 4 5; do \
if apt-get update --fix-missing && apt-get install -y --no-install-recommends \
jq ca-certificates curl gettext-base; then ok=1; break; fi; \
jq ca-certificates curl; then ok=1; break; fi; \
echo "apt attempt $i failed; retry" >&2; sleep $((i*5)); \
done; \
[ "$ok" = 1 ] || { echo "apt install failed after 5 tries" >&2; exit 1; }; \
Expand Down
2 changes: 1 addition & 1 deletion containers/core/benchmark-base-slim/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ LABEL eval.base.kind="slim"
RUN set -e; ok=0; \
for i in 1 2 3 4 5; do \
if apt-get update --fix-missing && apt-get install -y --no-install-recommends \
jq ca-certificates curl gettext-base; then ok=1; break; fi; \
jq ca-certificates curl; then ok=1; break; fi; \
echo "apt attempt $i failed; retry" >&2; sleep $((i*5)); \
done; \
[ "$ok" = 1 ] || { echo "apt install failed after 5 tries" >&2; exit 1; }; \
Expand Down
4 changes: 2 additions & 2 deletions containers/gateways/bifrost/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# /opt/gateway/data/ also where bifrost writes its SQLite
#
# Honest size: ~105 MB. The Go binary is statically linked, ca-certs +
# envsubst are the only runtime deps. UI is `go:embed`-ed into the
# curl are the only runtime deps. UI is `go:embed`-ed into the
# binary unconditionally upstream (no flag to strip without forking).
#
# Bifrost natively exposes the framework's path-prefix namespace
Expand All @@ -32,7 +32,7 @@ LABEL gateway.kind="bifrost"
LABEL gateway.bifrost_version="${BIFROST_VERSION}"
LABEL gateway.translates_protocols="true"

RUN apk add --no-cache ca-certificates curl gettext
RUN apk add --no-cache ca-certificates curl

# Pull just the binary out of the upstream distribution. The /app/data
# directory and the dashboard's docker-entrypoint.sh are not copied —
Expand Down
2 changes: 1 addition & 1 deletion containers/gateways/litellm/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ LABEL gateway.translates_protocols="true"
# eval image (FROM benchmark-base-hf, debian) already has it; we add
# it here so the same /opt/gateway/health works when this image runs
# standalone in compose mode.
RUN apt-get update && apt-get install -y --no-install-recommends curl ca-certificates gettext-base \
RUN apt-get update && apt-get install -y --no-install-recommends curl ca-certificates \
&& rm -rf /var/lib/apt/lists/*

# Caddy: tiny shim that maps the framework's protocol-namespaced URLs
Expand Down
2 changes: 1 addition & 1 deletion containers/gateways/portkey/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ LABEL gateway.portkey_version="${PORTKEY_VERSION}"
LABEL gateway.translates_protocols="false"
LABEL gateway.protocols="openai"

RUN apk add --no-cache ca-certificates curl gettext
RUN apk add --no-cache ca-certificates curl

# Caddy: path-prefix + header-injection shim.
COPY --from=docker.io/library/caddy:2.8-alpine /usr/bin/caddy /usr/bin/caddy
Expand Down
4 changes: 2 additions & 2 deletions containers/models/litellm/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Provider-agnostic litellm gateway image. Provider, model name, and
# credentials are resolved at runtime from env:
# UPSTREAM_MODEL <provider>/<model> — e.g., openai/gpt-4,
# EVAL_MODEL <provider>/<model> — e.g., openai/gpt-4,
# anthropic/claude-3-5-sonnet, bedrock/anthropic...
# Litellm parses the prefix natively to dispatch
# to the right provider plugin.
Expand All @@ -14,5 +14,5 @@ LABEL gateway.kind="litellm"

# litellm reads /opt/gateway/config.yaml (set by gateways/litellm's
# `start` script via --config /opt/gateway/config.yaml). We ship a
# TEMPLATE; the start script envsubsts UPSTREAM_MODEL into it at boot.
# TEMPLATE; the start script fills EVAL_MODEL into it at boot.
COPY config.yaml.template /opt/gateway/config.yaml.template
2 changes: 1 addition & 1 deletion containers/models/litellm/config.yaml.template
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Provider-agnostic litellm config. EVAL_MODEL=<provider>/<model>
# (envsubst'd at startup) tells litellm which provider plugin to dispatch
# (rendered at startup) tells litellm which provider plugin to dispatch
# to and which upstream model name to use. Upstream credentials are read
# via `os.environ/VAR` — by provider-native names, whatever the user's
# SDK conventionally reads (OPENAI_API_KEY+OPENAI_API_BASE for OpenAI-
Expand Down