Run Enterprise-Bench L1–L2 with Codex + GPT-5.6 Sol (Harbor harness)

## Summary

Track an evaluation of **OpenAI Codex** (`-a codex`) with **GPT-5.6 Sol** (`-m gpt-5.6-sol`) against **`Enterprise-Bench/l1-l2-bench`** on Harbor Hub.

**Harness:** Harbor (job runner, Docker trials, judge, jobs artifacts)  
**Agent under test:** OpenAI Codex CLI (`@openai/codex`, installed per trial container)  
**Dataset:** [Enterprise-Bench/l1-l2-bench](https://hub.harborframework.com/datasets/enterprise-bench/l1-l2-bench) (14 tasks)  
**Primary model:** `gpt-5.6-sol` (flagship of the GPT-5.6 family)  
**Judge:** OpenAI GPT-5 via `OPENAI_API_KEY` (required for all agents)

Sister plan: Grok Build + grok-4.5 → #3

## Why this run

- Add a **Codex + GPT-5.6 Sol** data point next to existing Claude Code / DevRev Computer (and planned Grok) jobs.
- Validate Harbor’s built-in `codex` adapter with Enterprise-Bench MCP servers (crm / pm / file-server).
- Produce submit-ready results per [`docs/submit-results.md`](./docs/submit-results.md).

## Architecture

```text
Harbor Hub dataset (Enterprise-Bench/l1-l2-bench)
        ↓
Harbor job runner (Docker trials, scoring, jobs/)
        ↓  -a codex -m gpt-5.6-sol
OpenAI Codex CLI (npm @openai/codex in container)
        ↓  MCP (host.docker.internal:8011–8013)
Maple Payments synthetic data
        ↓  OPENAI_API_KEY
GPT-5 judge → reward 0.0 | 1.0
```

## Model choice (GPT-5.6 family)

GPT-5.6 is a family. Codex CLI model ids:

| Model | Role | Plan usage |
|---|---|---|
| **`gpt-5.6-sol`** | Flagship coding / agentic | **Primary eval** |
| `gpt-5.6-terra` | Balanced / lower cost | Optional cost ablation after Sol |
| `gpt-5.6-luna` | Fastest / cheapest | Optional efficiency ablation |
| `gpt-5.6` | Generic alias in some docs | Prefer explicit `gpt-5.6-sol` for reproducibility |

Harbor strips an optional provider prefix: `-m openai/gpt-5.6-sol` → `gpt-5.6-sol`.

Default Harbor Codex reasoning flag: `model_reasoning_effort=high`. Document any override (`--ak reasoning_effort=…`). Prefer `--ak web_search=disabled` for closed-book parity with the Grok plan’s default.

## Codex adapter facts (Harbor)

- Agent name: `codex` (available in Harbor **0.18+** — no special upgrade required, unlike `grok-build`).
- Install: `npm install -g @openai/codex@latest` inside trial (nvm/node 22 path on Debian images).
- Run: `codex exec --dangerously-bypass-approvals-and-sandbox --skip-git-repo-check --model <id> --json -- …`
- MCP servers from `--mcp-config` written to `$CODEX_HOME/config.toml` as `[mcp_servers.*]`.
- Trajectories: ATIF supported; sessions copied to `agent/sessions` then converted.

## Authentication

### Default (recommended for submit-ready runs): API key

```bash
export OPENAI_API_KEY=sk-...

# Same key is used for:
# 1) Codex agent (Harbor writes into container auth.json)
# 2) LLM judge (verifier)
```

Always forward into the agent sandbox:

```bash
--ae OPENAI_API_KEY=$OPENAI_API_KEY
```

### Optional: ChatGPT / Codex subscription via auth.json

Harbor Codex defaults to API key. Opt into host login:

```bash
export CODEX_FORCE_AUTH_JSON=1
# or
export CODEX_AUTH_JSON_PATH=$HOME/.codex/auth.json
```

Still pass `OPENAI_API_KEY` for the **judge** unless judge env is separately provisioned.

**Recommendation:** use API key for reproducible public submissions.

## Prerequisites

| Requirement | Notes |
|---|---|
| Harbor CLI | 0.18+ already has `codex` |
| Docker Desktop | ~2 GB RAM + 2 CPUs per trial; on ~8 GiB Docker use **`-n 3`** max |
| Local base image | `make build-image` → `enterprise-bench/conversational-base:latest` |
| MCP servers | `make start-servers` (REST 9001–9003 + MCP 8011–8013) |
| `OPENAI_API_KEY` | Agent (default path) + judge |

## Known gotchas

1. **`make install` before `make setup`** — setuptools flat-layout on `data/` / `images/`.
2. **Base image built locally** — not a registry pull problem.
3. **MCP 421** — add `allowed_hosts=["*"]` to each `mcp.run(...)`, restart servers; verify Host-header probe returns 200.
4. Ports **801x MCP / 900x REST** are both correct.
5. First trial slower: Codex npm install + cold model.
6. Missing `--ae OPENAI_API_KEY` → agent or judge 401.
7. Prefer explicit `gpt-5.6-sol` over bare `gpt-5.6` for reported results.
8. OOM at high concurrency → lower `-n`.

## Execution plan

### Phase 0 — Tooling

```bash
harbor --version
harbor run -h | rg -i codex
harbor auth status
```

### Phase 1 — Dataset + environment

```bash
# Preferred local tree (MCP fix already applied on nimit’s machine):
cd ~/Desktop/nitmit-dev/harbor-testing-benchmarks/enterprise-bench/l1-l2-bench

# Or fresh Hub download:
# harbor download enterprise-bench/l1-l2-bench -o ./enterprise-bench
# cd ./enterprise-bench/l1-l2-bench && make install && make setup
# apply MCP allowed_hosts fix if needed

make build-image
make start-servers
```

Host-header probe:

```bash
curl -s -o /dev/null -w "%{http_code}\n" -X POST http://localhost:8012/mcp \
  -H "Host: host.docker.internal:8012" \
  -H "Accept: application/json, text/event-stream" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"t","version":"1"}}}'
# expect 200
```

### Phase 2 — Smoke (1 task, 1 attempt)

```bash
export OPENAI_API_KEY=...

harbor run \
  -p eng-l1-a \
  -a codex \
  -m gpt-5.6-sol \
  --mcp-config mcp.json \
  --ae OPENAI_API_KEY=$OPENAI_API_KEY \
  --ak web_search=disabled \
  -k 1 -n 1 \
  --yes \
  --jobs-dir jobs/codex-gpt-5.6-sol-smoke
```

**Smoke pass criteria:** Codex installs, MCP tools appear in trajectory, judge writes `reward.txt`, no auth failure.

### Phase 3 — Reliability pass (compare to Claude baseline)

Prior local claude-code / claude-opus-4-8 full job (`2026-07-09__09-26-22`): **70 trials**, mean reward **~0.74**, pass@5 **~0.93**.

```bash
harbor run \
  -p . \
  -a codex \
  -m gpt-5.6-sol \
  --mcp-config mcp.json \
  --ae OPENAI_API_KEY=$OPENAI_API_KEY \
  --ak web_search=disabled \
  -k 5 -n 3 \
  --yes \
  --jobs-dir jobs/codex-gpt-5.6-sol-k5
```

Run long jobs under `tmux`. Prefer explicit `harbor run` over Makefile defaults (Makefile still centers on claude-code).

### Phase 4 — Optional methodology-grade + submit

```bash
harbor run -p . -a codex -m gpt-5.6-sol --mcp-config mcp.json \
  --ae OPENAI_API_KEY=$OPENAI_API_KEY \
  --ak web_search=disabled \
  -k 10 -n 3 --yes \
  --jobs-dir jobs/codex-gpt-5.6-sol-k10
```

Submission package (`docs/submit-results.md`):

- Agent: Codex + CLI version
- Model: `gpt-5.6-sol` (+ reasoning_effort, web_search setting)
- Harbor version
- Dataset ref: `enterprise-bench/l1-l2-bench`
- Full command, `-k`, `-n`
- OS, Docker memory/CPU
- Harbor Hub job link or archived `jobs/`
- Known failures / exclusions

### Phase 5 — Teardown

```bash
make stop-servers
```

## Optional ablations (after Sol baseline)

```bash
# cost / efficiency
-m gpt-5.6-terra
-m gpt-5.6-luna
```

Keep k/n and MCP config identical so comparisons are fair.

## Success metrics

- Mean reward / success rate
- pass@k (k=2,4,5 and/or 10)
- Per-task reliability (note historical hardness of eng-l1-a criterion 6)
- Infra errors vs genuine fails
- Rough wall-clock and OpenAI cost (agent + judge share the same provider)

## Checklist

- [ ] Harbor lists `codex`; Hub login OK
- [ ] `OPENAI_API_KEY` available for agent + judge
- [ ] Base image built; MCP healthy (Host-header 200)
- [ ] Smoke: eng-l1-a k=1 complete
- [ ] Reliability: 14 × k=5 complete
- [ ] Results summarized vs Claude baseline (+ Grok if available)
- [ ] Optional: k=10 + submit package
- [ ] Servers stopped; keys rotated if pasted

## Time / cost rough order

| Scenario | Trials | Notes |
|---|---:|---|
| Smoke eng-l1-a | 1 | Includes first Codex install in container |
| Reliability k=5 | 70 | Several hours at n=3 |
| Methodology k=10 | 140 | Overnight / long day |

Task agent timeout typically **600s**; verifier **300s**.

## Comparison (same protocol)

| | Claude Code | Grok Build (#3) | Codex (this issue) |
|---|---|---|---|
| `-a` | `claude-code` | `grok-build` (≥0.20) | `codex` (0.18+) |
| `-m` | `claude-opus-4-8` | `grok-4.5` | `gpt-5.6-sol` |
| Agent auth | Anthropic / Bedrock | `XAI_API_KEY` | `OPENAI_API_KEY` |
| Judge | OpenAI GPT-5 | OpenAI GPT-5 | OpenAI GPT-5 |

## References

- Dataset: https://hub.harborframework.com/datasets/enterprise-bench/l1-l2-bench
- Harbor: https://github.com/harbor-framework/harbor
- Codex models: https://developers.openai.com/codex/models
- GPT-5.6: https://openai.com/index/gpt-5-6/
- Methodology: https://devrev.ai/enterprise-bench-methodology
- This repo: `README.md`, `SKILL.md`, `docs/submit-results.md`

## Artifacts (local, for internal share)

- HTML plan: `enterprise-bench-codex-execution-plan.html`
- PDF plan: `enterprise-bench-codex-execution-plan.pdf` (generated for team share)

## Owner / next step

Execute Phase 0–2 when `OPENAI_API_KEY` is available in the runner shell, then Phase 3 for a submit-ready Codex / GPT-5.6 Sol data point.

	Claude Code	Grok Build (#3)	Codex (this issue)
`-a`	`claude-code`	`grok-build` (≥0.20)	`codex` (0.18+)
`-m`	`claude-opus-4-8`	`grok-4.5`	`gpt-5.6-sol`
Agent auth	Anthropic / Bedrock	`XAI_API_KEY`	`OPENAI_API_KEY`
Judge	OpenAI GPT-5	OpenAI GPT-5	OpenAI GPT-5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Run Enterprise-Bench L1–L2 with Codex + GPT-5.6 Sol (Harbor harness) #4

Summary

Why this run

Architecture

Model choice (GPT-5.6 family)

Codex adapter facts (Harbor)

Authentication

Default (recommended for submit-ready runs): API key

Optional: ChatGPT / Codex subscription via auth.json

Prerequisites

Known gotchas

Execution plan

Phase 0 — Tooling

Phase 1 — Dataset + environment

Phase 2 — Smoke (1 task, 1 attempt)

Phase 3 — Reliability pass (compare to Claude baseline)

Phase 4 — Optional methodology-grade + submit

Phase 5 — Teardown

Optional ablations (after Sol baseline)

Success metrics

Checklist

Time / cost rough order

Comparison (same protocol)

References

Artifacts (local, for internal share)

Owner / next step

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Model	Role	Plan usage
`gpt-5.6-sol`	Flagship coding / agentic	Primary eval
`gpt-5.6-terra`	Balanced / lower cost	Optional cost ablation after Sol
`gpt-5.6-luna`	Fastest / cheapest	Optional efficiency ablation
`gpt-5.6`	Generic alias in some docs	Prefer explicit `gpt-5.6-sol` for reproducibility

Requirement	Notes
Harbor CLI	0.18+ already has `codex`
Docker Desktop	~2 GB RAM + 2 CPUs per trial; on ~8 GiB Docker use `-n 3` max
Local base image	`make build-image` → `enterprise-bench/conversational-base:latest`
MCP servers	`make start-servers` (REST 9001–9003 + MCP 8011–8013)
`OPENAI_API_KEY`	Agent (default path) + judge

Scenario	Trials	Notes
Smoke eng-l1-a	1	Includes first Codex install in container
Reliability k=5	70	Several hours at n=3
Methodology k=10	140	Overnight / long day

Uh oh!

Run Enterprise-Bench L1–L2 with Codex + GPT-5.6 Sol (Harbor harness) #4

Description

Summary

Why this run

Architecture

Model choice (GPT-5.6 family)

Codex adapter facts (Harbor)

Authentication

Default (recommended for submit-ready runs): API key

Optional: ChatGPT / Codex subscription via auth.json

Prerequisites

Known gotchas

Execution plan

Phase 0 — Tooling

Phase 1 — Dataset + environment

Phase 2 — Smoke (1 task, 1 attempt)

Phase 3 — Reliability pass (compare to Claude baseline)

Phase 4 — Optional methodology-grade + submit

Phase 5 — Teardown

Optional ablations (after Sol baseline)

Success metrics

Checklist

Time / cost rough order

Comparison (same protocol)

References

Artifacts (local, for internal share)

Owner / next step

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions