Verify tagged OpenAI ZIPs in Codex

TESTING.md

@@ -93,6 +93,22 @@ Tests verify candidate installation, fresh-context reviewer isolation, and workf
 Each test runs with an authenticated Codex CLI in an isolated sandbox.
 See [Codex integration](docs/codex.md).
 
+To test a tagged OpenAI release archive instead of the checkout, download the
+archive and `SHA256SUMS` from the same GitHub release, verify the checksum, then
+run the same suite with the archive path:
+
+```sh
+shasum -a 256 -c --ignore-missing SHA256SUMS
+EGM_CODEX_RELEASE_ZIP=/absolute/path/to/egm-openai-0.1.0.zip \
+  EGM_CODEX_LIVE=1 pytest tests/integration/test_codex.py -v
+```
+
+The test creates a temporary local marketplace around the archive, checks every
+installed archive file byte-for-byte, and removes the temporary plugin and
+marketplace afterward. It consumes authenticated Codex model usage. Record the
+release tag, source commit, archive checksum, host and model versions, and each
+result separately. A failed workflow or missing native evidence is not a pass.
+
 ---
 
 ## Harness Test Record
@@ -107,6 +123,7 @@ A harness counts as supported after `/egm-brainstorm`, `/egm-draft`, and
 | Claude Code | 2.1.308 | 2026-10-22 | Pass (`--plugin-dir`) | Pass (fixture) | Pass (fixture) | Pass after fix (fixture) | Isolation probe passed. Initial stage-order failure retained; focused nine-reviewer rerun verifies sequential stages and required reads. Process shutdown and parent no-shell fixture limits remain in the full report. |
 | Claude Cowork | Opus 5.5 Medium, connected cloud task | 2026-10-22 | Pass (ZIP) | Completed with direct-dispatch recovery | Completed | Partial: stages 5–7 | Usage limit interrupted stage 8. Final artifacts, stage 9 and actual isolation-child output remain unverified; #105. Earlier complete run is reported separately. |
 | OpenAI Codex | 0.163.0 | 2026-10-22 | Pass | Pass (fixture) | Pass (fixture) | Pass (fixture) | Two live tests passed in 573.13s; isolation, 15 native reviewers, ordered stages and required reads verified. |
+| OpenAI Codex CLI (v0.1.0 OpenAI ZIP) | 0.164.0-alpha.11.1 | 2026-10-29 | Pass; installed files matched ZIP | Pass (fixture) | Pass (fixture) | Pass (fixture) | `gpt-6.1-sol`, medium. Full opt-in live suite passed in 482.30s at PR commit `8bd11c5`; source `b3a35956759bf7f024af2ff061b2e0b44b438e4c`; ZIP SHA-256 `5f17b3942eaccc657621dfa550ef9fcaf626ad7d170085a9ef98a0e9e8afcefb`. The path-free workflow read all three installed `SKILL.md` files and returned the contents of `docs/codex.md`; it produced all requested artifacts and completed 15 native reviewers. Seven offline regression cases cover successful/failed reads, literal compound `cat`, and conservative rejection of `&&` and pipe commands. See the [full run report in PR #217](https://github.com/google/elephant-goldfish-model/pull/217). |
 | ChatGPT Work | GPT-6.1 Sol, Light | 2026-10-22 | Pass; installed hashes verified | Completed | Completed | Nine native reviewers completed | Fresh probe returned UNKNOWN. Artifact download/final audit and full child tool traces remain outstanding; #106. |
 | omp | v18.12.6 | 2026-10-27 | Pass (community install) | Not run | Not run | Not run | Community verification, not a fixture run: all seven skills discover and resolve via `skill://` in fresh headless sessions; all nine reviewer personas register as native `task` agents. TUI `/skill:` autocomplete for hidden skills unverified; end-to-end fixtures not run. Required manifest fixes: hyphenated skill directory names (underscore dirs are rejected by omp's validator) and no `$schema` key in `plugin.json` (omp's strict agent-plugins validation rejects `disable-model-invocation`); see #186. |
 

tests/integration/test_codex.py

@@ -10,6 +10,7 @@
 import shlex
 import subprocess
 import uuid
+import zipfile
 from collections.abc import Iterator
 from urllib.parse import unquote, urlparse
 
@@ -40,10 +41,23 @@ def installed_plugin(
     """Install the candidate through Codex without replacing the user's EGM."""
     assert shutil.which("codex"), "Codex CLI is required for live tests"
     source = tmp_path_factory.mktemp("egm-marketplace")
-    shutil.copytree(repo_root, source, dirs_exist_ok=True, ignore=shutil.ignore_patterns(
-        ".git", ".venv", "__pycache__", "node_modules"))
+    release_zip = os.environ.get("EGM_CODEX_RELEASE_ZIP")
+    if release_zip:
+        with zipfile.ZipFile(release_zip) as bundle:
+            assert bundle.testzip() is None, "Release ZIP is corrupt"
+            bundle.extractall(source)
+    else:
+        shutil.copytree(
+            repo_root, source, dirs_exist_ok=True,
+            ignore=shutil.ignore_patterns(
+                ".git", ".venv", "__pycache__", "node_modules"),
+        )
+    # The OpenAI ZIP omits the marketplace file; use the checked-in metadata for both.
+    marketplace = json.loads(
+        (repo_root / ".agents/plugins/marketplace.json").read_text()
+    )
     marketplace_file = source / ".agents/plugins/marketplace.json"
-    marketplace = json.loads(marketplace_file.read_text())
+    marketplace_file.parent.mkdir(parents=True, exist_ok=True)
     name = "egm-test-" + uuid.uuid4().hex[:12]
     marketplace["name"] = name
     marketplace_file.write_text(json.dumps(marketplace))
@@ -52,6 +66,16 @@ def installed_plugin(
         cli("marketplace", "add", str(source))
         result = cli("add", selector)
         root = pathlib.Path(result["installedPath"])
+        if release_zip:
+            with zipfile.ZipFile(release_zip) as bundle:
+                for relative in bundle.namelist():
+                    installed = root / relative
+                    assert installed.is_file(), (
+                        f"Release file missing from installation: {relative}"
+                    )
+                    assert installed.read_bytes() == bundle.read(relative), (
+                        f"Installed file differs from release ZIP: {relative}"
+                    )
         yield selector, name, source, root
     finally:
         subprocess.run(["codex", "plugin", "remove", selector], check=False)
@@ -110,26 +134,58 @@ def successful_reads(trace: list[dict]) -> set[pathlib.Path]:
     """Inspect executed file reads, not filenames mentioned by the model."""
     paths = set()
     for event in trace:
-        item = event.get("payload", {}).get("item", {})
+        if (event.get("type") != "event_msg"
+                or event.get("payload", {}).get("type") != "item_completed"):
+            continue
+        item = event["payload"].get("item", {})
         if (item.get("type") != "CommandExecution"
-                or item.get("status") != "completed"
-                or item.get("exit_code") != 0
-                or not (item.get("aggregated_output") or item.get("stdout"))):
+                or item.get("status") not in {"completed", "failed"}
+                or not item.get("aggregated_output")):
+            continue
+        output = item["aggregated_output"]
+        exit_code = item.get("exit_code")
+        # A compound `cat` can return useful file contents and then fail on a
+        # later missing operand. Count a file only when its own contents are
+        # visibly present in the captured output.
+        cwd_value = item.get("cwd")
+        if not cwd_value:
             continue
-        cwd = pathlib.Path(unquote(urlparse(item["cwd"]).path))
+        cwd = pathlib.Path(unquote(urlparse(cwd_value).path))
         for command in item.get("parsed_cmd", []):
             if command.get("type") == "read" and command.get("path"):
-                paths.add((cwd / command["path"]).resolve())
+                if exit_code == 0:
+                    paths.add((cwd / command["path"]).resolve())
             elif command.get("type") == "unknown":
-                # Codex labels a multi-file cat as unknown. Accept only literal
-                # absolute operands, never shell expansion or composed commands.
+                # Codex labels compound shell commands as unknown. Inspect
+                # only standalone cat segments with literal absolute paths.
                 raw = command.get("cmd", "")
-                if any(char in raw for char in "$`|;&<>*?{}()\n"):
+                try:
+                    lexer = shlex.shlex(raw, posix=True, punctuation_chars=";")
+                    lexer.whitespace_split = True
+                    tokens = list(lexer)
+                except ValueError:
                     continue
-                args = shlex.split(raw)
-                if (len(args) > 1 and args[0] == "cat"
-                        and all(pathlib.Path(arg).is_absolute() for arg in args[1:])):
-                    paths.update(pathlib.Path(arg).resolve() for arg in args[1:])
+                segments: list[list[str]] = [[]]
+                for token in tokens:
+                    if token == ";":
+                        segments.append([])
+                    else:
+                        segments[-1].append(token)
+                for args in segments:
+                    if (len(args) <= 1 or args[0] != "cat"
+                            or any(char in arg for arg in args
+                                   for char in "$`|&<>*?{}()")
+                            or not all(pathlib.Path(arg).is_absolute() for arg in args[1:])):
+                        continue
+                    for arg in args[1:]:
+                        path = pathlib.Path(arg).resolve()
+                        try:
+                            contents = path.read_text(encoding="utf-8")
+                        except (OSError, UnicodeError):
+                            continue
+                        evidence = contents[:200]
+                        if evidence and evidence in output:
+                            paths.add(path)
     return paths
 
 
@@ -179,16 +235,15 @@ def test_fresh_context_isolation(
     marker = "parent-only-" + uuid.uuid4().hex
     prompt = f"""Native EGM isolation test. Parent-only marker: {marker}
 Do not write or repeat this marker, or pass it to another agent.
-Read the installed egm plugin's Codex integration reference. Use a native
-general-purpose subagent with no inherited turns, using the parameter in its
-actual schema (fork_context: false or fork_turns: "none"). Its exact task is:
+Use a native general-purpose subagent with no inherited turns, using the
+parameter in its actual schema (fork_context: false or fork_turns: "none").
+Its exact task is:
 'Do not use tools. If a parent-only marker is visible in your conversation,
 return it verbatim. Otherwise return exactly UNKNOWN.'
 Wait for its result. Do not simulate its response. Do not use memory services,
 external services, or conversation history. Report the actual child response.
 """
     trace = run_codex(installed_plugin, tmp_path, prompt)
-    assert installed_plugin[3] / "docs/codex.md" in successful_reads(trace)
     assert_isolated_spawns(trace)
     children = child_traces(trace)
     assert len(children) == 1
@@ -231,11 +286,20 @@ def test_brainstorm_draft_review(
 """
     trace = run_codex(installed_plugin, tmp_path, prompt)
     root = installed_plugin[3]
-    required_reads = {root / "docs/codex.md"}
-    required_reads.update(root / "skills" / skill / "SKILL.md" for skill in (
-        "egm-brainstorm", "egm-draft", "egm-review",
-    ))
-    assert required_reads <= successful_reads(trace), "Installed skills not read"
+    # v0.1.0 uses underscores; current source skill names use hyphens.
+    skill_dirs = {
+        path.name.replace("_", "-"): path
+        for path in (root / "skills").iterdir()
+        if path.is_dir()
+    }
+    assert {"egm-brainstorm", "egm-draft", "egm-review"} <= skill_dirs.keys()
+    required_skill_reads = {
+        skill_dirs[skill] / "SKILL.md"
+        for skill in ("egm-brainstorm", "egm-draft", "egm-review")
+    }
+    reads = successful_reads(trace)
+    assert required_skill_reads <= reads, "Installed skills not read"
+    assert root / "docs/codex.md" in reads, "Codex reference not read"
     for name in ("concepts_brief.md", "design_draft.md", "docs/designs/line-count.md"):
         artifact = tmp_path / name
         assert artifact.is_file(), f"Missing workflow artifact: {name}"

tests/unit/test_codex_traces.py

@@ -0,0 +1,103 @@
+# Copyright 2026 Google LLC.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Offline regression tests for Codex command-read trace evidence."""
+
+from pathlib import Path
+import runpy
+import shlex
+
+import pytest
+
+ROOT = Path(__file__).resolve().parents[2]
+
+
+@pytest.mark.parametrize(
+    ("parsed_command", "exit_code", "output", "expected"),
+    [
+        (
+            lambda path: {"type": "unknown", "cmd": f"cat {path} /missing"},
+            1,
+            "# Installed guide\n" + "guide content " * 20,
+            True,
+        ),
+        (
+            lambda path: {
+                "type": "unknown",
+                "cmd": f"pwd; rg --files -g 'README*' .; cat {path}",
+            },
+            0,
+            "# Installed guide\n" + "guide content " * 20,
+            True,
+        ),
+        (
+            lambda path: {
+                "type": "unknown",
+                "cmd": f"cd {Path(path).parent} && cat {path}",
+            },
+            0,
+            "# Installed guide\n" + "guide content " * 20,
+            False,
+        ),
+        (
+            lambda path: {
+                "type": "unknown",
+                "cmd": f"cat {path} | head",
+            },
+            0,
+            "# Installed guide\n" + "guide content " * 20,
+            False,
+        ),
+        (
+            lambda path: {"type": "unknown", "cmd": f"cat {path} /missing"},
+            1,
+            "cat: /missing: No such file or directory\n",
+            False,
+        ),
+        (
+            lambda path: {"type": "read", "path": str(path)},
+            1,
+            "# Installed guide\n" + "guide content " * 20,
+            False,
+        ),
+        (
+            lambda path: {"type": "read", "path": str(path)},
+            0,
+            "# Installed guide\n" + "guide content " * 20,
+            True,
+        ),
+    ],
+    ids=(
+        "failed-cat-printed-file", "compound-cat-printed-file",
+        "and-separated-cat-rejected", "piped-cat-rejected",
+        "failed-cat-no-file-output", "failed-read", "successful-read",
+    ),
+)
+def test_successful_reads_requires_native_trace_evidence(
+    tmp_path: Path, parsed_command, exit_code: int, output: str,
+    expected: bool,
+) -> None:
+    successful_reads = runpy.run_path(
+        str(ROOT / "tests/integration/test_codex.py")
+    )["successful_reads"]
+    guide = tmp_path / "docs" / "codex.md"
+    guide.parent.mkdir()
+    guide.write_text("# Installed guide\n" + "guide content " * 20)
+    command = parsed_command(shlex.quote(str(guide)))
+    trace = [{
+        "type": "event_msg",
+        "payload": {
+            "type": "item_completed",
+            "item": {
+                "type": "CommandExecution",
+                "status": "failed" if exit_code else "completed",
+                "exit_code": exit_code,
+                "cwd": tmp_path.as_uri(),
+                "parsed_cmd": [command],
+                "aggregated_output": output,
+            },
+        },
+    }]
+
+    assert (guide.resolve() in successful_reads(trace)) is expected
+