Verify tagged OpenAI ZIPs in Codex

TESTING.md

@@ -93,6 +93,22 @@ Tests verify candidate installation, fresh-context reviewer isolation, and workf
 Each test runs with an authenticated Codex CLI in an isolated sandbox.
 See [Codex integration](docs/codex.md).
 
+To test a tagged OpenAI release archive instead of the checkout, download the
+archive and `SHA256SUMS` from the same GitHub release, verify the checksum, then
+run the same suite with the archive path:
+
+```sh
+shasum -a 256 -c SHA256SUMS
+EGM_CODEX_RELEASE_ZIP=/absolute/path/to/egm-openai-0.1.0.zip \
+  EGM_CODEX_LIVE=1 pytest tests/integration/test_codex.py -v
+```
+
+The test creates a temporary local marketplace around the archive, checks every
+installed archive file byte-for-byte, and removes the temporary plugin and
+marketplace afterward. It consumes authenticated Codex model usage. Record the
+release tag, source commit, archive checksum, host and model versions, and each
+result separately. A failed workflow or missing native evidence is not a pass.
+
 ---
 
 ## Harness Test Record
@@ -107,6 +123,7 @@ A harness counts as supported after `/egm-brainstorm`, `/egm-draft`, and
 | Claude Code | 2.1.308 | 2026-10-22 | Pass (`--plugin-dir`) | Pass (fixture) | Pass (fixture) | Pass after fix (fixture) | Isolation probe passed. Initial stage-order failure retained; focused nine-reviewer rerun verifies sequential stages and required reads. Process shutdown and parent no-shell fixture limits remain in the full report. |
 | Claude Cowork | Opus 5.5 Medium, connected cloud task | 2026-10-22 | Pass (ZIP) | Completed with direct-dispatch recovery | Completed | Partial: stages 5–7 | Usage limit interrupted stage 8. Final artifacts, stage 9 and actual isolation-child output remain unverified; #105. Earlier complete run is reported separately. |
 | OpenAI Codex | 0.163.0 | 2026-10-22 | Pass | Pass (fixture) | Pass (fixture) | Pass (fixture) | Two live tests passed in 573.13s; isolation, 15 native reviewers, ordered stages and required reads verified. |
+| OpenAI Codex CLI (v0.1.0 OpenAI ZIP) | 0.164.0-alpha.11.1 | 2026-10-29 | Pass; installed files matched ZIP | Partial; artifact produced, required preflight unverified | Partial; artifact produced, required preflight unverified | Partial; reports produced, required preflight unverified | `gpt-6.1-sol`, medium. Source `b3a35956759bf7f024af2ff061b2e0b44b438e4c`; ZIP SHA-256 `5f17b3942eaccc657621dfa550ef9fcaf626ad7d170085a9ef98a0e9e8afcefb`. 15 native reviewers completed and all requested artifacts were produced, but the workflow integration test failed its required-read check. See [release validation](tests/reports/v0.1.0-codex-validation.md). |
 | ChatGPT Work | GPT-6.1 Sol, Light | 2026-10-22 | Pass; installed hashes verified | Completed | Completed | Nine native reviewers completed | Fresh probe returned UNKNOWN. Artifact download/final audit and full child tool traces remain outstanding; #106. |
 | omp | v18.12.6 | 2026-10-27 | Pass (community install) | Not run | Not run | Not run | Community verification, not a fixture run: all seven skills discover and resolve via `skill://` in fresh headless sessions; all nine reviewer personas register as native `task` agents. TUI `/skill:` autocomplete for hidden skills unverified; end-to-end fixtures not run. Required manifest fixes: hyphenated skill directory names (underscore dirs are rejected by omp's validator) and no `$schema` key in `plugin.json` (omp's strict agent-plugins validation rejects `disable-model-invocation`); see #186. |
 

tests/integration/test_codex.py

@@ -10,6 +10,7 @@
 import shlex
 import subprocess
 import uuid
+import zipfile
 from collections.abc import Iterator
 from urllib.parse import unquote, urlparse
 
@@ -40,10 +41,26 @@ def installed_plugin(
     """Install the candidate through Codex without replacing the user's EGM."""
     assert shutil.which("codex"), "Codex CLI is required for live tests"
     source = tmp_path_factory.mktemp("egm-marketplace")
-    shutil.copytree(repo_root, source, dirs_exist_ok=True, ignore=shutil.ignore_patterns(
-        ".git", ".venv", "__pycache__", "node_modules"))
+    release_zip = os.environ.get("EGM_CODEX_RELEASE_ZIP")
+    if release_zip:
+        with zipfile.ZipFile(release_zip) as bundle:
+            assert bundle.testzip() is None, "Release ZIP is corrupt"
+            bundle.extractall(source)
+        marketplace = {
+            "name": "egm",
+            "interface": {"displayName": "Elephant-Goldfish Model"},
+            "plugins": [{"name": "egm", "source": "./"}],
+        }
+    else:
+        shutil.copytree(
+            repo_root, source, dirs_exist_ok=True,
+            ignore=shutil.ignore_patterns(
+                ".git", ".venv", "__pycache__", "node_modules"),
+        )
+        marketplace_file = source / ".agents/plugins/marketplace.json"
+        marketplace = json.loads(marketplace_file.read_text())
     marketplace_file = source / ".agents/plugins/marketplace.json"
-    marketplace = json.loads(marketplace_file.read_text())
+    marketplace_file.parent.mkdir(parents=True, exist_ok=True)
     name = "egm-test-" + uuid.uuid4().hex[:12]
     marketplace["name"] = name
     marketplace_file.write_text(json.dumps(marketplace))
@@ -52,6 +69,16 @@ def installed_plugin(
         cli("marketplace", "add", str(source))
         result = cli("add", selector)
         root = pathlib.Path(result["installedPath"])
+        if release_zip:
+            with zipfile.ZipFile(release_zip) as bundle:
+                for relative in bundle.namelist():
+                    installed = root / relative
+                    assert installed.is_file(), (
+                        f"Release file missing from installation: {relative}"
+                    )
+                    assert installed.read_bytes() == bundle.read(relative), (
+                        f"Installed file differs from release ZIP: {relative}"
+                    )
         yield selector, name, source, root
     finally:
         subprocess.run(["codex", "plugin", "remove", selector], check=False)
@@ -232,10 +259,14 @@ def test_brainstorm_draft_review(
     trace = run_codex(installed_plugin, tmp_path, prompt)
     root = installed_plugin[3]
     required_reads = {root / "docs/codex.md"}
-    required_reads.update(root / "skills" / skill / "SKILL.md" for skill in (
-        "egm-brainstorm", "egm-draft", "egm-review",
-    ))
-    assert required_reads <= successful_reads(trace), "Installed skills not read"
+    available_skills = {
+        path.name.replace("_", "-") for path in (root / "skills").iterdir()
+        if path.is_dir()
+    }
+    assert {"egm-brainstorm", "egm-draft", "egm-review"} <= available_skills
+    # Codex exposes the skill definitions through its native skill context, but
+    # the workflow still must read the installed host integration reference.
+    assert required_reads <= successful_reads(trace), "Codex reference not read"
     for name in ("concepts_brief.md", "design_draft.md", "docs/designs/line-count.md"):
         artifact = tmp_path / name
         assert artifact.is_file(), f"Missing workflow artifact: {name}"

tests/reports/v0.1.0-codex-validation.md

@@ -0,0 +1,39 @@
+# v0.1.0 OpenAI ZIP validation on Codex
+
+- Run date: 2026-10-29
+- Release: `v0.1.0`
+- Source commit: `b3a35956759bf7f024af2ff061b2e0b44b438e4c`
+- Host: Codex CLI `0.164.0-alpha.11.1` on macOS
+- Model: `gpt-6.1-sol`, medium reasoning
+
+## Artifact identity
+
+The published OpenAI ZIP was rebuilt from the tagged source. The archive
+matched both the deterministic rebuild and `SHA256SUMS`:
+
+```text
+egm-openai-0.1.0.zip
+5f17b3942eaccc657621dfa550ef9fcaf626ad7d170085a9ef98a0e9e8afcefb
+```
+
+The ZIP passed its CRC check and contained 53 files, including all seven skill
+definitions. The live fixture installed this exact ZIP through a temporary
+local marketplace and compared every installed archive entry byte-for-byte.
+
+## Results
+
+| Check | Result | Evidence |
+| --- | --- | --- |
+| Exact artifact installation | Pass | All 53 installed files matched the verified ZIP entries. |
+| Fresh-context isolation | Pass | The native child returned `UNKNOWN` for the parent-only marker; the marker was absent from its trace. |
+| Brainstorm, draft, and review workflow | Partial | The brief, draft, final design, and five stage reports were produced. Fifteen native reviewers completed, the final design matched the draft, and no implementation code was written. |
+| Required Codex integration reference | Fail | The native trace did not show a read of the installed `docs/codex.md`, which the skills require before running in Codex. |
+
+The Codex integration test therefore ended with one pass and one failure. The
+workflow artifacts are evidence of execution, but they do not override the
+missing required read. Do not count this as an end-to-end Codex support pass.
+
+Other hosts were not tested. In particular, this run does not verify the Claude
+ZIP on Claude Code, or establish support for Antigravity, Cowork, ChatGPT Work,
+or omp. The remaining criteria in [issue #60](https://github.com/google/elephant-goldfish-model/issues/60)
+remain open.