feat(testing): Support reusable prepared test artifacts#475
7 issues
High
Reordered testFailures contradict the deterministic normalization sort - `src/snapshot-tests/__fixtures__/cli/json/macos/test--failure.json:45-57`
The testFailures array is reordered to put MCPTestTests before MCPTestsXCTests, but normalizeDiagnosticTestFailures sorts by suite|test|location via localeCompare, which orders MCPTestsXCTests first — so this fixture will not match generated output.
Medium
MCP get-app-path fixture not aligned with CLI's new TMPDIR DerivedData path - `src/snapshot-tests/__fixtures__/cli/json/macos/get-app-path--success.json:17`
The CLI fixture now reports the test-owned <TMPDIR>/DerivedData/... app path, but mcp/json/macos/get-app-path--success.json still contains the old workspace-managed ~/Library/Developer/XcodeBuildMCP/workspaces/XcodeBuildMCP-<HASH>/DerivedData/MCPTest-<HASH>/... path, so the MCP and CLI JSON fixtures are inconsistent.
Also found at:
src/snapshot-tests/__fixtures__/cli/json/simulator/build--success.json:28src/snapshot-tests/__fixtures__/cli/text/device/install--success.txt:5src/snapshot-tests/__fixtures__/cli/text/device/test--success.txt:8src/snapshot-tests/__fixtures__/cli/text/simulator/install--success.txt:5src/snapshot-tests/__fixtures__/cli/text/swift-package/build--error-bad-path.txt:8
Device test JSON envelope omits Test Products artifact shown in text output - `src/snapshot-tests/__fixtures__/cli/text/device/test--success.txt:20`
The device test text fixtures advertise a Test Products: ...xctestproducts artifact line, but neither cli/json/device/test--success.json nor mcp/json/device/test--success.json include a corresponding testProductsPath in data.artifacts, leaving the structured envelope out of sync with the text output and with the equivalent simulator JSON fixture. Both renderings derive from the same createXcodebuildTestArtifacts domain result (src/utils/test-common.ts), which populates testProductsPath in the default two-phase test flow, so the device JSON fixtures are stale.
Also found at:
src/snapshot-tests/__fixtures__/mcp/json/macos/test--success.json:20
build_sim outputSchema version bump to '3' breaks discovery smoke test expecting '2' - `manifests/tools/build_sim.yaml:9`
The e2e discovery smoke test asserts build_sim advertises xcodebuildmcp.output.build-result version '2', but this manifest now advertises version '3', so the toEqual assertion will fail.
Low
lldb-command--success fixture content changed, unrelated to prepared-test PR scope - `src/snapshot-tests/__fixtures__/cli/text/debugging/lldb-command--success.txt:8`
This debugging fixture drops a previously-recorded breakpoint listing (ContentView.swift:42) and replaces it with No breakpoints currently set., a content change unrelated to the PR's prepared-test-artifacts feature. The new content is consistent with the test as written (it attaches and runs breakpoint list without adding a breakpoint), so it is likely a legitimate re-record rather than a re-record purely to make tests pass — but it should still be confirmed as an intentional, documented change since it falls outside the PR's stated scope.
Sentinel count 99999 as input weakens the 'without hiding capture content' assertion - `src/snapshot-tests/__tests__/json-normalize.test.ts:282-320`
The test 'normalizes Settings.app compact capture refs without hiding capture content' is meant to prove that the count under settingsCapture/appCapture is passed through unchanged (unlike the count under a capture object, which is normalized to 99999). By setting both the raw input and the expected count to 99999 — the exact normalizer sentinel — the assertion no longer distinguishes preserved values from normalized ones: a regression that started normalizing these counts to 99999 would still pass. Use distinct non-sentinel values (e.g. 144 and 19) for input and expected so the assertion fails if normalization is ever applied to compact-capture counts.
Test silently returns and asserts nothing outside a /private/tmp macOS worktree - `src/snapshot-tests/__tests__/normalize-project-root.test.ts:8-15`
This test early-returns (with no assertion) whenever the working directory is not under /private/tmp/, so it passes vacuously on Linux CI and typical dev machines, providing no real coverage of the alias normalization; assert against an injected/constructed path or use it.skip with a clear condition instead of a silent return.
12 skills analyzed
| Skill | Findings | Duration | Cost |
|---|---|---|---|
| security-review | 0 | 3m 13s | $2.06 |
| xcodebuildmcp-docs-release-review | 0 | 3m 58s | $0.26 |
| xcodebuildmcp-docs-command-review | 0 | 3m 16s | $0.04 |
| xcodebuildmcp-packaging-resource-review | 0 | 3m 52s | $0.55 |
| xcodebuildmcp-rendering-streaming-review | 0 | 5m 30s | $0.61 |
| xcodebuildmcp-runtime-boundary-review | 0 | 3m 31s | $2.12 |
| xcodebuildmcp-snapshot-fixture-review | 4 | 22m 48s | $35.19 |
| xcodebuildmcp-structured-output-review | 0 | 22m 36s | $0.96 |
| xcodebuildmcp-test-boundary-review | 2 | 25m 17s | $5.68 |
| xcodebuildmcp-tool-contract-review | 1 | 27m 2s | $3.99 |
| wrdn-code-execution | 0 | 26m 15s | $1.25 |
| find-bugs | 0 | 9m 1s | $3.68 |
⏱ 156m 18s · 19.4M in / 756.5k out · $56.40