feat(testing): Support reusable prepared test artifacts by cameroncooke · Pull Request #475 · getsentry/XcodeBuildMCP

cameroncooke · 2026-07-16T15:55:39Z

Why

Agents and CI workflows need to prepare tests once and execute them later without falling back to raw xcodebuild. The existing build tools always finished with a normal build action, so they could not expose the reusable products produced by build-for-testing.

What Changed

Extend the existing simulator, device, and macOS build tools with buildForTesting and optional testProductsPath inputs.
Return managed .xctestproducts bundles and discovered .xctestrun files as structured build artifacts.
Let test tools execute either prepared .xctestproducts bundles or advanced .xctestrun inputs without rebuilding, while returning the new .xcresult produced by the run.
Move conditional next-step selection into tool manifests so normal builds and prepared-test builds advertise the appropriate follow-up without changing existing output behavior.
Add unit, schema, lifecycle, and MCP/CLI snapshot coverage for successful and failing prepared-test workflows.

Fixes #450

Extend existing build and test tools to prepare, return, and consume reusable test artifacts while preserving normal build behavior. Fixes #450

Move conditional next-step selection into tool manifests and preserve the behavior across MCP, CLI, daemon, and snapshot outputs. Refs #450

pkg-pr-new · 2026-07-16T15:57:34Z

Open in StackBlitz

npm i https://pkg.pr.new/xcodebuildmcp@475

commit: 191b74e

Keep source-only xcodebuild arguments out of the generated test-without-building phase and sort final test failures independently of emission order.

Share per-test environment setup and cleanup across snapshot suites, cache suite-level simulator builds, and normalize volatile output so fixtures remain stable. Update non-environmental fixtures and add coverage for cleanup and normalization behavior.

Prevent session defaults and selectors from leaking into prepared test runs, remove a useless simulator state initializer, and align volatile snapshot fixtures across CLI and MCP.

Keep only-testing and skip-testing arguments in the prepared test phase while retaining session-default conflict filtering.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit cda637e. Configure here.}

sentry-warden · 2026-07-18T08:44:56Z

      "screenHash": "<SCREEN_HASH>",
      "seq": 1,
-      "count": 28,
+      "count": 99999,


tap--success fixture count changed to implausible 99999 unrelated to PR scope

The count field was changed from 28 to 99999 in a UI-automation tap fixture, which is unrelated to the prepared-test-artifacts feature and does not reflect a plausible real output.

Evidence

PR scope is buildForTesting/testProductsPath build and test tools; this fixture is ui-automation/tap--success.json.

count changed from 28 to 99999 while the targets array still lists only 19 entries, so 99999 does not correspond to observed data.

The identical value appears in both the CLI and MCP fixtures, indicating a hand-patched constant rather than a captured snapshot value.

If count is volatile it should be normalized in code like screenHash → <SCREEN_HASH>, not hard-coded to an arbitrary literal.

Also found at 2 additional locations

src/snapshot-tests/__fixtures__/cli/json/ui-automation/tap--success-verbose.json:611-617

src/snapshot-tests/__fixtures__/cli/json/ui-automation/tap--success.json:48

_{Identified by Warden xcodebuildmcp-snapshot-fixture-review · G2H-E5J}

sentry-warden · 2026-07-18T08:44:56Z

  "error": null,
  "data": {
    "simulators": [
+      {


Simulator-list fixture expanded with environment-specific entries unrelated to prepared-test feature

This fixture is regrown with many machine-specific simulators (watchOS Series 11, iPhone 17 family, etc.) that don't relate to the PR's prepared-test-artifacts change; simulator names and the device count are not normalized and reflect the installed Xcode on one machine, so this looks like a regeneration to match a local environment rather than an intentional behavior change.

Evidence

json-normalize.ts only normalizes state (line 83), runtime version (lines 95-97), and UUIDs; simulator name and the number of simulators entries are emitted verbatim.

The added entries (Apple Watch Series 11, iPhone 17 Pro, iPad Pro (M5), etc.) are literal device names tied to a specific installed simulator set, making them volatile across machines.

The PR scope is reusable prepared test artifacts (buildForTesting/testProductsPath), which has no bearing on the output of the simulator-list tool.

_{Identified by Warden xcodebuildmcp-snapshot-fixture-review · UXY-GFK}

sentry-warden · 2026-07-18T08:44:56Z

+        "Fetching https://github.com/apple/swift-argument-parser.git from cache",
+        "Fetched https://github.com/apple/swift-argument-parser.git from cache (<DURATION>)",
+        "Computing version for https://github.com/apple/swift-argument-parser.git",
+        "Computed https://github.com/apple/swift-argument-parser.git at 1.5.1 (<DURATION>)",
+        "Computed https://github.com/apple/swift-argument-parser.git at 1.5.1 (<DURATION>)",
+        "Creating working copy for https://github.com/apple/swift-argument-parser.git",
+        "Working copy of https://github.com/apple/swift-argument-parser.git resolved at 1.5.1",


CLI JSON run--success stderr diverges from MCP JSON fixture for the same scenario

The CLI JSON fixture (cli/json/swift-package/run--success.json) records seven SPM dependency-resolution stderr lines (Fetching/Fetched/Computing/Computed/Creating/Working copy) that the MCP JSON fixture (mcp/json/swift-package/run--success.json) does not contain. Both fixtures capture the same deterministic xcodebuildmcp.output.build-run-result scenario, so their data.output.stderr arrays should match. The divergence indicates the CLI fixture was regenerated against a different dependency-cache state (cold vs. warm) rather than kept aligned, breaking the MCP/CLI/JSON fixture alignment guardrail.

Evidence

cli/json/swift-package/run--success.json stderr begins with Fetching .../swift-argument-parser.git from cache through Working copy ... resolved at 1.5.1, then Building for debugging....

mcp/json/swift-package/run--success.json stderr for the identical scenario starts directly at Building for debugging... with none of those seven lines (grep for Fetching https://github.com/apple/swift-argument-parser matches only the CLI file).

Text fixtures (cli/text and mcp/text run--success.txt) surface no stderr detail, so the mismatch is isolated to the JSON envelopes.

The extra lines reflect volatile dependency-cache state (cold fetch/resolve) that is inconsistently captured between transports rather than normalized identically.

Also found at 1 additional location

src/snapshot-tests/__fixtures__/cli/json/swift-package/run--success.json:31-32

_{Identified by Warden xcodebuildmcp-snapshot-fixture-review · 7LW-6HX}

cameroncooke added 2 commits July 16, 2026 12:39

feat(xcodebuild): Support prepared test artifacts

79d6b5c

Extend existing build and test tools to prepare, return, and consume reusable test artifacts while preserving normal build behavior. Fixes #450

feat(manifest): Support conditional next steps

65b862b

Move conditional next-step selection into tool manifests and preserve the behavior across MCP, CLI, daemon, and snapshot outputs. Refs #450

cameroncooke marked this pull request as ready for review July 16, 2026 16:19

sentry-warden Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread src/snapshot-tests/__fixtures__/cli/text/macos/test--failure.txt

cursor Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread src/utils/test-common.ts

fix(testing): Stabilize prepared test execution

e359897

Keep source-only xcodebuild arguments out of the generated test-without-building phase and sort final test failures independently of emission order.

sentry Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread src/mcp/tools/device/build_device.ts

cursor Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread src/mcp/tools/device/build_device.ts

sentry-warden Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread src/snapshot-tests/__fixtures__/mcp/json/simulator/test--error-compiler.json

test(snapshot): Align device test schema versions

88da782

cursor Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread src/utils/test-common.ts Outdated

fix(test): Extract failures from managed result bundles

e7b8d53

cursor Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread src/utils/test-common.ts

cameroncooke added 3 commits July 16, 2026 20:08

fix(test): Preserve explicit test destinations

f95b573

meta: Merge main into feature branch

c3683ea

github-code-quality Bot found potential problems Jul 17, 2026

View reviewed changes

Comment thread src/snapshot-tests/preflight/simulator.ts Fixed

cursor Bot reviewed Jul 17, 2026

View reviewed changes

Comment thread src/utils/test-source.ts

sentry Bot reviewed Jul 17, 2026

View reviewed changes

Comment thread src/utils/test-common.ts

github-actions Bot reviewed Jul 17, 2026

View reviewed changes

Comment thread src/snapshot-tests/__fixtures__/cli/json/swift-package/run--success.json Outdated

Comment thread src/snapshot-tests/__fixtures__/mcp/text/device/test--error-compiler.txt

sentry-warden Bot reviewed Jul 17, 2026

View reviewed changes

fix(testing): address prepared test review findings

a925f49

Prevent session defaults and selectors from leaking into prepared test runs, remove a useless simulator state initializer, and align volatile snapshot fixtures across CLI and MCP.

cursor Bot reviewed Jul 17, 2026

View reviewed changes

Comment thread src/utils/test-source.ts

sentry Bot reviewed Jul 17, 2026

View reviewed changes

Comment thread src/mcp/tools/macos/build_macos.ts

cameroncooke added 2 commits July 17, 2026 20:35

fix(testing): preserve prepared test selectors

2ee73fa

Keep only-testing and skip-testing arguments in the prepared test phase while retaining session-default conflict filtering.

fix(testing): only mark successful test products complete

ae6d5ba

cursor Bot reviewed Jul 17, 2026

View reviewed changes

Comment thread src/mcp/tools/device/build_device.ts

sentry-warden Bot reviewed Jul 17, 2026

View reviewed changes

fix: stabilize prepared test snapshot fixtures

a233718

sentry Bot reviewed Jul 18, 2026

View reviewed changes

Comment thread src/utils/workspace-filesystem-lifecycle.ts

fix: honor test product retention options

9a22305

cursor Bot reviewed Jul 18, 2026

View reviewed changes

Comment thread src/mcp/tools/device/build_device.ts

Comment thread src/mcp/tools/device/build_device.ts

fix: reject device destinations without prepared tests

f0a9b1e

sentry Bot reviewed Jul 18, 2026

View reviewed changes

Comment thread src/utils/test-common.ts

cursor Bot reviewed Jul 18, 2026

View reviewed changes

Comment thread src/utils/typed-tool-factory.ts

fix: isolate session destinations for device builds

cda637e

cursor Bot reviewed Jul 18, 2026

View reviewed changes

Comment thread src/mcp/tools/device/build_device.ts

fix: mark failed prepared builds complete

191b74e

sentry-warden Bot reviewed Jul 18, 2026

View reviewed changes

Uh oh!

Uh oh!

Conversation

cameroncooke commented Jul 16, 2026

Why

What Changed

Uh oh!

pkg-pr-new Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sentry-warden Bot Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

sentry-warden Bot Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

sentry-warden Bot Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

pkg-pr-new Bot commented Jul 16, 2026 •

edited

Loading