Skip to content

Improve alpha testing coverage for MCP output, host compatibility, and npm smoke tests #1

Description

@shreyash-chonkie

Context

The current alpha test suite is credible: it covers compiler discovery, MCP runtime methods, auth, pagination, widget bundling, platform-specific files, diagnostics, package metadata, and npm install/create smoke tests. Before treating Sidecar as stable, we should raise the bar around generated artifacts and real host compatibility.

Current coverage

  • Unit/runtime tests for tools, resources, prompts, auth, proxy, pagination, schema enforcement, and the MCP Apps bridge.
  • Compiler tests for reserved files, folder-derived ids/URIs, widget resources, generated clients, platform-specific file selection, and diagnostics.
  • E2E build tests for plain MCP, ChatGPT target, Claude plugin output, widget URI cache busting, path containment, and plugin filename/frontmatter sanitization.
  • Packaging checks for exports/bins and smoke tests for npm install sidecar-ai and npm create sidecar-app@latest.

Work to add

  • Add golden artifact snapshots for every generated manifest/plugin file.
  • Validate emitted MCP/plugin artifacts against official JSON Schemas where available.
  • Add live host acceptance tests for ChatGPT and Claude where automation is possible.
  • Add browser visual regression tests for native components across light/dark mode and host recipes.
  • Add CI smoke tests that install from npm, scaffold an app, run sidecar check, and run sidecar build.
  • Add more adversarial/fuzz tests for JSON-RPC request handling, auth, proxy middleware, and path handling.

Acceptance criteria

  • CI has a job that validates generated artifacts structurally, not only through hand-written expectations.
  • CI has a published-package smoke test path that exercises sidecar-ai and create-sidecar-app from npm.
  • We have a documented distinction between fast unit tests, build artifact tests, visual tests, and live host acceptance tests.
  • Failures identify whether the breakage is compiler output, MCP protocol behavior, host-specific behavior, packaging, or visual/native UI parity.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions