cli-e2e-test-patterns.md

CLI E2E Test Fixture Patterns

Overview

CLI e2e tests use a shared fixture app pattern to ensure proper module resolution and reduce duplication. This pattern creates isolated test environments that can load config files with workspace dependencies.

Test Types: In-Process vs Subprocess

There are two approaches to CLI testing:

In-Process Tests (Current test/integration/*.e2e.test.ts)

• Import command factories directly (e.g., createDbInitCommand())
• Mock process.exit and console.log to capture output
• Run commands via command.parseAsync() in the same Node process
• Pros: Faster, easier to mock database connections
• Cons: Not truly E2E (doesn't test CLI entrypoint, shebang, ESM/CJS interop)

Subprocess Tests (True E2E, e.g., cli.emit-cli-process.e2e.test.ts)

• Spawn the CLI as a separate Node process using execFileAsync('node', [cliPath, ...])
• Test the actual built CLI binary (dist/bin.mjs)
• Pros: Tests real CLI behavior, catches ESM/CJS issues
• Cons: Slower, requires build first, harder to mock

Recommendation: Use in-process tests for fast iteration during development, but add subprocess tests for critical CLI flows.

Structure

Shared fixture app directory:

packages/framework/tooling/cli/test/cli-e2e-test-app/
  package.json          # Static, for pnpm install (dependencies must be installed)
  fixtures/
    {command}/          # Fixtures organized by command
      contract.ts
      prisma.config.ts
      prisma.config.{variant}.ts

Test execution:

• Each test creates an ephemeral directory (e.g., test-1234567890-abc123/)
• Files are copied from the fixture subdirectory to the ephemeral directory
• Test directories are subdirectories of the fixture app and inherit workspace dependencies from the parent package.json at the root
• Tests run commands from within the ephemeral directory

Pattern

✅ CORRECT: Use shared fixture app with command-specific subdirectories

import { withTempDir, setupTestDirectoryFromFixtures } from './utils/test-helpers';

// Fixture subdirectory for this command's tests
const fixtureSubdir = 'emit';

withTempDir(({ createTempDir }) => {
  describe('command e2e tests', () => {
    it('test description', async () => {
      // Set up test directory from fixtures
      const testSetup = setupTestDirectoryFromFixtures(
        createTempDir,
        fixtureSubdir,
        'prisma.config.emit.ts', // Config file name
        { '{{PLACEHOLDER}}': 'value' }, // Optional replacements
      );
      const testDir = testSetup.testDir;
      const configPath = testSetup.configPath;

      // Run command from testDir
      const originalCwd = process.cwd();
      try {
        process.chdir(testDir);
        await executeCommand(command, ['--config', 'prisma.config.ts']);
      } finally {
        process.chdir(originalCwd);
      }
    });
  });
});

Helper Function

setupTestDirectoryFromFixtures(createTempDir: () => string, fixtureSubdir, configFileName?, replacements?)

• createTempDir: Required. Function that returns a new ephemeral test directory path.
• fixtureSubdir: Name of the fixture subdirectory (e.g., 'emit', 'db-verify')
• configFileName: Optional. Name of the config file to copy (defaults to 'prisma.config.ts')
• replacements: Optional. Object mapping placeholders to values (e.g., { '{{DB_URL}}': connectionString })

Returns:

• testDir: Path to the ephemeral test directory
• contractPath: Path to the copied contract.ts file
• outputDir: Path to the output/ subdirectory
• configPath: Path to the copied config file

Note: If you need a cleanup function, use setupIntegrationTestDirectoryFromFixtures(...) instead (it returns { ..., cleanup }).

Key Points

1. Static package.json: The package.json at the root of cli-e2e-test-app must have dependencies installed via pnpm install. This ensures workspace packages can be resolved when loading config files.

2. No package.json in test directories: Test directories are subdirectories of the fixture app and inherit workspace dependencies from the parent package.json at the root. jiti will resolve workspace packages by walking up to find the parent package.json. This avoids pnpm workspace conflicts when multiple test directories exist.

3. Fixture organization: Organize fixtures by command in subdirectories. This keeps related fixtures together and makes it easy to find command-specific configs.

4. Placeholder replacement: Use placeholders like {{DB_URL}} in fixture configs and replace them at test time. This allows tests to inject dynamic values (e.g., database connection strings).

5. Ephemeral directories: Each test gets its own directory, ensuring isolation. Each test must clean up its own directory.

6. Cleanup responsibility: Prefer withTempDir(...) for per-test cleanup (it tracks directories and removes them in afterEach). If you need an explicit cleanup function, use setupIntegrationTestDirectoryFromFixtures(...). Never use afterAll hooks for cleanup - they can interfere with tests that are still running and cause race conditions. Never use global cleanup functions that scan and delete directories - this causes race conditions when tests run in parallel.

7. Module resolution: Tests run from within the ephemeral directory, which is a subdirectory of the fixture app. Node's module resolution walks up to find the parent package.json and node_modules, allowing workspace packages to be resolved correctly.

When to Use

• ✅ CLI e2e tests that need to load config files
• ✅ Tests that need to verify command behavior with different config variants
• ✅ Tests that need isolated environments per test case

When NOT to Use

• ❌ Unit tests (use mocks instead)
• ❌ Integration tests that don't need config file loading
• ❌ Tests that don't need workspace package resolution

Examples

Multiple config variants:

fixtures/db-verify/
  contract.ts
  prisma.config.ts                    # Basic config
  prisma.config.with-db.ts           # Config with database
  prisma.config.no-driver.ts         # Config missing driver
  prisma.config.no-verify.ts         # Config missing verify.readMarkerSql

Using placeholders:

withTempDir(({ createTempDir }) => {
  const testSetup = setupTestDirectoryFromFixtures(
    createTempDir,
    'db-verify',
    'prisma.config.with-db.ts',
    { '{{DB_URL}}': connectionString },
  );

  // ... test code ...
});

Using afterEach for cleanup:

withTempDir(({ createTempDir }) => {
  describe('command tests', () => {
    let testSetup: ReturnType<typeof setupTestDirectoryFromFixtures>;

    beforeEach(() => {
      testSetup = setupTestDirectoryFromFixtures(createTempDir, fixtureSubdir);
    });

    it('test description', async () => {
      // ... test code using testSetup.testDir ...
    });
  });
});

Testing CLI as Separate Process

For tests that need to run the CLI as a separate Node.js process (e.g., to test the built CLI binary), use execFileAsync with the cwd option:

✅ CORRECT: Use execFileAsync with cwd option

import { execFile } from 'node:child_process';
import { promisify } from 'node:util';
import { setupIntegrationTestDirectoryFromFixtures } from './utils/test-helpers';

const execFileAsync = promisify(execFile);

describe('CLI process e2e', () => {
  it('executes CLI as separate process', async () => {
    const testSetup = setupIntegrationTestDirectoryFromFixtures('emit-command');
    const { testDir, cleanup } = testSetup;
    const cliPath = resolve(__dirname, '../dist/bin.mjs');

    try {
      // Set cwd for spawned process so relative paths in config resolve correctly
      await execFileAsync(
        'node',
        [cliPath, 'contract', 'emit', '--config', 'prisma.config.ts'],
        {
          cwd: testDir, // Set working directory for spawned process
        },
      );
    } catch (error: unknown) {
      // Only log output on errors for debugging
      if (error && typeof error === 'object' && 'stderr' in error) {
        console.error('CLI stderr:', error.stderr);
      }
      if (error && typeof error === 'object' && 'stdout' in error) {
        console.log('CLI stdout:', error.stdout);
      }
      throw error;
    } finally {
      cleanup();
    }
  });
});

Key points:

• Use execFileAsync (promisified execFile) to run CLI as separate process
• Always set cwd option to testDir so relative paths in config files resolve correctly
• execFile captures stdout/stderr by default - no need for stdio option (that's for spawn)
• Output files are written to the temp test directory and cleaned up automatically
• Use setupIntegrationTestDirectoryFromFixtures for tests that need package resolution (e.g., loadContractFromTs)

❌ WRONG: Don't use stdio option with execFile

// ❌ WRONG: execFile doesn't support stdio option (that's for spawn)
await execFileAsync('node', [cliPath, 'emit'], {
  stdio: 'pipe', // Type error: stdio doesn't exist on ExecFileOptions
  cwd: testDir,
});

Why?

• execFile doesn't support stdio option - that's for spawn
• execFile captures stdout/stderr by default in the result object
• Only log output on errors for cleaner test output

In-Process Command Tests

The commander shell and its test-helpers.ts harness were deleted in the S5 cutover. In-process command tests now mount the engine command tree with createTestCli (see test/orm/ in the CLI package) and assert on settled envelopes, events, and exit codes; only the small smoke set (test/integration/test/cli.bin-smoke.e2e.test.ts) spawns the real dist/bin.mjs.

Related Patterns

• docs/Testing Guide.md: General testing patterns
• docs/reference/test-import-patterns.md: Test import patterns
• test/integration/test/utils/cli-test-helpers.ts: In-process engine harness (createTestCli, runOnEngine) used by CLI integration tests