Support custom subject commands

[yusufozturk]

Summary by CodeRabbit

• New Features

  • Added configurable value resolvers for test cases.
  • Resolvers can run containerized commands and substitute results into endpoint commands and environment variables.
  • Added validation to ensure resolver definitions are unique, complete, and correctly referenced.

• Bug Fixes

  • Test execution now stops immediately when value resolution fails or produces empty output.
  • Isolated test-case settings for each run, preventing changes from one execution affecting others.
  • Preserved subject-specific configuration when running queued tests.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b230339b-218e-4425-944c-afdbf47f022e

📥 Commits

Reviewing files that changed from the base of the PR and between 6cbf2f3 and ea32b69.

📒 Files selected for processing (4)

• cmd/harness/main.go
• internal/config/case.go
• internal/config/clone_test.go
• internal/runner/runner.go

🚧 Files skipped from review as they are similar to previous changes (1)

• internal/runner/runner.go

---

Walkthrough

The change adds resolver definitions to test cases, validates their references, clones cases for each run, and executes resolvers before test dispatch. Resolver output replaces placeholders in endpoint commands and environment variables.

Changes

Resolver values and per-run execution

Layer / File(s)	Summary
Resolver configuration and validation internal/config/case.go	TestCase now defines ResolveValue entries. Validation checks names, images, commands, duplicate names, and ${NAME} references.
Per-run case cloning and queue integration internal/config/case.go, internal/config/clone_test.go, cmd/harness/main.go	CloneForRun deep-copies mutable endpoint, environment, and Vault data. Test queues clone the case for each subject. Tests cover mutation isolation and empty cases.
Resolver execution and substitution internal/runner/runner.go	Runner executes Docker-based resolvers with subject overrides and the run timeout. It rejects failures and empty output, substitutes values, and aborts before test dispatch on errors.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Harness
  participant Runner
  participant Docker
  participant TestFlow
  Harness->>Runner: Provide cloned test case
  Runner->>Docker: Execute resolver command
  Docker-->>Runner: Return output or stderr
  Runner->>Runner: Substitute output in commands and environment
  Runner->>TestFlow: Dispatch resolved test case

Loading

Suggested reviewers: erenaslandev

Poem

A rabbit clones each case with care,
Resolver values hop through air.
Docker returns the output bright,
Placeholders change before the flight.
Each test runs with its own fresh state.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: support for custom commands that resolve subject-specific values before test execution.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch dev-subject-updates

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/runner/runner.go`:
- Around line 183-193: Replace the hardcoded 60-second timeout in the resolve
execution flow around exec.CommandContext with a configurable duration,
preferably deriving it from r.opts.Timeout or the existing runner timeout
configuration. Ensure cold image pulls receive the full configured timeout while
preserving cancellation and error propagation behavior.
- Around line 167-215: The non-`--all-tests` subject-run setup must create an
independent copy of `*config.TestCase` for every `runPair` instead of reusing
the same pointer. Update the code that constructs each run pair so
`resolveValues` can mutate that case’s endpoint commands and environment without
affecting subsequent subjects, while preserving the existing per-subject
execution flow.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 4693e7d7-23b6-4c49-8321-d25509524a18

📥 Commits

Reviewing files that changed from the base of the PR and between 8b48d27 and 6cbf2f3.

📒 Files selected for processing (2)

• internal/config/case.go
• internal/runner/runner.go

[cloudflare-workers-and-pages]

Deploying pipebench with    Cloudflare Pages

Latest commit:	ea32b69
Status:	✅  Deploy successful!
Preview URL:	https://e069bb4d.pipebench.pages.dev
Branch Preview URL:	https://dev-subject-updates.pipebench.pages.dev

View logs