Skip to content

Add provider-conformance harness, contract test, and reframe docs to 'agent harness' - #3006

Merged
asim merged 2 commits into
masterfrom
codex/assess-go-micro-project-for-development-strategy
Jun 24, 2026
Merged

asim merged 2 commits into
masterfrom
codex/assess-go-micro-project-for-development-strategy

Conversation

@asim

@asim asim commented Jun 24, 2026

Copy link
Copy Markdown
Member

Motivation

  • Provide a repeatable cross-provider conformance check so the same end-to-end harnesses can be exercised against every configured live model provider without silently burning API credits.
  • Make the documented 0→1 getting-started path impossible to accidentally break by locking the micro new output with an automated contract test.
  • Clarify product messaging and internal strategy by repositioning the project as an "agent harness" and capturing the roadmap/assessment in docs.

Description

  • Add a new provider conformance harness at internal/harness/provider-conformance/main.go that runs specified harnesses (universe, agent-flow, plan-delegate) across a configurable list of providers and skips providers without API keys.
  • Update CI workflow .github/workflows/universe.yml to call internal/harness/provider-conformance for scheduled/manual runs and export common provider env vars so configured providers are checked; the previous Atlas-only live check was replaced.
  • Add a provider-conformance target to the Makefile and document harness/provider-conformance usage, and wire the deterministic harness targets to make harness.
  • Add an automated contract test cmd/micro/cli/new/contract_test.go to validate micro new helloworld produces a buildable Go service and update many docs/website files and internal/docs/DEVELOPMENT_STRATEGY_ASSESSMENT.md to adopt the "agent harness" framing and guidance.

Testing

  • Ran unit tests with go test ./..., including the new TestZeroToOneContract, and they completed successfully.
  • Executed the deterministic harnesses with make harness (which runs go run ./internal/harness/universe, agent-flow, and plan-delegate) and observed they passed locally.
  • Verified the provider conformance command runs deterministically against the mock provider via go run ./internal/harness/provider-conformance -providers mock and succeeds, while live provider runs are skipped in environments lacking API keys (CI scheduled runs will skip missing keys and fail if a configured provider fails).

Codex Task

asim pushed a commit that referenced this pull request Jun 24, 2026
Follow-up to #3006:

- provider-conformance: build each harness to a temp binary and run that
  instead of 'go run'. 'go run' launches the harness as a child it doesn't
  kill on context cancellation, so a timed-out harness (which starts local
  services) could be orphaned and outlive the run. Running the built binary
  makes the per-run timeout actually terminate the work.
- contract test: skip under -short, and use 'go build ./...' instead of
  'go test ./...' (the contract is that the generated service builds). This
  keeps the default unit-test suite from shelling out to the toolchain and
  the network on every run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CmdEY7pYmV5zzwCjNJ4ykL

asim commented Jun 24, 2026

Copy link
Copy Markdown
Member Author

Review (from a second pass with Claude Code). Verified all three harnesses (universe, agent-flow, plan-delegate) accept -provider and handle mock, so the conformance runner is wired correctly. Lint/mock-universe/etcd are green.

Good: the conformance harness (skip-on-missing-key, per-run timeout, runs all and reports every failure before exiting non-zero) is solid; the "lock the micro new path" contract test is a valuable idea; and the docs thoughtfully reconcile the old "no harness" language with the new positive framing.

Mechanical fixes pushed as #3007 (stacked on this branch):

Left for you to decide (judgment, not mechanical):

  • Scope — this bundles the conformance harness, the contract test, and a broad "agent harness" rebrand (README headline, landing hero, <title>, docs, comparison, + a 194-line strategy doc). I'd split the rebrand into its own PR you sign off on explicitly — it's a positioning change that flips a term the docs previously used negatively, and it shouldn't ride in on test infra.
  • Minor: localRPCEnv sets NO_PROXY=*, which also bypasses the proxy for provider API calls (fine on CI runners, would bite behind a corporate proxy). And the strategy doc quotes the pre-rebrand README line as the current positioning.

No correctness blockers — land the conformance + contract test (with #3007), and consider pulling the rebrand out.


Reviewed with Claude Code.


Generated by Claude Code

@asim
asim merged commit 844ab4a into master Jun 24, 2026
8 checks passed
asim added a commit that referenced this pull request Jun 24, 2026
#3007)

Follow-up to #3006:

- provider-conformance: build each harness to a temp binary and run that
  instead of 'go run'. 'go run' launches the harness as a child it doesn't
  kill on context cancellation, so a timed-out harness (which starts local
  services) could be orphaned and outlive the run. Running the built binary
  makes the per-run timeout actually terminate the work.
- contract test: skip under -short, and use 'go build ./...' instead of
  'go test ./...' (the contract is that the generated service builds). This
  keeps the default unit-test suite from shelling out to the toolchain and
  the network on every run.


Claude-Session: https://claude.ai/code/session_01CmdEY7pYmV5zzwCjNJ4ykL

Co-authored-by: Claude <noreply@anthropic.com>
asim pushed a commit that referenced this pull request Jun 24, 2026
Follow-up to #3006:

- provider-conformance: build each harness to a temp binary and run that
  instead of 'go run'. 'go run' launches the harness as a child it doesn't
  kill on context cancellation, so a timed-out harness (which starts local
  services) could be orphaned and outlive the run. Running the built binary
  makes the per-run timeout actually terminate the work.
- contract test: skip under -short, and use 'go build ./...' instead of
  'go test ./...' (the contract is that the generated service builds). This
  keeps the default unit-test suite from shelling out to the toolchain and
  the network on every run.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CmdEY7pYmV5zzwCjNJ4ykL
asim added a commit that referenced this pull request Jun 24, 2026
#3008)

Follow-up to #3006:

- provider-conformance: build each harness to a temp binary and run that
  instead of 'go run'. 'go run' launches the harness as a child it doesn't
  kill on context cancellation, so a timed-out harness (which starts local
  services) could be orphaned and outlive the run. Running the built binary
  makes the per-run timeout actually terminate the work.
- contract test: skip under -short, and use 'go build ./...' instead of
  'go test ./...' (the contract is that the generated service builds). This
  keeps the default unit-test suite from shelling out to the toolchain and
  the network on every run.


Claude-Session: https://claude.ai/code/session_01CmdEY7pYmV5zzwCjNJ4ykL

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant