Repository navigation
Add provider-conformance harness, contract test, and reframe docs to 'agent harness' - #3006
Conversation
Follow-up to #3006: - provider-conformance: build each harness to a temp binary and run that instead of 'go run'. 'go run' launches the harness as a child it doesn't kill on context cancellation, so a timed-out harness (which starts local services) could be orphaned and outlive the run. Running the built binary makes the per-run timeout actually terminate the work. - contract test: skip under -short, and use 'go build ./...' instead of 'go test ./...' (the contract is that the generated service builds). This keeps the default unit-test suite from shelling out to the toolchain and the network on every run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CmdEY7pYmV5zzwCjNJ4ykL
|
Review (from a second pass with Claude Code). Verified all three harnesses ( Good: the conformance harness (skip-on-missing-key, per-run timeout, runs all and reports every failure before exiting non-zero) is solid; the "lock the Mechanical fixes pushed as #3007 (stacked on this branch):
Left for you to decide (judgment, not mechanical):
No correctness blockers — land the conformance + contract test (with #3007), and consider pulling the rebrand out. Reviewed with Claude Code. Generated by Claude Code |
#3007) Follow-up to #3006: - provider-conformance: build each harness to a temp binary and run that instead of 'go run'. 'go run' launches the harness as a child it doesn't kill on context cancellation, so a timed-out harness (which starts local services) could be orphaned and outlive the run. Running the built binary makes the per-run timeout actually terminate the work. - contract test: skip under -short, and use 'go build ./...' instead of 'go test ./...' (the contract is that the generated service builds). This keeps the default unit-test suite from shelling out to the toolchain and the network on every run. Claude-Session: https://claude.ai/code/session_01CmdEY7pYmV5zzwCjNJ4ykL Co-authored-by: Claude <noreply@anthropic.com>
Follow-up to #3006: - provider-conformance: build each harness to a temp binary and run that instead of 'go run'. 'go run' launches the harness as a child it doesn't kill on context cancellation, so a timed-out harness (which starts local services) could be orphaned and outlive the run. Running the built binary makes the per-run timeout actually terminate the work. - contract test: skip under -short, and use 'go build ./...' instead of 'go test ./...' (the contract is that the generated service builds). This keeps the default unit-test suite from shelling out to the toolchain and the network on every run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CmdEY7pYmV5zzwCjNJ4ykL
#3008) Follow-up to #3006: - provider-conformance: build each harness to a temp binary and run that instead of 'go run'. 'go run' launches the harness as a child it doesn't kill on context cancellation, so a timed-out harness (which starts local services) could be orphaned and outlive the run. Running the built binary makes the per-run timeout actually terminate the work. - contract test: skip under -short, and use 'go build ./...' instead of 'go test ./...' (the contract is that the generated service builds). This keeps the default unit-test suite from shelling out to the toolchain and the network on every run. Claude-Session: https://claude.ai/code/session_01CmdEY7pYmV5zzwCjNJ4ykL Co-authored-by: Claude <noreply@anthropic.com>
Motivation
micro newoutput with an automated contract test.Description
internal/harness/provider-conformance/main.gothat runs specified harnesses (universe,agent-flow,plan-delegate) across a configurable list of providers and skips providers without API keys..github/workflows/universe.ymlto callinternal/harness/provider-conformancefor scheduled/manual runs and export common provider env vars so configured providers are checked; the previous Atlas-only live check was replaced.provider-conformancetarget to theMakefileand documentharness/provider-conformanceusage, and wire the deterministic harness targets tomake harness.cmd/micro/cli/new/contract_test.goto validatemicro new helloworldproduces a buildable Go service and update many docs/website files andinternal/docs/DEVELOPMENT_STRATEGY_ASSESSMENT.mdto adopt the "agent harness" framing and guidance.Testing
go test ./..., including the newTestZeroToOneContract, and they completed successfully.make harness(which runsgo run ./internal/harness/universe,agent-flow, andplan-delegate) and observed they passed locally.mockprovider viago run ./internal/harness/provider-conformance -providers mockand succeeds, while live provider runs are skipped in environments lacking API keys (CI scheduled runs will skip missing keys and fail if a configured provider fails).Codex Task