Skip to content

P0: Add a canonical EvidenceManifest bound to case results #630

Description

@wanghuan-520

What needs doing

Add one canonical EvidenceManifest contract that is digest-bound to canonical Case results and shared by every revised execution route.

Current test-artifacts.summary.v1 is intentionally shallow. The older test-artifacts.manifest.v1 has stronger integrity semantics, but revised CLI/HTTP and agentic-browser routes do not consistently produce it. A Case result can currently point at evidence without one public manifest proving identity, media, digest, size, producer, and provenance.

Dependency

Blocked by #629. Do not route this issue to an FKST session until #629 is merged into fkst-hosted-default.

Required implementation

Define and validate testing-evidence-manifest.v1 with:

  • repository, run, and immutable plan identity;
  • manifest schema/version and canonical digest;
  • entries containing:
    • evidence_id;
    • case_id;
    • optional assertion_id;
    • Evidence role;
    • artifact pointer;
    • SHA-256 digest;
    • media type;
    • byte size;
    • producer and producer version;
    • creation timestamp;
    • sensitivity/redaction classification;
    • redaction or evidence-policy version/status;
    • provenance/source identity.

Add a contract-level binding from canonical CaseResultSet to EvidenceManifest ref/digest. The manifest must be pointer-only and must not embed raw evidence bytes.

Likely files

  • libraries/contract/**
  • contracts/**
  • packages/test-artifacts/**
  • packages/testing-runner/tests/**
  • shared fixture/conformance directories

Required semantic rules

  • Every Evidence reference in Case/Assertion results must resolve to exactly one manifest entry.
  • Evidence IDs are unique within a manifest.
  • Entry case/assertion identities must belong to the bound result set.
  • Pointer, digest, media type, size, producer, and policy fields are immutable.
  • Cross-run, foreign-plan, duplicate, missing, or tampered Evidence fails closed.
  • Raw DOM, cookies, headers, credentials, environments, argv, and absolute local paths are not valid public Evidence values.

Out of scope

  • Implementing a raw quarantine or redaction engine.
  • Capturing screenshots or browser output.
  • Uploading bytes to Hosted.
  • Migrating CLI/HTTP or Browser executors.
  • Adding new Evidence media beyond the bounded initial allowlist.

Acceptance criteria

  • testing-evidence-manifest.v1 is documented and exported from one public contract owner.
  • CaseResultSet can bind the manifest by pointer and digest.
  • Positive fixtures cover runner log, screenshot, and bounded sanitized JSON roles.
  • Negative fixtures cover duplicate IDs, foreign cases/assertions, digest mismatch, size/media mismatch, unsupported role/media, missing entry, and cross-run pointers.
  • Canonical manifest digest is deterministic.
  • No fixture or event contains raw secret-bearing data or absolute paths.
  • Existing test-artifacts and testing-runner tests remain green.
  • scripts/run.sh test test-artifacts passes.
  • scripts/run.sh test testing-runner passes.
  • Full scripts/run.sh test passes, or an exact unrelated failure and clean targeted rerun are reported.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions