Skip to content

DONE-52291ad1.md

Latest commit

 

History

History
118 lines (109 loc) · 6.96 KB

DONE-52291ad1.md

File metadata and controls

118 lines (109 loc) · 6.96 KB
Raw

DONE evidence — task 52291ad1 (self-serve template create/update for Orin)

Re-verified on current origin/main (fdfc6929) and read-only prod on 2026-11-05. The mechanism the task asks for already exists end to end: Orin creates, updates, pins, and resets Templates through the CLI/portal with his portal identity (no Git access, no Developer), and the knowledge repository's main-branch CI re-seeds the live Templates from workflow-templates/*.yaml on every push. No product-code gap was found, so this PR is docs plus one test that locks in the CI-only sync route. No prod writes were made.

1. Mechanism, end to end

  • Auto-sync on knowledge main (dataops side). The knowledge publisher handler routes POST /api/templates/sync to syncTemplates (backend/src/auth/knowledgePublisher.ts:129), which runs seedDeploymentRuntime — "the same reconciliation a deploy runs" (backend/src/auth/knowledgePublisher.ts:138-161) — and reports pinned types (:152-156).
  • Auto-sync (caller side). DataTalksClub/dataops-knowledge .github/workflows/validate.yml runs the sync_workflow_templates job on main pushes (and non-dry-run workflow_dispatch), gated on needs: [regenerate, validate, publish], and executes node .dataops/cli/bin/dataops.mjs templates sync with an OIDC token minted for audience dataops-knowledge-publish. A push to knowledge main therefore refreshes live Templates without a deploy.
  • Auth for publish/sync. GitHub OIDC only: the bearer must be a GitHub Actions token whose signature verifies against GitHub's JWKS with audience dataops-knowledge-publish (backend/src/auth/knowledgePublisher.ts:24-28, :86-108), bound to the configured repository, refs/heads/main, and workflow .github/workflows/validate.yml (:27, policy :77-83, binding check :103-106; env wiring in infra/template.full.yaml:1330-1331). No other credential reaches those routes, so Orin cannot trigger a sync directly — by design it rides his knowledge-repo pushes' CI (or a deploy, backend/src/handler.ts:327).
  • Contract docs. docs/decisions/dataops-knowledge-repository/part-2-contents-and-workflow-templates.md:109-130: the work engine seeds from Git YAML on every deploy; operators edit the effective definition in the portal (Tasks > Templates > Edit definition) or the CLI; both write the same PUT /api/templates/:id compare-and-swap API validated by the same shared validators the knowledge repo's CI enforces; an edit pins the whole effective definition, a later deploy/sync refreshes the base but never overwrites the pin, and reset restores the base. part-4-runtime-portal-and-ci.md:73-101 specifies the knowledge-repo CI workflow and the OIDC refresh model.

2. Safe CLI update path

  • Commands (cli/src/commands/templates.mjs:394): list, show, create, update, reset, delete, sync.
  • update (:307-338) sends PUT /api/templates/:id with the expectedVersion/revision CAS pair (versionPair, :149-168: explicit flags > definition file's own pair > fresh fetch) and --dry-run (:324); conflicts surface as exit-code 3 with the current pair (describeTemplateWriteError, :170-194, cli/README.md:154). reset (:340-359) and create (:253-269) preview with --dry-run too.
  • Server gates (backend/src/routes/templates.ts): update/pin and reset need an active operator or admin (:249, :173); custom create and delete need an admin (:133-135, :278-280); the invoice-reader and knowledge-publisher principals never reach these routes with usable credentials (:106-112). The server rejects client-supplied server-owned fields (rejectServerOwned, :72-77) and version bodies (:55-63).
  • Validation parity with the seed path. normalizeTemplateDefinition accepts runtime shape (taskDefinitions) or authored YAML shape (tasks, what workflow-templates/*.yaml uses) and validates both with validateEffectiveDefinition (backend/src/templates/templateEdits.ts:98-125, :128-162), which runs the same validateAuthoredTemplate the knowledge CI and the seed loader run (backend/src/templates/authoredTemplates.ts:136, :340) plus a dependency-graph pass (:144-160). The DB layer re-validates on create and pin (backend/src/db/templates.ts:626, :659).

3. Orin-specific gap analysis

  • Orin's identity. Read-only prod check (dataops users list): Orin Vale (orin@datatalks.club) holds portal role admin, so both gates pass: create/delete (admin-only) and update/reset/pin (operator-or-admin). Interactive actors authenticate by browser/bearer session or a scoped API token (templates:write permits /api/templates writes, backend/src/auth/tokenScopes.ts:75-79).
  • Templates without Git. dataops templates create --from-file def.yaml [--dry-run] creates a custom Template; dataops templates update <ref> (--from-file … | --task …) pins an edited effective definition; reset restores the Git base. No Git checkout, no knowledge-repo push, no Developer involved. The portal exposes the same pin via Tasks > Templates > Edit definition (contract doc cited above).
  • Survival of the auto-sync. The seed preflight never deletes or overwrites operator work: custom rows are left alone unless an authored Git template later claims the same type, which fails closed (backend/src/templates/authoredTemplates.ts:324-328, :384-387); pinned Git rows keep their effective definition and follow the new base (:391-410). Derived KB views are regenerated after custom create, pin, and reset (backend/src/routes/templates.ts:142-150, :264-266, :205-207; landed in #348 / 23c6c9aa).
  • Prod evidence (read-only). dataops templates list: aishl-workshop is origin=custom (10 tasks) — a Template created through the API path, alive across many knowledge pushes and deploys; 10+ Git templates carry pinned=yes — operator pins surviving syncs; dev-sync-smoke is the live Git-sync proof row.
  • Conclusion: no gap. The flow the task asks for (create/update without Developer, validation-guarded, safe preview) is exactly the existing pin and custom-create design, already documented in the ADR and already used in prod. The only stale doc was the CLI README: the command table omitted templates sync, and the Template editing section did not state the event-driven Git re-seed or its CI-only auth.

4. What this PR changes

  • cli/README.md: add sync to the templates command-family row; new "Git stays canonical without a deploy" subsection stating the knowledge-main CI re-seed, its GitHub-OIDC-only auth (interactive 405), and that pins and custom Templates survive it — i.e. the operator self-serve flow needs no Git and no Developer.
  • backend/tests/api-templates.test.ts: one test asserting POST /api/templates/sync with an interactive actor is 405 without side effects, locking in the CI-only routing the new doc paragraph describes.
  • DONE-52291ad1.md: this file.