SCA-40: add digest-pinned Pusher chart-only recovery

[Git-on-my-level]

Root cause

Historical Pusher pods can retain a named REDIS_DB_HOST secretKeyRef when Helm strategically merges the env list, even though the desired chart uses the shared backend ConfigMap. This produces CreateContainerConfigError without requiring a new application build.

Contract

Normal build-and-push remains the default. A new explicit Pusher-only chart-only mode accepts only the selected environment's approved repository, an exact sha256:<64 lowercase hex> digest, and a declared Ready containment baseline. The chart renders either repository:tag or exactly repository@digest, never both.

chart-only skips Docker registry login, Buildx, build, push, and non-secret ConfigMap writes. It uses the admitted workflow source while retaining the caller-approved serving image digest.

Safety and rollback

Before Helm mutation, the workflow captures the Helm revision plus Deployment/ReplicaSet/pod identity and validates target namespace/release, ConfigMap key presence without printing its value, Ready serving repository-and-digest image IDs, exact rendered digest, ConfigMap-only REDIS_DB_HOST, and an allowlisted structural diff. Service/HPA/PDB/Ingress/ServiceAccount/BackendConfig, probes, resources, and all unrelated Deployment fields must be unchanged.

It rechecks live ConfigMap/image identity immediately before apply, uploads only redacted evidence, waits for rollout, and post-verifies the exact digest and ConfigMap-only source. No automatic rollback is introduced: rollback means restoring the captured Helm revision/known seven-ready containment baseline, not claiming that the prior failed pods were healthy.

Verification

• backend/.venv/bin/pytest -q backend/tests/unit/test_verify_pusher_chart_only_recovery.py backend/tests/unit/test_verify_pusher_config_references.py backend/tests/unit/test_verify_pusher_rollout_budget.py backend/tests/unit/test_workflow_contracts.py backend/tests/unit/test_llm_gateway_deploy_contract.py — 81 passed
• Helm digest render and malformed/ambiguous identity rejection
• actionlint -ignore 'SC.*' .github/workflows/gcp_backend_pusher.yml
• python3 .github/scripts/check-deployment-concurrency.py
• python3 .github/scripts/check-direct-backend-production-admission.py
• make preflight

Dev rehearsal: dispatch only service=pusher, deployment_mode=chart-only with the dev approved repository, an exact digest that matches all Ready serving Pusher image IDs, and a baseline no greater than the observed Ready count; review the redacted artifact before considering production.

Production was not touched. No workflow dispatch, Helm mutation, or cluster mutation was performed.

Failure-Class: FC-runtime-image-boundary

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3e4207e7da

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Git-on-my-level]

@codex review exact current head 1c4f7dffa7891784e96211fd7d2cb513749014ec for release safety and chart-only/no-build correctness. Prior P1/P2 findings were addressed; please review the complete current diff.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1c4f7dffa7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Git-on-my-level]

@codex review exact current head 2bbd423534b04b5afa0437223fecfc627ec899ff after normalization fixes and latest-main integration. Review complete current diff for release safety, no-build proof, exact digest identity, structural drift fail-closed behavior, and rollback evidence.

[Git-on-my-level]

@codex review exact current head 2bbd423534b04b5afa0437223fecfc627ec899ff for the combined Pusher chart-only recovery fixes; please assess release safety and real-drift preservation.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2bbd423534

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Git-on-my-level]

Addressed Codex review feedback on verify_pusher_chart_only_recovery.py:

• P1 — Normalize Kubernetes defaults before drift compare (discussion_r3628552041): normalize() did not strip API-server/controller-populated Deployment fields (revisionHistoryLimit, restartPolicy, dnsPolicy, schedulerName, terminationMessagePath/Policy, deployment.kubernetes.io/revision annotation) that are absent from helm template output. This caused allowed_recovery_drift() to always fail when comparing a live Deployment against a rendered chart. Added _strip_deployment_defaults() and a new VOLATILE_DEPLOYMENT_* constant set.

• P2 — Allow omitted secretKeyRef after repair (discussion_r3628552054): validate_redis_source() used a sentinel object() default for .get("secretKeyRef"), so a post-repair env entry (where Kubernetes omits the key entirely after Helm applies secretKeyRef: null) was incorrectly flagged as a retained Secret source. Changed to plain .get("secretKeyRef") which returns None for a missing key.

• P2 — Ignore Helm annotations and Service cluster defaults (discussion_r3629894370): Helm-managed annotations (meta.helm.sh/release-name, meta.helm.sh/release-namespace) and cluster-added Service fields (internalTrafficPolicy) are absent from helm template output. Moved annotation stripping to normalize() for all kinds and added internalTrafficPolicy to VOLATILE_SERVICE_SPEC.

All 14 unit tests pass. The pre-existing test_smoke_what_matters_now and test_desktop_transcribe main regressions are unrelated to this PR.

[Git-on-my-level]

@codex review exact current head 8f759f921e31eb9e6edef825f47624d69bf594dc; focus on HPA-owned replica normalization and fresh pre-Helm Service/HPA/PDB captures preserving fail-closed drift detection.

[Git-on-my-level]

@codex review exact current head 8f759f921e31eb9e6edef825f47624d69bf594dc after HPA-authority and immediate-preapply recapture fixes. Review complete current diff for release safety, no-build proof, digest/config identity, concurrent-drift prevention, and fail-closed semantics.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8f759f921e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Git-on-my-level]

@codex review exact current head 6a4d6be9caaf51880e7bbbb301a07b957c213955; focus on canonical all-key rendered reference validation and GKE-only normalization preserving real drift.

[Git-on-my-level]

@codex review exact current head 6a4d6be9caaf51880e7bbbb301a07b957c213955 after full rendered ConfigMap-key validation and GKE/API default normalization fixes. Review complete diff for no-build release safety, key-name-only validation, real-drift preservation, digest identity, race closure, and rollback evidence.

[Git-on-my-level]

@codex review exact current head 5ca9810b6c098df10bec8cdb62ba6a2d04e4fbe3; focus on the immediate pre-Helm full rendered ConfigMap/Secret key verification, narrow GKE-only metadata/default normalization, and preservation of real policy/resource/probe drift detection.

[Git-on-my-level]

@codex review exact current head 5ca9810b6c098df10bec8cdb62ba6a2d04e4fbe3 after final pre-Helm ConfigMap reference recheck. Review complete current diff for no-build release safety, all rendered key references, known-only default normalization, real-drift preservation, and rollback evidence.

[Git-on-my-level]

@codex review exact current head e7538a31b33355423c00ba8f022ab679621cfd2e; the last independent P1 is corrected. Focus on allowing only named GKE controller metadata/defaults while detecting desired Helm/custom annotation drift, and on the immediate pre-Helm full rendered ConfigMap/Secret reference gate.

[Git-on-my-level]

@codex review exact current head e7538a31b33355423c00ba8f022ab679621cfd2e after final chart-annotation-drift correction. Review complete current diff for known-only normalization, all rendered config references, no-build safety, digest identity, real-drift preservation, and rollback evidence.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e7538a31b3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".