Skip to content

Refuse to persist an empty control-plane snapshot in rad shutdown - #12849

Open
pujitha24 wants to merge 6 commits into
radius-project:mainfrom
pujitha24:auto/issue-12847
Open

Refuse to persist an empty control-plane snapshot in rad shutdown#12849
pujitha24 wants to merge 6 commits into
radius-project:mainfrom
pujitha24:auto/issue-12847

Conversation

@pujitha24

Copy link
Copy Markdown
Contributor

Summary

rad shutdown now refuses to commit a control-plane backup to the durable
state archive when the backed-up ucp database contains zero rows in the
resources table (the table that stores every UCP resource, including
resource groups). It bails out with a clear error and leaves the existing
archive untouched, instead of silently persisting the degenerate snapshot.

Reason for change

The repo-based deploy pipeline restores durable state with rad startup and
persists it with rad shutdown. A prior fix (#12840) added a workflow-level
guard so rad shutdown only runs when rad startup actually restored state
earlier in the same run — but that guard only proves rad startup ran, not
that the control plane is still healthy by the time rad shutdown runs. If
the control plane degrades after a successful startup (e.g. the PostgreSQL
pod crash-loops, or rad install is re-run mid-pipeline), rad shutdown
would still dump and commit an empty database, corrupting the shared archive
for every future run. This is defense-in-depth inside the command itself, so
the archive is protected regardless of how rad shutdown is invoked.

Fixes #

How to test

go test ./pkg/cli/pgbackup/... ./pkg/cli/cmd/shutdown/... -v

All tests pass, including:

  • pkg/cli/pgbackup/pgbackup_test.go: Test_IsControlPlaneEmpty_* — unit
    tests for the new IsControlPlaneEmpty helper against hand-built
    COPY ... FROM stdin; fixtures (no data rows, no resources table at all,
    data rows present, missing file).
  • pkg/cli/cmd/shutdown/shutdown_test.go: Test_Run_EmptyControlPlaneStopsBeforeCommit
    — asserts Run returns an error and never reaches BackupTerraform or
    session.Commit when the database backup is empty.

Validation

  • go build ./pkg/cli/pgbackup/... ./pkg/cli/cmd/shutdown/... passes.
  • go test ./pkg/cli/pgbackup/... ./pkg/cli/cmd/shutdown/... -v passes (all
    cases, old and new).
  • Fail-then-pass check: with only the two source files (pgbackup.go,
    shutdown.go) reverted and the test files left in place, the package fails
    to build (undefined: IsControlPlaneEmpty) and the new shutdown test fails
    because Commit is called unexpectedly — confirming the added check is
    what prevents the commit.
  • go vet ./pkg/cli/pgbackup/... ./pkg/cli/cmd/shutdown/... is clean;
    gofmt -l reports no formatting issues on the changed files.
  • golangci-lint run (pinned repo version v2.13.1, from build/tools.yaml)
    on both changed packages reports 0 issues.
  • The IsControlPlaneEmpty parsing logic (matching the pg_dump plain-format
    COPY <table> (<cols>) FROM stdin; ... rows ... \. block) was verified
    against a real, locally-run PostgreSQL 16 instance: a resources table was
    created with the exact schema this codebase writes
    (pkg/components/database/postgres/postgresclient.go), and real
    pg_dump --data-only output was captured for both an empty table and a
    populated one (including a row with escaped backslashes/tabs), then fed
    through the function directly — both cases classified correctly. This
    environment has no live Radius cluster or kubectl, so the change was not
    exercised end-to-end through rad shutdown against a real control plane;
    the above is the closest available proxy for that.
  • Branch HEAD is the current tip of upstream/main (commit 745ce9cc0), and
    upstream/main's own CI (Unit Tests, CodeQL) is green as of this writing.

File change summary

File Summary of change
pkg/cli/pgbackup/pgbackup.go Add IsControlPlaneEmpty(stateDir), which detects a ucp.sql dump with zero rows in the resources table (or no such table at all).
pkg/cli/pgbackup/pgbackup_test.go Add unit tests for IsControlPlaneEmpty covering empty, missing-table, populated, and missing-file cases.
pkg/cli/cmd/shutdown/shutdown.go Call IsControlPlaneEmpty after the database backup and before committing; refuse to persist and return a clear error when the backup is empty.
pkg/cli/cmd/shutdown/shutdown_test.go Make fakeStateBackupClient.BackupDatabases write a realistic ucp.sql fixture; add Test_Run_EmptyControlPlaneStopsBeforeCommit.

Fixes #12847

Copilot AI lite review requested due to automatic review settings August 27, 2026 14:43
@pujitha24
pujitha24 requested review from a team as code owners August 27, 2026 14:43
@pujitha24
pujitha24 had a problem deploying to external-contributor-approval August 27, 2026 14:44 — with GitHub Actions Error

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens rad shutdown so it will not overwrite the durable state archive with an obviously-degenerate control-plane snapshot (specifically: a ucp.sql dump that has zero rows in the resources table). This adds defense-in-depth beyond workflow-level guards by enforcing the invariant inside the command itself.

Changes:

  • Add pgbackup.IsControlPlaneEmpty(stateDir) to detect a ucp.sql dump with no resources rows (or no resources COPY block).
  • Update rad shutdown to bail out before Terraform backup / archive commit when the control-plane snapshot is empty.
  • Add unit tests for the dump-inspection helper and a shutdown-flow test asserting no commit occurs on empty snapshots.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
pkg/cli/pgbackup/pgbackup.go Adds IsControlPlaneEmpty with a pg_dump COPY-header match for resources and an empty-dump check.
pkg/cli/pgbackup/pgbackup_test.go Adds unit tests covering empty/no-table/non-empty/missing-file dump scenarios.
pkg/cli/cmd/shutdown/shutdown.go Calls IsControlPlaneEmpty after DB backup and returns a user-facing error before committing when empty.
pkg/cli/cmd/shutdown/shutdown_test.go Updates the DB-backup fake to write a realistic ucp.sql fixture and adds a regression test ensuring empty backups stop before commit.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +100 to +105
func IsControlPlaneEmpty(stateDir string) (bool, error) {
path := filepath.Join(stateDir, "ucp.sql")
data, err := os.ReadFile(path)
if err != nil {
return false, fmt.Errorf("failed to read backup file %q: %w", path, err)
}
@pujitha24
pujitha24 had a problem deploying to external-contributor-approval August 27, 2026 19:32 — with GitHub Actions Error
@pujitha24
pujitha24 had a problem deploying to external-contributor-approval August 28, 2026 10:14 — with GitHub Actions Error
@pujitha24
pujitha24 had a problem deploying to external-contributor-approval August 29, 2026 09:08 — with GitHub Actions Error
@willdavsmith
willdavsmith requested a review from sk593 August 31, 2026 18:21
@pujitha24
pujitha24 had a problem deploying to external-contributor-approval August 31, 2026 20:28 — with GitHub Actions Error
@pujitha24
pujitha24 had a problem deploying to external-contributor-approval September 1, 2026 02:39 — with GitHub Actions Error
@pujitha24
pujitha24 had a problem deploying to external-contributor-approval September 1, 2026 17:37 — with GitHub Actions Error
@pujitha24
pujitha24 had a problem deploying to external-contributor-approval September 2, 2026 10:14 — with GitHub Actions Error
@pujitha24
pujitha24 had a problem deploying to external-contributor-approval September 3, 2026 10:36 — with GitHub Actions Error
Motivation:
The repo-based deploy pipeline restores durable state with `rad startup`
and persists it with `rad shutdown`. A prior fix gated `rad shutdown` on a
`state-restored` signal so it only runs after `rad startup` succeeded
earlier in the same run -- but that only proves startup ran, not that the
control plane is still healthy by the time shutdown runs. If it degrades
afterward (a PostgreSQL pod crash-loop, or `rad install` re-run mid-run),
`rad shutdown` would still dump and commit an empty database, corrupting
the shared archive for every future run.

Approach:
Add pgbackup.IsControlPlaneEmpty, which inspects the backed-up ucp.sql dump
for the pg_dump COPY block of the "resources" table (the table that stores
every UCP resource, including resource groups) and reports whether it has
zero data rows, or the table is missing entirely. rad shutdown calls this
after backing up the databases and before backing up Terraform state or
committing to the archive; when the backup is empty it returns a clear
clierrors.Message and leaves the existing archive untouched. This is
defense-in-depth inside the command itself, independent of the
workflow-level guard.

Validation:
go test ./pkg/cli/pgbackup/... ./pkg/cli/cmd/shutdown/... -v passes,
including new tests for IsControlPlaneEmpty (no data rows, missing table,
populated, missing file) and Test_Run_EmptyControlPlaneStopsBeforeCommit
(asserts Run stops before BackupTerraform/Commit on an empty backup).
Reverting only the two source files while keeping the test files causes a
build failure and an unexpected-Commit-call test failure, confirming the
new check is what prevents the commit. go vet and gofmt are clean, and
golangci-lint (pinned repo version v2.13.1) reports 0 issues on both
changed packages. The COPY-block parsing was additionally verified against
a real, locally-run PostgreSQL 16 instance with the exact "resources"
schema this codebase writes, using genuine pg_dump output for both an
empty and a populated table; this environment has no live Radius cluster,
so the change was not exercised end-to-end through a real control plane.

Report: radius-project#12847
Signed-off-by: Pujitha Paladugu <10557236+pujitha24@users.noreply.github.com>
Assisted-by: claude-sonnet-5 (via Claude Code)
@pujitha24
pujitha24 had a problem deploying to external-contributor-approval September 5, 2026 13:51 — with GitHub Actions Error
@pujitha24
pujitha24 had a problem deploying to external-contributor-approval September 8, 2026 18:42 — with GitHub Actions Error
@pujitha24
pujitha24 had a problem deploying to external-contributor-approval September 9, 2026 14:44 — with GitHub Actions Error
@sk593

sk593 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

The validation is thorough and the approach is sound. A couple of suggestions to strengthen the PR further:

  1. Test coverage for schema-qualified table names: The regex already handles variants like COPY myschema.resources (...) and COPY "public"."resources" (...), but these aren't explicitly tested. Adding a unit test for schema-qualified names would round out coverage and confirm the edge case works as intended.

This is an optional enhancement; the PR is solid as-is. The test fixtures are realistic and accurately represent real pg_dump output, and the integration test correctly verifies that empty backups block before commit.

@pujitha24
pujitha24 had a problem deploying to external-contributor-approval September 10, 2026 08:54 — with GitHub Actions Error
@pujitha24
pujitha24 had a problem deploying to external-contributor-approval September 11, 2026 12:03 — with GitHub Actions Error
@pujitha24
pujitha24 requested a deployment to external-contributor-approval September 12, 2026 15:31 — with GitHub Actions Waiting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rad shutdown should refuse to persist a degenerate/empty state snapshot

3 participants