Skip to content

[ci] Add GPG-signed DEB packages with nfpm native signing#5371

Closed
PlatCore wants to merge 3 commits into
PlatCore/5109-refactor-rpm-for-reuse-v2from
PlatCore/5109-add-deb-gpg-signing-v3-nfpm
Closed

[ci] Add GPG-signed DEB packages with nfpm native signing#5371
PlatCore wants to merge 3 commits into
PlatCore/5109-refactor-rpm-for-reuse-v2from
PlatCore/5109-add-deb-gpg-signing-v3-nfpm

Conversation

@PlatCore
Copy link
Copy Markdown
Contributor

@PlatCore PlatCore commented May 13, 2026

Why this should be merged

Replaces #5180 (v2). Adds GPG-signed DEB packages. v2 used dpkg-sig, which is removed from Ubuntu 24.04. v3 switches to nfpm's native DEB signing.

Depends on #5179.

How this works

  • nfpm native signingsignature: { method: debsign, type: origin } in the DEB nfpm configs; passphrase via NFPM_DEB_PASSPHRASE.
  • Tool-independent verificationverify-deb-signature.sh extracts _gpgorigin from the .deb ar archive and runs gpg --verify. Works on jammy and noble.
  • Dockerfile.deb — Ubuntu 22.04 (glibc 2.35) builder.
  • Shared build-package.sh — dispatches on PKG_FORMAT (RPM/DEB); both formats now pass PKG_GPG_KEY_FILE.
  • validate-deb.sh — verifies + installs + smoke-tests in fresh ubuntu:22.04 and ubuntu:24.04 containers.
  • build-deb-release.yml — matrix amd64/arm64; reuses RPM_GPG_PRIVATE_KEY (single signing key for both formats).
  • Taskfile — secrets passed to docker run via -e VAR pass-through, not shell interpolation.
  • Removes build-deb-pkg.sh, debian/template/control.

How this was tested

  • task packaging:test-build-debs locally on macOS: built, signed, verified, installed, and smoke-tested in jammy and noble containers.
  • PR CI exercises the full path via build-deb-release.yml.

Need to be documented in RELEASES.md?

No

PlatCore added 3 commits May 12, 2026 18:28
Delegates DEB signing entirely to nfpm (same model as RPM)

— no custom signing shell,
- no gpg-agent,
- no gpg-preset-passphrase,
- no post-build dpkg-sig step.

Signing:
- nfpm config uses `deb.signature.key_file` with method: debsign,
  type: origin, embedding a detached PGP signature as the
  `_gpgorigin` member of the .deb ar archive
- gzip compression (broadest consumer tool compatibility)
- nfpm reads NFPM_DEB_PASSPHRASE from env for release keys — same
  interface as RPM's NFPM_RPM_PASSPHRASE

Verification:
- New verify-deb-signature.sh uses `ar x` + `gpg --verify` on the
  extracted signature member — tool-independent, identical in jammy
  and noble (no dpkg-sig dependency, no debsig-verify policy XML)

Files added:
- .github/packaging/Dockerfile.deb (Ubuntu 22.04 builder)
- .github/packaging/nfpm/{avalanchego,subnet-evm}-deb.yml
- .github/packaging/scripts/verify-deb-signature.sh
- .github/packaging/scripts/validate-deb.sh (validates in jammy + noble)
- .github/workflows/build-deb-release.yml

Files modified:
- .github/packaging/scripts/build-package.sh — accept PKG_GPG_KEY_FILE
  and dynamic NFPM_<FORMAT>_PASSPHRASE; keep RPM_GPG_KEY_FILE fallback
- .github/packaging/scripts/lib-validate-common.sh — DEB host-arch case
- .github/packaging/Taskfile.yml — DEB twin tasks (docker run model)

Files removed:
- .github/workflows/build-deb-pkg.sh (legacy unsigned dpkg-deb script)
- .github/workflows/debian/template/control (legacy control template)

Verified locally:
- task packaging:test-build-debs passes with ephemeral GPG key
- task packaging:test-build-debs passes with passphrase-protected GPG
  key (release path simulated via throwaway GNUPGHOME)
- Both runs: signature verified + dpkg -i + smoke test in fresh
  ubuntu:22.04 AND ubuntu:24.04 containers
`cp -r src dst` with an existing dst copies INTO dst, leaving the
old tree intact at .github/packaging/* and stashing the workflow-
branch tree one level deeper at .github/packaging/packaging/*. The
workflow then runs the tag's stale Taskfile instead of the workflow-
branch Taskfile — the exact compatibility path the overlay is meant
to provide.

This affects workflow_dispatch on any tag where the checkout already
contains .github/packaging/ (anything from c0a00d4 onward).

Add `rm -rf .github/packaging` before the cp so the destination does
not exist when cp runs and a clean copy lands at the intended path.
Apply to both build-rpm-release.yml and build-deb-release.yml.

Verified locally via the documented reproducer:
- Pre-fix demo earlier in the session showed the nesting bug
- Post-fix synthetic test prints "overlay sequence OK":
  no nesting, no stale files, no leftover overlay dir
NFPM_<FMT>_PASSPHRASE and PKG_GPG_KEY_FILE were interpolated directly
into a folded YAML scalar that becomes the docker run shell command.
A passphrase containing whitespace, $, !, quotes, or other shell
metacharacters would be split or expanded before docker received it —
nfpm would sign with the wrong (or empty) passphrase and signing fails.

Move to the task-env + docker pass-through pattern (the same one used
in the build-deb-release.yml S3 step earlier in the branch):

- Each task gains an env: block that promotes PKG_GPG_KEY_FILE and the
  format-specific NFPM_<FMT>_PASSPHRASE from task vars. YAML parses
  these literally — no shell interpretation.
- In docker run, replace `-e VAR={{.VAR}}` (interpolated) with `-e VAR`
  (no value). Docker reads VAR from the parent process env (set by the
  env: block).
- Replace `{{if .PKG_GPG_KEY_FILE}}-v ...:...:ro{{end}}` (Go-template
  conditional) with shell `${PKG_GPG_KEY_FILE:+-v "..."}` parameter
  expansion. mvdan.cc/sh (task's shell interpreter) supports `:+` and
  emits nothing when the variable is empty.

Applied identically across all four build tasks: build-{avalanchego,
subnet-evm}-{rpm,deb}.

The build-package.sh ephemeral-key branch already treats empty
PKG_GPG_KEY_FILE / NFPM_<FMT>_PASSPHRASE as "no key / no passphrase",
so passing the variables through unconditionally (even when empty)
preserves existing behavior.

Verified locally:
- Shell-hostile passphrase `pass with $space!` via throwaway GNUPGHOME
  → DEB pipeline completes end-to-end, `gpg --verify` reports
  "Good signature" on both .deb files in jammy and noble.
- Ephemeral-key regression check (no secrets set) → DEB pipeline still
  completes end-to-end.
@PlatCore PlatCore requested a review from a team as a code owner May 13, 2026 01:37
@PlatCore PlatCore requested a review from maru-ava May 13, 2026 02:41
@PlatCore PlatCore self-assigned this May 13, 2026
@PlatCore PlatCore added ci This focuses on changes to the CI process devinfra labels May 13, 2026
@PlatCore PlatCore linked an issue May 13, 2026 that may be closed by this pull request
2 tasks
@maru-ava
Copy link
Copy Markdown
Contributor

maru-ava commented May 13, 2026

As per my comment on #5180, I asked you to refactor the existing PR rather than creating a new one. I made this request to preserve review continuity, in keeping with our previous offline discussion in the context of your Firewood CI PR initial gpg signing PR. Can you explain why you chose to open this PR rather than continuing the review in #5180?

@JuanLeon2
Copy link
Copy Markdown
Contributor

Maru, speaking to Vlad, he thought your comment on the previous P #5180 "I think refactoring this PR would be preferable" suggested a new PR. I understand the preference for keeping PR context, is it important enough to go back to 5180 and replace it with this?

@maru-ava
Copy link
Copy Markdown
Contributor

Maru, speaking to Vlad, he thought your comment on the previous P #5180 "I think refactoring this PR would be preferable" suggested a new PR. I understand the preference for keeping PR context, is it important enough to go back to 5180 and replace it with this?

I said refactoring this (#5180) PR would be preferable. And yes, I think the work should continue in #5180 rather than here.

@PlatCore
Copy link
Copy Markdown
Contributor Author

@maru-ava

I said refactoring this (#5180) PR would be preferable.

This PR delegates DEB signing entirely to nfpm (same model as RPM), which means:

  • no custom signing shell
  • no gpg-agent
  • no gpg-preset-passphrase
  • no post-build dpkg-sig step

which is roughly about 80% of the #5180 code. While, of course, technically it is / was possible to refactor it on the original branch / PR, practically it didn't make much of a sense.

And yes, I think the work should continue in #5180 rather than here.

Could you please elaborate on it, why?

@maru-ava
Copy link
Copy Markdown
Contributor

Could you please elaborate on it, why?

As I said on #5145 and as I’ve had to repeat yet again on this PR - review continuity. Continuing on the same PR gives you an opportunity to evaluate whether the existing review comments are still relevant (e.g. regarding documentation, which is entirely absent from this PR). It also avoids requiring me as the reviewer to start again from scratch.

@JuanLeon2
Copy link
Copy Markdown
Contributor

That makes sense to me. Does that make sense to you Vlad? If not, we should align.

@PlatCore
Copy link
Copy Markdown
Contributor Author

PlatCore commented May 14, 2026

review continuity

Fair point: if this repo requires roughly 2x review time for each x unit of coding effort, it makes sense to optimize for reviewability first.

I’m OK with aligning the code with #5180 and rebasing onto it, so please hold off on reviewing until the rebase is done.

However, from the author’s perspective, I’d like to note that this approach:

  • tends to keep branches/PRs open longer
  • as branches stay open longer, their number increases because authors need to unblock themselves while waiting for code review feedback, so they pick up new tasks
  • as authors pick up new tasks, they tend to choose work from the same stream to reduce context-switching overhead; in turn, this increases the probability of code overlap or conflicts, which increases branch/PR maintenance time
  • shifts the burden of maintaining long-lived branches/PRs onto the author
  • creates many open loops/PRs, which tend to drain authors’ focus and increase communication overhead because of the aforementioned conflicts

My intuition is that the optimum is, as usual, somewhere in the middle. It would be useful to discuss this approach and maybe adjust the current cost function.

@PlatCore PlatCore added the DO NOT MERGE This PR must not be merged in its current state label May 14, 2026
@PlatCore
Copy link
Copy Markdown
Contributor Author

The changes were rebased to the #5180

@PlatCore PlatCore closed this May 14, 2026
@github-project-automation github-project-automation Bot moved this to Done 🎉 in avalanchego May 14, 2026
@JuanLeon2
Copy link
Copy Markdown
Contributor

JuanLeon2 commented May 15, 2026 via email

@JuanLeon2
Copy link
Copy Markdown
Contributor

This work is continued in PR #5180

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci This focuses on changes to the CI process devinfra DO NOT MERGE This PR must not be merged in its current state

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

3 participants