[ci] Add GPG signing for DEB packages#5145
Conversation
Replace the old unsigned dpkg-deb workflow with nfpm-based DEB packaging that includes GPG signing, mirroring the existing RPM signing approach. Adds subnet-evm DEB package for parity with RPM.
nfpm does not expand ${VAR} in top-level fields like changelog and
signature.key_file. Preprocess configs with envsubst before passing
them to nfpm. Add gettext to RPM builder Dockerfile for envsubst.
nfpm defaults to debsign method which produces signatures incompatible with dpkg-sig --verify. Set method explicitly to dpkg-sig so signing and verification use the same format.
nfpm's Go openpgp inline signatures are incompatible with dpkg-sig --verify. Remove nfpm deb.signature config and sign post-build with dpkg-sig itself so the same tool signs and verifies.
dpkg-sig 0.13.1 (Ubuntu 22.04) predates zstd support in dpkg and cannot verify signatures on zstd-compressed .deb archives, causing BADSIG. Switch to gzip for dpkg-sig compatibility.
maru-ava
left a comment
maru-ava
left a comment
There was a problem hiding this comment.
This is a good start. My original expectation was that this would mainly add the signing already used for the RPM package builds, but it goes further by moving DEB packaging toward the same standard as RPM in both signing and testing.
That broader effort has also introduced a couple of concerns:
- Noble-specific coverage appears to have been removed. The old flow had separate noble builds/tests, but the new flow publishes to noble paths without building or smoke-testing in a noble environment.
- The new DEB build/test path is not locally reproducible in the same way as the RPM path, because it still depends on Linux host tooling and validation assumptions. A developer on macOS cannot exercise it locally in the same straightforward way. If the goal is to move DEB toward the RPM model, local reproducibility seems like a reasonable bar.
Would it make sense to split this into two parts? One option would be to first refactor the RPM workflows for reuse, with no intended functional change, and then build the DEB flow on top of that. That would reduce duplication, make the review easier, and help keep the RPM and DEB paths aligned over time.
| build/deb/DEB-GPG-KEY-avalanchego | ||
|
|
||
| - name: Configure AWS credentials | ||
| if: github.event_name != 'pull_request' && startsWith(github.ref, 'refs/tags/') |
There was a problem hiding this comment.
The github.event_name != 'pull_request' part makes sense, but gating this on startsWith(github.ref, 'refs/tags/') means the upload path will never run on workflow_dispatch, even when a tag is explicitly provided via github.event.inputs.tag. If manual dispatch is only intended to build and validate, that seems fine. But if the intent is for a manually supplied tag to exercise the release/upload path as well, this condition does not do that.
There was a problem hiding this comment.
this condition does not do that.
Do you mean it prevents it or it doesn't capture / trigger on it?
I'm not sure that I clearly / fully understand the "manual" scenario when you'd like it to be triggered upon, mb point me to some docs?
| env: | ||
| PACKAGING_TAG: ${{ env.TAG }} | ||
| DEB_GPG_KEY_FILE: ${{ env.GPG_KEY_FILE }} | ||
| NFPM_DEB_PASSPHRASE: ${{ secrets.RPM_GPG_PASSPHRASE }} |
There was a problem hiding this comment.
I may be missing it, but I don’t see the passphrase being used anywhere after NFPM_DEB_PASSPHRASE is passed into the job. If the release key is passphrase-protected, what ensures that key import and dpkg-sig signing work non-interactively in the release path? My concern is that this may work in PR/testing with an unprotected ephemeral key, while the real release key path remains unexercised.
There was a problem hiding this comment.
My assumption was that the signing key/s are non-interactive. Good catch! Addressed in the commit 83b1975
| "s3://${{ secrets.BUCKET }}/linux/debs/ubuntu/${release}/" | ||
| done | ||
|
|
||
| - name: Cleanup |
There was a problem hiding this comment.
build-deb.sh copies the private signing key into the workspace at build/gpg/signing-key.asc, but this cleanup step only removes the original temp key file and build/deb. Should build/gpg also be removed here? As written, the private key appears to remain in the workspace after the job completes.
There was a problem hiding this comment.
It looks like the RPM w/f is lacking this kind of clean-up in the same way.
dpkg-sig delegates to gpg which needs the passphrase from gpg-agent. Configure allow-preset-passphrase and use gpg-preset-passphrase to cache the release key passphrase so signing works non-interactively. Use gpgconf --list-dirs libexecdir for portable path resolution.
build-deb.sh and build-rpm.sh copy the private signing key to build/gpg/signing-key.asc, but cleanup only removed the temp file and package output dirs. Add build/gpg to both DEB and RPM cleanup.
The S3 upload was gated on startsWith(github.ref, 'refs/tags/') which excluded workflow_dispatch (where github.ref is a branch ref). Allow upload on tag push OR manual dispatch to match old behavior.
The RPM Docker container creates build/gpg/ as root via bind mount. The host runner cannot remove root-owned files without sudo.
The old workflows built and tested on both Ubuntu 22.04 and 24.04 separately. Restore noble coverage by running the validation smoke test in both ubuntu:22.04 and ubuntu:24.04 containers to confirm glibc forward compatibility of the jammy-built binary.
dpkg-sig was removed from Ubuntu 24.04 (noble) repositories. Split validation into signature verification (jammy only, where dpkg-sig exists) and install + smoke test (both jammy and noble). The signature is embedded in the .deb and does not vary by release.
|
Request for refactoring into two PRs has been addressed, see |
|
I thought I had made it clear that this PR should evolve to be one of the 2 parts asked for. That way the review context gets retained as part of what gets merged rather than orphaned as is now the case. |
2 participants
Why this should be merged
DEB packages for avalanchego are currently built unsigned, while RPM packages already have full GPG signing via nfpm. Unsigned packages cannot be verified by users, leaving a gap in the supply chain security posture. This PR adds GPG signing to DEB packages using the RPM approach as a blueprint, and also adds the subnet-evm DEB package for parity.
How this works
Mirrors the existing RPM signing pipeline (
build-rpm-release.yml+build-rpm.sh) for DEB packages:dpkg-deb --buildprocess — uses nfpm configs for both avalanchego and subnet-evmdpkg-sig— nfpm builds the unsigned.deb, thendpkg-sig --signsigns it. This is necessary because nfpm's Goopenpgpinline signatures are incompatible withdpkg-sig --verifybuild-deb-release.yml) with matrix strategy for amd64 + arm64, replacing the two separate unsigned workflowssecrets.RPM_GPG_PRIVATE_KEYfor release builds; generates ephemeral keys for PR smoke testsvalidate-deb.shruns in a freshubuntu:22.04container — imports public key, runsdpkg-sig --verify, installs packages, and smoke-tests--versionoutputtest-build-debstask orchestrates build + validation end-to-end${VAR}in top-level config fields (changelog, signature), so build scripts preprocess configs withenvsubstbefore calling nfpmKey changes:
avalanchego-deb.yml/subnet-evm-deb.ymlnfpm configs (install to/usr/local/binand/usr/local/lib/avalanchego/plugins/, gzip compression fordpkg-sigcompatibility)build-deb.sh— builds binary, sets up GPG, packages with nfpm, signs withdpkg-sigvalidate-deb.sh— verifies signature + installs + smoke-tests in fresh Ubuntu containerbuild-deb-release.yml— triggers onv*tags,workflow_dispatch, and PRs touching.github/packaging/**NFPM_CHANGELOGandNFPM_SIGNING_KEYpaths in RPM nfpm configs via envsubst (non-breaking)gettextto RPM builder Dockerfile for envsubstbuild-ubuntu-amd64-release.yml,build-ubuntu-arm64-release.yml,build-deb-pkg.sh,debian/template/control)How this was tested
.github/packaging/**and runs the fulltest-build-debspipeline with ephemeral GPG keys (signature verification + install + smoke test in a fresh Ubuntu container)workflow_dispatchwith a test tag to verify real-key signing and S3 uploadexportadded to existing variables)Need to be documented in RELEASES.md?
No