[ci] Add GPG-signed DEB packages#5180
Conversation
PlatCore
force-pushed
the
PlatCore/5109-add-deb-gpg-signing-v2
branch
3 times, most recently
from
c6cdc18 to
84aa1f4
Compare
maru-ava
left a comment
maru-ava
left a comment
There was a problem hiding this comment.
I'm intentionally keeping this review narrow until 5179 is ready. That said, there is at least one DEB-specific correctness issue here that seems worth calling out now: the release-signing path appears to rely on passphrase handling that PR CI does not exercise, and the current gpg-preset-passphrase invocation does not look correct.
| | awk -F: '$1 == "grp" { print $10 }') | ||
| local kg | ||
| for kg in ${keygrips}; do | ||
| echo "${NFPM_DEB_PASSPHRASE}" | "${preset_pass}" --preset "${kg}" |
There was a problem hiding this comment.
This looks like the DEB release-signing hole.
PR/local builds can pass here because the ephemeral key is unprotected, so dpkg-sig never needs the gpg-agent preset path. But release builds use a protected key, and gpg-preset-passphrase expects the passphrase via --passphrase/-P, not on stdin. As written, this does not appear to preload the passphrase successfully, which would leave the tagged/manual DEB signing path unvalidated and likely broken.
There was a problem hiding this comment.
The stdin path is correct IMO. gpg-preset-passphrase reads the passphrase from stdin when --preset is used without -P/--passphrase:
Source: https://www.gnupg.org/documentation/manuals/gnupg/Invoking-gpg_002dpreset_002dpassphrase.html
--preset
Preset a passphrase. This is what you usually will use. gpg-preset-passphrase will then read the passphrase from stdin.
The pipe form (echo "$PASSPHRASE" | gpg-preset-passphrase --preset "$KG") also avoids exposing the passphrase in the process argument list, which -P would ... or did I mis/-read/-understand your concern?
There was a problem hiding this comment.
Tested the DEB signing path w/ a faked
- gpg key chain (temp.dir.) and
- a fake key
- protected by a passphrase
passed to the jammy and noble containers, signature verification check -- passed.
Does it address your concern in full this time?
There was a problem hiding this comment.
I realize now that the RPM packaging missed the opportunity to ensure a minimal divergence between release and non-release workflow execution. Thoughts on updating ephemeral key generation for both the RPM and DEB paths to use a known throwaway passphrase so that both execution paths use a password-secured key?
There was a problem hiding this comment.
I will lift it into the parent PR somewhere into the .github/packaging/scripts/lib-build-common.sh.
There was a problem hiding this comment.
This looks like the DEB release-signing hole.
PR/local builds can pass here because the ephemeral key is unprotected, so
dpkg-signever needs the gpg-agent preset path. But release builds use a protected key, andgpg-preset-passphraseexpects the passphrase via--passphrase/-P, not on stdin. As written, this does not appear to preload the passphrase successfully, which would leave the tagged/manual DEB signing path unvalidated and likely broken.
We are no longer using dpkg-sig so this is no longer relevant is this correct?
There was a problem hiding this comment.
This is not relevant anymore. nFPM is used instead.
There was a problem hiding this comment.
I will lift it into the parent PR somewhere into the
.github/packaging/scripts/lib-build-common.sh.
Vlad is going to merge this from another branch today
There was a problem hiding this comment.
I realize now that the RPM packaging missed the opportunity to ensure a minimal divergence between release and non-release workflow execution. Thoughts on updating ephemeral key generation for both the RPM and DEB paths to use a known throwaway passphrase so that both execution paths use a password-secured key?
Addressed.
- Elevated the change to the parent branch at the d267f8b
- tested locally and in the CI -- it passed.
- Updated the packaging docs to reflect the change.
Please review and resolve.
(I lost track of the countless git rebases and conflict resolutions I had to jump through; Hope it was worth it;)
PlatCore
force-pushed
the
PlatCore/5109-refactor-rpm-for-reuse-v2
branch
from
f2c029c to
402dd0c
Compare
There was a problem hiding this comment.
I might be wrong but it seems like the key is generated each time we sign? Why do we do that and not use a long term public key?
There was a problem hiding this comment.
The release builds (tag push / workflow_dispatch) always use the persistent secret key. The ephemeral path only runs for PRs and local testing.
So, you are right, just .. partially )
There was a problem hiding this comment.
to be clear (for myself) we do use a long-term key from github secrets but we do generate an ephemeral one for local testing and PRs.
There was a problem hiding this comment.
Yacov, please resolve the conversation if this addresses your comment.
There was a problem hiding this comment.
@yacovm can you resolve if you think that would be appropriate?
PlatCore
force-pushed
the
PlatCore/5109-refactor-rpm-for-reuse-v2
branch
from
402dd0c to
d336cf3
Compare
PlatCore
force-pushed
the
PlatCore/5109-add-deb-gpg-signing-v2
branch
2 times, most recently
from
5f4fb6a to
e9e97d1
Compare
PlatCore
force-pushed
the
PlatCore/5109-add-deb-gpg-signing-v2
branch
from
be87574 to
e03431d
Compare
|
DEB-specific build hooks (gpg-agent setup, passphrase caching, dpkg-sig signing) are now sourced by the unified build-package.sh.
Use detect_host_arch(), resolve_subnet_evm_vm_id(), and assert_files_exist() from shared libs, eliminating duplicated code.
DEB Taskfile tasks now call build-package.sh with FORMAT=DEB. DEB workflow uses setup-packaging composite action.
build-package.sh needs +x to run inside the container.
Move ${{ inputs.tag-input }} and ${{ inputs.gpg-private-key }} to env
blocks in setup-packaging/action.yml to prevent shell injection via
workflow_dispatch inputs.
Add libc6 (>= 2.34) dependency to DEB nfpm configs, matching the RPM
glibc requirement, so dpkg rejects installs on incompatible systems.
Changes to .github/actions/setup-packaging/ were not triggering packaging CI on pull requests, so regressions in tag resolution or GPG handling could merge without signal.
The workflow referenced a non-existent composite action (.github/actions/setup-packaging). Replace with the same task-based setup pattern the RPM workflow uses. Also add the packaging overlay steps for workflow_dispatch builds of older tags.
The setup-packaging Taskfile task was removed and replaced by workflow-setup-packaging.sh. The RPM workflow was updated but the DEB workflow still referenced the old task name.
RPM refactor. After rebasing this branch onto the post-excision tip of #5179, the hooks need to return as commits owned by #5180 so they survive the eventual merge into master. build-package.sh: source lib-build-${fmt_lower}.sh on FORMAT=DEB, add the DEB arm to the GPG_KEY_FILE case, configure gpg-agent via setup_deb_gpg_agent before setup_gpg, cache the passphrase via cache_deb_gpg_passphrase for non-interactive dpkg-sig signing, and invoke sign_deb_package post-nfpm (nfpm's openpgp signatures are incompatible with dpkg-sig --verify). lib-validate-common.sh: re-add the DEB arm of detect_host_arch's format dispatch (x86_64 -> amd64, aarch64/arm64 -> arm64). The corresponding helper functions are defined in lib-build-deb.sh. Verified locally: task -t .github/packaging/Taskfile.yml test-build-debs passes end-to-end. Both .debs built, dpkg-sig signatures verify (GOODSIG), and install + smoke test pass in fresh ubuntu:22.04 (jammy) and ubuntu:24.04 (noble) containers.
Mirrors the RPM template refactor. build-package.sh exports only
${BINARY_PATH}; the DEB templates' references to
${AVALANCHEGO_BINARY} / ${SUBNET_EVM_BINARY} resolved to empty via
envsubst, causing nfpm to fail with 'glob failed: no matching files'
on the amd64 CI run.
actionlint flagged 'custom-arm64-jammy' as an unknown label; master migrated arm64 to the GitHub-hosted ubuntu-22.04-arm runner. The DEB workflow was authored before that sweep.
nfpm now signs DEBs inline via deb.signature.key_file, matching the RPM side. Validation extracts the _gpgorigin ar member and runs gpg --verify against debian-binary+control+data, so the same flow works on jammy and noble (dpkg-sig was unavailable in noble). Builder image, build helper, Taskfile comment, and design doc no longer reference dpkg-sig.
- workflow-setup-packaging.sh now fails fast when no signing key is
provided for a tag push or workflow_dispatch (previously fell back
to ephemeral-key generation, so unsigned-by-real-key packages could
reach S3).
- build-{deb,rpm}-release.yml replace .github/packaging atomically
(rm -rf + mv) instead of `cp -r` into an existing directory, which
was leaving the checked-out tag's stale Taskfile in place and
copying the overlay to .github/packaging/packaging/.
- resolve_subnet_evm_vm_id falls back to grepping
graft/subnet-evm/scripts/constants.sh when the new
default-vm-data.sh is absent, restoring workflow_dispatch builds of
tags predating the data file.
- Taskfile.yml forwards NFPM_{RPM,DEB}_PASSPHRASE via a task-level
`env:` block and value-less `-e NFPM_*_PASSPHRASE` so passphrases
containing whitespace or shell metacharacters reach the container
intact (no shell evaluation of the secret).
Taskfile.yml: the four build-{avalanchego,subnet-evm}-{rpm,deb} tasks
collapse into one internal `build-package` task that the four wrappers
delegate to with `task: build-package` + vars. The wrappers only carry
the format/output/builder-image differences; the docker-run block lives
in one place.
build-package.sh: switch the format env var from PKG_FORMAT (RPM|DEB) to
NFPM_PACKAGER (rpm|deb) to match nfpm's CLI naming, and collapse
RPM_GPG_KEY_FILE / DEB_GPG_KEY_FILE into a single GPG_KEY_FILE since the
caller already picks the right one.
The value-less `-e NFPM_*_PASSPHRASE` passphrase forwarding survives the
refactor (verified via task --dry with a metachar-laden passphrase).
Moves VERSION (derived from TAG via trimPrefix "v") and TAG into the build-package task's env: block and switches their docker flags to the value-less `-e VAR` form, matching the pattern already used for the NFPM_*_PASSPHRASE secrets. Keeps the docker invocation free of any template-substituted values that could carry shell metacharacters.
Aligns the RPM-side var names with the DEB-side pattern (format prefix between PACKAGING_ and the rest): PACKAGING_HOST_ARCH -> PACKAGING_RPM_HOST_ARCH PACKAGING_HOST_DEB_ARCH -> PACKAGING_DEB_HOST_ARCH PACKAGING_DOCKER_IMAGE -> PACKAGING_RPM_DOCKER_IMAGE PACKAGING_OUTPUT_DIR -> PACKAGING_RPM_OUTPUT_DIR All references inside the Taskfile updated; no live consumers outside it.
Replaces docs/design/deb-packaging.md and docs/design/rpm-packaging.md with a single shared design doc per reviewer comment on PR #5180. The new doc has format-specific subsections for RPM and DEB and shared subsections for build tooling, GPG signing, version smoke test, CI workflow, and architecture mapping. Also brings the prose in line with the current code: - VM-ID source notes default-vm-data.sh with a fallback to constants.sh for older tags, matching the resolver in lib-build-common.sh. - CI workflow examples use the actual --taskfile invocation form used in build-{rpm,deb}-release.yml, with a note about the packaging: namespace available locally via the root Taskfile include. - GPG-signing section documents the fail-fast release-key gate that workflow-setup-packaging.sh enforces for tag pushes and workflow_dispatch.
The `.github/actions/setup-packaging/**` composite-action path was removed from this branch, so the pull_request.paths filter referencing it never matches. Drop it from both build-rpm-release.yml and build-deb-release.yml.
Both release workflows pipe the same `gpg-key-file` step output into
the build env, and the per-format wrappers immediately funnel that into
build-package's GPG_KEY_FILE var. The {RPM,DEB}_ prefix carried no
information.
- Taskfile: four wrappers now read .GPG_KEY_FILE directly instead of
the format-prefixed equivalents.
- build-deb-release.yml: env block sets GPG_KEY_FILE (was
DEB_GPG_KEY_FILE) to match the already-normalized RPM workflow.
- build-deb-release.yml: GPG_KEY_PASSPHRASE replaces NFPM_DEB_PASSPHRASE
(the latter was no longer read by the Taskfile; releases were silently
unsigned) and is gated on non-pull_request like the RPM side.
NFPM_RPM_PASSPHRASE / NFPM_DEB_PASSPHRASE remain *inside*
build-package.sh as the nfpm-mandated env var names; the script
re-exports GPG_KEY_PASSPHRASE under the right NFPM_<FORMAT>_PASSPHRASE
just before invoking nfpm.
The RPM builder-image task was the only build-* task that omitted the format prefix. With a parallel build-deb-builder-docker-image already in place, the RPM task is now build-rpm-builder-docker-image so every build-* task carries the same format suffix (or is the format-agnostic internal build-package). - Task definition at line 59. - deps: refs in build-avalanchego-rpm and build-subnet-evm-rpm. The underlying scripts/build-builder-image.sh stays unrenamed — it is already format-agnostic (drives off the DOCKERFILE arg).
build-package.sh sets set -euo pipefail. When the Taskfile invokes
build-package without a real signing key, the docker run omits the
`-e GPG_KEY_FILE` flag entirely (Task conditional), so the container env
is unset. The script then references ${GPG_KEY_FILE} in the ephemeral-
key branch and aborts on `unbound variable` before signing setup runs.
Default GPG_KEY_FILE to empty alongside the existing NFPM_*_PASSPHRASE
defaults; the ephemeral-key conditional handles the empty case correctly.
f946cc6 renamed PACKAGING_HOST_ARCH to PACKAGING_RPM_HOST_ARCH but missed the default chain in the validate-rpms task. With no explicit PACKAGE_ARCH (the normal CI / test-build-rpms path), PACKAGE_ARCH resolves to empty and validate-rpm.sh aborts at its `: "${PACKAGE_ARCH:?...}"` assertion -- after the packages have been built.
build-package.sh now exports the public key as
${pkg_format_upper}-GPG-KEY-avalanchego, so the RPM build produces
build/rpm/RPM-GPG-KEY-avalanchego. The two consumers still referenced
the unprefixed path:
- validate-rpm.sh checked /rpms/GPG-KEY-avalanchego and silently skipped
signature verification on signed RPMs.
- build-rpm-release.yml uploaded build/rpm/GPG-KEY-avalanchego, so the
release artifact never included the key.
Mirror the DEB side (which already uses DEB-GPG-KEY-avalanchego
end-to-end).
build-deb runs on pull_request and executes PR-controlled packaging scripts. Granting id-token: write at the job level let PR scripts request a GitHub OIDC JWT regardless of which steps actually ran. Split into two jobs: - build-deb: contents: read only. Builds, validates, and uploads artifacts on non-PR runs. Exposes the resolved tag as a job output. - upload-debs-s3: needs: build-deb; runs only on tag push or workflow_dispatch; has id-token: write. Downloads artifacts and pushes them to S3. The PR-executed job can no longer mint OIDC tokens; release uploads keep working via the new dedicated job.
resolve_subnet_evm_vm_id sources constants.sh inside SUBNET_EVM_VM_ID=$(...)
so it captures whatever stdout the sourced file produces along with the
trailing echo "${DEFAULT_VM_ID}".
Older subnet-evm constants.sh revisions (e.g., v1.14.1 line 62) do
`echo "Using branch: ${CURRENT_BRANCH}"` at source time. Under
workflow_dispatch tag-overlay builds this contaminates SUBNET_EVM_VM_ID
with a "Using branch: ..." line, which then corrupts the rendered nfpm
yaml plugin path.
Redirect the source's stdout to /dev/null; the explicit
`echo "${DEFAULT_VM_ID}"` at the end of the subshell is the only thing
captured.
PR #5179 (Thread #7, pushed as 7245fa9) shrank smoke-test.sh from 4 args to 3: the caller now passes the composed subnet-evm plugin binary path instead of the plugin dir + VM ID. validate-rpm.sh was updated in the same commit; validate-deb.sh was missed and kept the old 4-arg invocation. After rebasing onto PR #5179's tip the contract drift surfaced in build-deb-packages CI: positional-arg slip made $2 the plugins *directory*, and smoke-test.sh tried to exec it ("Is a directory", exit 126). Mirror the validate-rpm.sh fix.
4 participants
Why this should be merged
DEB packages are currently unsigned and built with an ad-hoc
dpkg-debflow. This adds GPG-signed DEB packages foravalanchegoandsubnet-evm, modelled on the existing RPM pipeline — containerized build, nfpm-native signing, end-to-end validation on both supported Ubuntu releases.Depends on #5179.
How this works
Mirrors the RPM pipeline pattern with a DEB-specific container:
Dockerfile.deb— Ubuntu 22.04 builder image (gcc, gnupg, Go, nfpm).build-package.sh+lib-build-common.sh— unified RPM/DEB build helper. Builds the binary, sets up GPG (ephemeral key for PR/local;secrets.RPM_GPG_PRIVATE_KEYfor releases), and packages with nfpm.deb.signature.key_file: "${NFPM_SIGNING_KEY}"so nfpm signs in-process (no post-build signing step). Install paths:/usr/local/bin/avalanchegoand/usr/local/lib/avalanchego/plugins/<VM_ID>.validate-deb.sh— in bothubuntu:22.04andubuntu:24.04: extracts_gpgoriginfrom the ar archive, runsgpg --verifyagainstdebian-binary + control.tar.* + data.tar.*, then installs the packages and runs the shared smoke test.Taskfile.yml— Docker-run-based DEB tasks matching the RPM task pattern.task packaging:test-build-debsworks locally on macOS.build-deb-release.yml— invokestask packaging:test-build-debs. Uploads artifacts on non-PR runs and pushes to S3 on tag push /workflow_dispatch.workflow-setup-packaging.shfails fast if the signing key secret is absent on a release event.build-ubuntu-amd64-release.yml,build-ubuntu-arm64-release.yml,build-deb-pkg.sh,debian/template/control.Key design decisions:
_gpgoriginar member) verified withgpg --verify. Works identically on jammy and noble — nodpkg-siganywhere.secrets.RPM_GPG_PRIVATE_KEY; the passphrase is forwarded into the container with a value-less-e VARflag so secrets with whitespace or shell metacharacters reachnfpmintact.How this was tested
task packaging:test-build-debslocally on macOS (Docker): both packages built and signed by nfpm,gpg --verifygood in jammy and noble containers, install + smoke test passed on both.task --dryof the build task confirming the passphrase is no longer interpolated into the docker command-line..github/packaging/**.Need to be documented in RELEASES.md?
No