ENT-14361, ENT-14362: Preparing build scripts for running build-in-container in Jenkins - #2395
Open
larsewi wants to merge 20 commits into
Open
ENT-14361, ENT-14362: Preparing build scripts for running build-in-container in Jenkins#2395larsewi wants to merge 20 commits into
larsewi wants to merge 20 commits into
Conversation
|
Thank you for submitting a PR! Maybe @craigcomstock can review this? |
larsewi
marked this pull request as draft
August 6, 2026 11:33
be57fc6 computed $my_dir at the top of the script, and 1f25dbc relied on it for the linux-install-protobuf.sh and linux-install-rust.sh calls. dbc1efb then moved the computation into the centos-7 branch so hosts without realpath would not abort under set -e, which left $my_dir empty for those two calls everywhere else. Resolved it once at the top again, calling realpath only when available so those hosts still get through. The fail-fast is gone with it: a relative dirname is enough to source a neighbouring script. Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
larsewi
force-pushed
the
container-packages-build-host
branch
3 times, most recently
from
August 6, 2026 12:22
3164a47 to
a58b1e2
Compare
larsewi
changed the title
larsewi
force-pushed
the
container-packages-build-host
branch
from
August 6, 2026 14:19
ee29f15 to
adeb404
Compare
fix-buildhost.sh had the cache host key inline, and the containerized build (ENT-14361) needs the same keys. setup-ci-host.sh keeps its own github-known-hosts, reading the keys from that file rather than repeating them. Also pins all three key types for the cache host rather than only ed25519, since which one is used is the client's choice. Ticket: ENT-14330 Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Hosts for the build-in-container job need Docker with BuildKit, which no build host has today. Ticket: ENT-14361 Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Ticket: ENT-14362 Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
autogen only writes a revision file when one is absent, and rsync brought the host's along, so a leftover from an earlier build decided the dependency cache key. Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Ticket: ENT-14361 Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
… burden Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Building several platforms into one output directory mixed their packages together. Keep them apart the way testing-pr does, which copies each cell's artifacts to upload/$label. The directory is emptied first, since the label says nothing about the project or the build type, and yesterday's packages should not look like this build's output. Ticket: ENT-14361 Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Silences the linter. Every call here inspects returncode itself, so check=False is what they were already doing implicitly. Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Silences the linter. The tags are non-empty by the check above, so the two are equivalent. Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
bootstrap-pr builds these once and testing-pr only unpacks and republishes them, so each platform label already publishes identical copies. Building them again per platform added nothing, and left it ambiguous which of them we would ship, since "make dist" embeds the generated build system and so differs with the image's autotools. Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
The tarballs are the same whichever platform builds them, so they get an image of their own rather than being built by each platform. Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
--output-dir defaults to ./output, which lands inside buildscripts, so rsync carried an earlier build's packages back in and the collector at the end of the inner script copied them out again as if this build had made them. Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Builds the core and masterfiles tarballs, and nothing else. They land in <output-dir>/tarballs, beside the per-label package directories, as they belong to no platform. Each tarball's timestamps come from its own repository's last commit, so one stays identical until its own sources change. bootstrap-tarballs was not reused for this: it does a good deal besides building tarballs, and it overwrites the timestamps with a fixed date. Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
The tarballs got one and the packages did not. Factored it out so both list their output the same way, and sorted the list so it comes out the same for the same set of files. Ticket: ENT-14361 Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Two builds of the same source produced different tarballs: the mtimes came from whenever "make install" had run, the member order was however the directory happened to be walked, the owner was whoever built it, and gzip recorded the time it ran. Sorting the file list ourselves stands in for --sort=name, which needs a newer tar than centos-7 has. The options core and masterfiles pass for their tarballs are otherwise deliberately not all repeated here: --format=posix would change an archive that users unpack by hand on old platforms, and --mode would hand them permissions install never gave. Ticket: ENT-14361 Changelog: The generic tar package is now reproducible Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
The revision came from the last commit touching deps-packaging as a whole, so bumping one dependency changed the cache key of all of them. Furthermore, the timestamps embedded in the packages came from core's last commit, so they moved whenever core did. revision-file now writes a revision per dependency, holding the hash and the time of the last commit touching that dependency or the scripts that build them all. install-dependencies reads it for both the cache key and SOURCE_DATE_EPOCH. Hence, a dependency's contents can only change when its key changes. Ticket: ENT-14362 Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
larsewi
force-pushed
the
container-packages-build-host
branch
from
August 6, 2026 14:47
adeb404 to
11d0f72
Compare
Contributor
Author
|
@cf-bottom Jenkins please :) |
|
Sure, I triggered a build: Jenkins: https://ci.cfengine.com/job/pr-pipeline/14412/ Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-14412/ |
larsewi
marked this pull request as ready for review
August 6, 2026 15:09
git gained -C in 1.8.5 and rhel-7 has 1.8.3.1, so deriving SOURCE_DATE_EPOCH from core, and every revision file, died with "Unknown option: -C" as soon as a build ran there. cd in a subshell instead, which every version understands. revision-file had not hit this because autogen only runs on the bootstrap host, which has a newer git; testing-pr unpacks tarballs and reads the revision files that host generated. Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
It is a supported platform for the next release and was missing. Named rhel-7 rather than centos-7 so the cache label comes out as PACKAGES_x86_64_linux_redhat_7 and shares testing-pr's entries. Its own Dockerfile, since Dockerfile.rhel is dnf throughout and centos 7 has yum, no modules and no CRB. There is no Rocky 7, so the base is centos:7, with everything from vault.centos.org now that it is EOL, and devtoolset-11 for a compiler newer than 4.8.5. Everything runs through scl so the build scripts get that compiler without a login shell to source it. Agent only: nothing builds PACKAGES_HUB_x86_64_linux_redhat_7, and Node 20 wants a newer glibc than this has, so the hub toolchain is left out. Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
aleksandrychev
left a comment
aleksandrychev
left a comment
Contributor
There was a problem hiding this comment.
A few blockers from reading through the diff.
Comment on lines
+29
to
+30
| tar --no-recursion --numeric-owner --owner=0 --group=0 \ | ||
| -cf - -T "$_tarball.filelist" | gzip --no-name >"$_tarball" |
Contributor
There was a problem hiding this comment.
tar | gzip throws away tar's exit status — POSIX sh has no pipefail, so set -e only ever sees gzip's 0. The old tar czvf aborted the build when tar failed; now a truncated tarball gets a sha256 written for it and ships as good.
Writing the tar to a temp file and gzipping it as a second command would keep the failure fatal.
| if [ -f /run/secrets/sftp-cache-key ]; then | ||
| echo "Installing dependency cache key..." | ||
| install -d -m 700 "$HOME/.ssh" | ||
| install -m 600 /run/secrets/sftp-cache-key "$HOME/.ssh/id_rsa" |
Contributor
There was a problem hiding this comment.
This runs as the unprivileged builder, but the source is a 0600 file on a read-only mount owned by the host user — so if the host uid differs from builder's, install can't read it and the build dies here (or at the sftp probe below). Every other mount in this script gets a sudo chown for exactly this reason; this one doesn't.
sudo install -m 600 -o "$(id -u)" -g "$(id -g)" ... would cover it.
Comment on lines
+389
to
+396
| subdir = "tarballs" if args.tarballs else label | ||
| output_dir = Path(args.output_dir).resolve() / subdir | ||
| cache_dir = Path(args.cache_dir).resolve() | ||
|
|
||
| # Start from an empty directory so that packages left by an earlier build, | ||
| # of another project or build type, cannot be taken for this build's output. | ||
| if output_dir.exists(): | ||
| shutil.rmtree(output_dir) |
Contributor
There was a problem hiding this comment.
label encodes platform and role, but not project or build type — so a nova build and a community build of the same platform resolve to the same subdir, and the rmtree means whichever runs second deletes the first one's packages.
The comment names "another project or build type" as the case being guarded against, but those are the ones that need to stay side by side. Folding args.project/args.build_type into the subdir would fix both.
| label = ( | ||
| None | ||
| if args.tarballs | ||
| else cache_label(args.platform, args.role, args.arch or host_docker_arch()) |
Contributor
There was a problem hiding this comment.
host_docker_arch() is the host's arch, not the one being built. With no --arch given, an amd64-only platform built on an arm64 host gets PACKAGES_arm_64_... — wrong buildcache namespace, and wrong output subdir. The arch the container actually runs as is what this wants.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.