sh: support building multiple kernel variants#5999
Draft
JackThomson2 wants to merge 6 commits into
Draft
Conversation
The secret hiding CI could only build a single kernel: the build scripts read kernel_url, kernel_commit_hash, kernel_config_overrides and linux_patches straight from the hiding_ci directory, so there was no way to add a second kernel without clobbering the first. Restructure the directory into a shared root plus per-variant subfolders. The root keeps everything version independent (the build scripts, the ENA helpers, the default kernel_url and a shared base_config), while each variant under kernels/<variant>/ carries its own kernel_commit_hash, linux_patches and optional kernel_url and config_overrides overrides. Both scripts now take a variant selector as their first argument and resolve all per-variant inputs from kernels/<variant>/. The repository URL falls back to the shared root default when a variant does not override it. The build merges config overrides base first then variant on top so later values win. check_override_presence resolves the effective last-wins value per option before validating, so a variant overriding a base value no longer trips a false "missing config" failure. When no variant is given and exactly one exists, the script uses it; otherwise it lists the available variants and exits. The existing patch series moves under kernels/6.18-secret-hiding/, whose pinned commit is Linux 6.18, and kernel_config_overrides becomes base_config to reflect its new shared role. Signed-off-by: Jack Thomson <jackabt@amazon.com>
The build test invoked the kernel script with a fixed path and so only exercised the single kernel. Discover the variants under resources/hiding_ci/kernels at collection time and parametrize the test over them, passing each variant to the build script. Every variant becomes its own test case and so gets independent pass/fail signal. The secret_hiding marker stays, so these builds remain excluded from the default test run. Signed-off-by: Jack Thomson <jackabt@amazon.com>
The PR pipeline emitted a single secret hiding build group covering the one kernel. Enumerate the variants under resources/hiding_ci/kernels and emit one build group per variant, selecting the matching test case with an exact -k expression. Each variant builds in its own job. The jobs run in parallel across variants and architectures, and each reports its own pass/fail in the Buildkite UI. The existing change-detection gate is unchanged: it keys on the hiding_ci path component, which still matches files under the new kernels/<variant>/ subfolders. Signed-off-by: Jack Thomson <jackabt@amazon.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## feature/secret-hiding #5999 +/- ##
======================================================
Coverage 81.08% 81.08%
======================================================
Files 279 279
Lines 31370 31370
======================================================
Hits 25436 25436
Misses 5934 5934
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Add a README describing the new hiding_ci structure: what lives at the shared root versus per-variant subfolders, how config overrides layer base first then variant on top, the <version>-<feature> naming convention, and the steps to add a new variant. It also documents how to build a variant locally with the build script and how to apply a variant's patch series to an existing tree. Signed-off-by: Jack Thomson <jackabt@amazon.com>
Add a second kernel variant built from the Amazon Linux linux-6.18.y branch. It overrides kernel_url to point at the amazonlinux/linux repository and pins the branch tip, which is Linux 6.18.36. The variant ships no linux_patches yet, so it builds a stock Amazon Linux 6.18 tree with only the shared base_config applied. This exercises the new variant mechanism end to end against a different repository and gives us an unpatched baseline to compare the secret hiding kernel against. Signed-off-by: Jack Thomson <jackabt@amazon.com>
JackThomson2
force-pushed
the
feat/dual_kernel_build
branch
from
4a71953 to
956e420
Compare
Secret-hiding PR builds schedule every hiding kernel variant when any file under resources/hiding_ci changes. A patch-only change for one variant should not wait for unrelated kernel builds. Select affected variants in the Buildkite generator instead. Changes under kernels/<variant>/ now schedule only that variant, while shared hiding_ci inputs keep scheduling all variants. Signed-off-by: Jack Thomson <jackabt@amazon.com>
JackThomson2
force-pushed
the
feat/dual_kernel_build
branch
from
027c766 to
a5bad4b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Creating a script to build and install a modified kernel with patches
applied.
Signed-off-by: Jack Thomson jackabt@amazon.com## Changes
...
Reason
...
License Acceptance
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.PR Checklist
tools/devtool checkbuild --allto verify that the PR passesbuild checks on all supported architectures.
tools/devtool checkstyleto verify that the PR passes theautomated style checks.
how they are solving the problem in a clear and encompassing way.
in the PR.
CHANGELOG.md.Runbook for Firecracker API changes.
integration tests.
TODO.rust-vmm.The secret hiding CI could only build a single kernel: the build scripts
read kernel_url, kernel_commit_hash, kernel_config_overrides and
linux_patches straight from the hiding_ci directory, so there was no way
to add a second kernel without clobbering the first.
Restructure the directory into a shared root plus per-variant
subfolders. The root keeps everything version independent (the build
scripts, the ENA helpers, the default kernel_url and a shared
base_config), while each variant under kernels// carries its
own kernel_commit_hash, linux_patches and optional kernel_url and
config_overrides overrides.
Both scripts now take a variant selector as their first argument and
resolve all per-variant inputs from kernels//. The repository
URL falls back to the shared root default when a variant does not
override it. The build merges config overrides base first then variant
on top so later values win. check_override_presence resolves the
effective last-wins value per option before validating, so a variant
overriding a base value no longer trips a false "missing config"
failure. When no variant is given and exactly one exists, the script
uses it; otherwise it lists the available variants and exits.
The existing patch series moves under kernels/6.18-secret-hiding/, whose
pinned commit is Linux 6.18, and kernel_config_overrides becomes
base_config to reflect its new shared role.
Signed-off-by: Jack Thomson jackabt@amazon.com