SRE-3703 ci: Fault injection testing stage on VM/bare metal by grom72 · Pull Request #17953 · daos-stack/daos

grom72 · 2026-04-09T06:02:13Z

This PR introduces logic that simplifies the Fault Injection testing stage in CI (the Jenkinsfile)
by moving it from a Docker container environment to a provisioned VM/bare metal environment.

Requires:

https://github.com/daos-do/ansible-lab/pull/153 (for KVM setup via ansible)
or
https://github.com/daos-do/toolbox/pull/28 (for manual KVM update, requires already VMs with 27GiB reserved)
https://github.com/daos-stack/system-pipeline-lib/pull/23
SRE-3704 ci: adapt NLT pipeline to cover Fault Injection testing procedure. pipeline-lib#516
SRE-3774 ci: Disable maldet on CI nodes #18203
or
https://github.com/daos-do/packer-images/pull/24

Background

The old Fault injection testing stage ran NLT fault injection inside a Docker container
(docker_runner_fi + Dockerfile.el.9) on a shared Jenkins agent host. Up to 10 FI
containers could execute simultaneously on the same host alongside other CI workloads,
resulting in severe CPU and network resource contention. The symptoms were well-documented:
RPC timeouts, SWIM protocol failures to make progress, and "Sluggish EC boundary" warnings —
all caused by infrastructure overload rather than real code defects.

Additionally, nlt_server.yaml had ABT_STACK_OVERFLOW_CHECK=mprotect set, which causes Argobots to issue mprotect() calls for ULT stack overflow detection. On KVM-based VMs, each such call triggers TLB shootdown IPIs across all vCPUs, making test execution significantly slower on VMs than on bare metal or inside Docker containers where this overhead is less pronounced. This was a known cause of very long and unpredictable FI test execution times when running on VMs. Now that the stage runs on a dedicated provisioned VM with proper resources, ABT_STACK_OVERFLOW_CHECK=mprotect is removed from nlt_server.yaml, restoring test execution duration comparable to bare metal.

Two workarounds were introduced to mask this instability:

PR DAOS-623 test: add allowed error for FI #17959 (DAOS-623 test: add allowed error for FI, commit e0fd4e3):
added skip_substrings filters in node_local_test.py and cart_logtest.py to suppress
SWIM/network-related error conditions ("sluggish ec boundary report from rank",
"sluggish stable epoch reporting", "progress callback was not called for too long",
"rpc failed; rc:") that were firing due to Docker resource contention.
PR DAOS-623 test: ignore the server errors in client FI tests too #17999 (DAOS-623 test: ignore the server errors in client FI tests too):
extended the same suppression to server-side errors seen in NLT client FI runs.

Both PRs were explicitly described as temporary workarounds, with the expectation that they
would be reverted once FI testing was moved to a dedicated, stable environment. This PR
delivers that fix and reverts both workarounds (e0fd4e3 / #17959 and #17999), restoring
full error checking in node_local_test.py and cart_logtest.py.

Solution

The NLT Fault Injection testing stage now runs on a dedicated provisioned VM
(CI_FI_1_LABEL, default ci_fi_vm1) using the same unitTest/unitTestPost pipeline
procedures as the NLT and Unit Test stages. This mirrors how NLT tests have always been
run — on bare metal/VM nodes exclusively allocated for that purpose — and brings the same
benefits to FI testing:

Dedicated CPU and memory resources (provisioned with at least 20 cores via
VM_CPUS=20 in pipeline-lib) eliminate the resource contention that caused SWIM
and RPC failures. With 20+ cores, AllocFailTest.launch() can run FI tests in
parallel (max_child = 15) instead of the forced serial mode (max_child = 1)
that occurred when the Docker container saw fewer than 20 vCPUs.
ABT_STACK_OVERFLOW_CHECK=mprotect is removed from utils/nlt_server.yaml,
eliminating the cascading TLB shootdown IPIs that occurred when multiple FI
containers ran simultaneously on a shared KVM host.
Predictable and stable execution duration — no more variance caused by competing
Docker containers on a shared host.
Pre-built RPMs are installed rather than building from source inside the container,
removing a full SCons build from the critical path and significantly reducing
stage runtime.
Full error checking is restored in node_local_test.py and cart_logtest.py;
the skip_substrings suppression introduced in DAOS-623 test: add allowed error for FI #17959 and DAOS-623 test: ignore the server errors in client FI tests too #17999 is removed.
Log collection, JUnit reporting, and issue tracking use the standard unitTestPost
path, consistent with all other test stages.

    stage('NLT Fault Injection testing') {
        agent { label params.CI_FI_1_LABEL }
        steps {
            job_step_update(
                unitTest(timeout_time: 240,
                         inst_repos: daosRepos(),
                         test_script: 'ci/unit/test_nlt.sh --memcheck no' +
                                      ' --system-ram-reserved 4 --server-debug WARN' +
                                      ' --log-usage-import nltr.json' +
                                      ' --log-usage-save nltr.xml' +
                                      ' --class-name fault-injection fi',
                         unstash_opt: true,
                         unstash_tests: false,
                         inst_rpms: unitPackages(target: 'el9') + ' daos-client-tests',
                         image_version: 'el9.7'))
        }
        post {
            always {
                unitTestPost artifacts: ['nlt_logs/'], ...
                archiveArtifacts artifacts: 'nlt_logs/fault-injection/', allowEmptyArchive: true
                job_status_update()
            }
        }
    }

The stage is renamed from Fault injection testing to NLT Fault Injection testing to
avoid confusion with the existing Fault injection testing stage and to enable detection
in parseStageInfo / skipStage in pipeline-lib.

Jenkinsfile:

Replace the Fault injection testing stage (Docker build + nlt_test()) with the new
NLT Fault Injection testing stage running on a provisioned VM via unitTest.
Remove the nlt_test() helper function entirely — its logic is now handled by
unitTest/unitTestPost in pipeline-lib.
Add the CI_FI_1_LABEL parameter (ci_fi_vm1) for the new FI VM pool; rename
CI_NLT_1_LABEL default from ci_nlt_1 to ci_nlt_vm1.
Remove the fault-inject-valgrind stash from valgrindReportPublish — FI runs
with --memcheck no and produces no memcheck XML.

ci/docker_nlt.sh:

Delete the file — the FI stage no longer builds in Docker; it runs on a provisioned VM
via the standard unitTest path.

ci/provisioning/post_provision_config_common_functions.sh:

Stop and disable maldet on provisioned nodes; maldet scans add CPU load during NLT tests.

ci/unit/test_nlt.sh:

Replace ssh -tt + inline heredoc execution with ssh -T … bash -s -- $* piping
test_nlt_node.sh over stdin, so that command-line arguments ($*) are forwarded
correctly to test_nlt_node.sh (required for the --memcheck no --class-name fault-injection fi arguments passed by the FI stage).

ci/unit/test_nlt_node.sh:

Remove sudo mkdir -p /mnt/daos (no longer needed on provisioned VMs).
Accept forwarded arguments via $*; default to the original NLT run parameters when
no arguments are given, making the script reusable for both plain NLT and FI.
Mount a tmpfs on nlt_logs/ and set TMPDIR to it before executing
node_local_test.py, so NLT log files land on a fast in-memory filesystem.
Use exec env to set HTTPS_PROXY/NO_PROXY cleanly.

ci/unit/test_nlt_post.sh:

Add a second rsync pass to also collect logs from build/nlt_logs/ on the node
(NLT with --no-root writes logs there instead of /tmp/).
Make both rsync calls non-fatal (|| true) so post steps do not fail on missing
log directories.

utils/nlt_server.yaml:

Remove ABT_STACK_OVERFLOW_CHECK=mprotect from engine env_vars; the mprotect-based
ULT stack overflow detection is no longer needed and was a source of TLB shootdown
overhead on shared KVM hosts.

utils/node_local_test.py:

Remove the skip_substrings workaround block (revert of DAOS-623 test: add allowed error for FI #17959 / e0fd4e3 and
DAOS-623 test: ignore the server errors in client FI tests too #17999): "sluggish ec boundary report from rank", "sluggish stable epoch reporting",
"progress callback was not called for too long", "rpc failed; rc:" are no longer
suppressed — these conditions should not occur on a dedicated VM.
Add fallback fault_status detection: if the initial detection fails, try fault_status
on $PATH and then /usr/bin/fault_status before giving up, improving robustness when
the binary is installed via RPM rather than built in-tree.

src/tests/ftest/cart/util/cart_logtest.py:

Remove self.skip_substrings = [] and the associated substring-suppression check block
(revert of the cart_logtest.py portion of DAOS-623 test: add allowed error for FI #17959 / DAOS-623 test: ignore the server errors in client FI tests too #17999), restoring full log
error detection.

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2026-04-09T06:02:31Z

Errors are Unable to load ticket data
https://daosio.atlassian.net/browse/SRE-3703

daosbuild3 · 2026-04-16T13:36:57Z

Test stage Test RPMs on EL 9.6 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17953/56/execution/node/424/log

daosbuild3 · 2026-04-16T14:01:44Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17953/56/display/redirect

daosbuild3 · 2026-04-16T14:01:52Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17953/56/display/redirect

daosbuild3 · 2026-04-20T07:49:02Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17953/62/execution/node/369/log

daosbuild3 · 2026-04-20T14:55:19Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17953/69/testReport/

daosbuild3 · 2026-04-21T15:15:28Z

Test stage NLT completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17953/76/testReport/

daosbuild3 · 2026-05-04T09:49:53Z

Test stage NLT Fault injection testing completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17953/90/execution/node/333/log

daosbuild3 · 2026-05-04T09:50:23Z

Test stage NLT completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17953/90/execution/node/346/log

daosbuild3 · 2026-05-04T09:54:43Z

Test stage NLT Fault injection testing completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17953/91/execution/node/346/log

daosbuild3 · 2026-05-04T09:54:56Z

Test stage NLT completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17953/91/execution/node/344/log

daltonbohning · 2026-05-14T13:38:41Z

                    } // post
                } // stage('Functional on Ubuntu 20.04')
-                stage('Fault injection testing') {
+                stage('NLT Fault injection testing') {


FYI this will require an update to merge requirements

Restore original name in a82e340

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true Skip-func-test-leap15: true

pipeline-lib now supports overriding NLT/FI defaults (always_script, testResults, valgrind_pattern, with_valgrind, NLT, FI) via the config map, taking priority over the values auto-detected from the stage name by parseStageInfo. Make the Jenkinsfile stages explicit to take advantage of this and to make the stage configuration self-documenting. NLT stage (unitTest call): - Add with_valgrind: 'memcheck', valgrind_pattern: '*memcheck.xml', always_script: 'ci/unit/test_nlt_post.sh', testResults: 'nlt-junit.xml' NLT stage (unitTestPost call): - Remove always_script (now passed to unitTest above) - Add NLT: true to explicitly activate the NLT post-processing block (recordIssues, discoverGitReferenceBuild) instead of relying on stage name detection - Add valgrind_pattern: '*memcheck.xml' for the valgrind_stash NLT Fault injection testing stage (unitTest call): - Add always_script: 'ci/unit/test_nlt_post.sh', testResults: 'nlt-junit.xml' - Add with_valgrind: '' to explicitly suppress valgrind for FI NLT Fault injection testing stage (unitTestPost call): - Replace always_script with FI: true to explicitly activate fault injection post-processing (nlt-client-leaks.json, 'Fault injection' naming, discoverGitReferenceBuild) instead of relying on the now- removed stage name auto-detection of FI in parseStageInfo Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true Skip-func-test-leap15: true

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true Skip-func-test-leap15: true

NLT in stage name is no longer needed as required information is transfered via parameters of unitTest and unitTestPost procedures. Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-python-bandit: true Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true Skip-func-test-leap15: true

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true Skip-func-test-leap15: true

daltonbohning

LGTM

daltonbohning · 2026-05-18T16:50:33Z

 // To use a test branch (i.e. PR) until it lands to master
 // I.e. for testing library changes
 //@Library(value='pipeline-lib@your_branch') _
+@Library(value=['pipeline-lib@grom72/SRE-3704']) _


To be removed before landing

…Test-FI Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true Skip-func-test-leap15: true

phender

Looks good

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Priority: 2 Cancel-prev-build: false Skip-unit-test: true Skip-unit-test-memcheck: true Skip-func-vm-all: true Skip-test-el-9-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test: true Skip-build-el8-gcc: true Skip-build-leap15-gcc: true Skip-func-test-el9: true Skip-func-test-leap15: true Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

grom72 · 2026-05-21T16:24:04Z

Final validation is here: https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17953/217/pipeline-overview/

grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch 4 times, most recently from 276641f to 23827b4 Compare April 13, 2026 07:37

github-advanced-security AI found potential problems Apr 13, 2026

View reviewed changes

Comment thread utils/docker/Dockerfile.el.9 Fixed

Comment thread utils/docker/Dockerfile.el.9 Fixed

grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch 3 times, most recently from e724b71 to 14b4ae9 Compare April 16, 2026 13:25

grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch 2 times, most recently from eea6d40 to 8bf0a15 Compare April 20, 2026 07:48

grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch 3 times, most recently from 3591870 to fdc56a7 Compare April 20, 2026 14:09

grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch 3 times, most recently from dd13687 to 07365a4 Compare April 21, 2026 13:00

grom72 force-pushed the grom72/SRE-3704-CI-Test-FI branch 3 times, most recently from 94807ff to df82b57 Compare April 29, 2026 13:54

grom72 requested a review from phender May 14, 2026 11:42

daltonbohning previously approved these changes May 14, 2026

View reviewed changes

grom72 changed the title ~~SRE-3704 ci: Fault injection testing stage on VM/bare metal~~ SRE-3703 ci: Fault injection testing stage on VM/bare metal May 14, 2026

grom72 marked this pull request as ready for review May 14, 2026 15:02

grom72 requested review from a team as code owners May 14, 2026 15:02

grom72 dismissed stale reviews from daltonbohning and janekmi via 97ca76e May 14, 2026 15:16

janekmi previously approved these changes May 14, 2026

View reviewed changes

grom72 dismissed janekmi’s stale review via 14bfc4e May 14, 2026 17:42

grom72 mentioned this pull request May 14, 2026

SRE-3703 ci: Fault injection testing stage on VM #18252

Draft

6 tasks

grom72 added 5 commits May 15, 2026 13:22

grom72 requested a review from daltonbohning May 18, 2026 13:04

daltonbohning reviewed May 18, 2026

View reviewed changes

grom72 requested a review from daltonbohning May 20, 2026 13:28

phender reviewed May 20, 2026

View reviewed changes

grom72 requested review from janekmi and phender May 21, 2026 13:35

daltonbohning approved these changes May 21, 2026

View reviewed changes

phender approved these changes May 21, 2026

View reviewed changes

grom72 requested a review from a team May 22, 2026 19:38

Conversation

grom72 commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Solution

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

daosbuild3 commented Apr 16, 2026

Uh oh!

daosbuild3 commented Apr 16, 2026

Uh oh!

daosbuild3 commented Apr 16, 2026

Uh oh!

daosbuild3 commented Apr 20, 2026

Uh oh!

daosbuild3 commented Apr 20, 2026

Uh oh!

daosbuild3 commented Apr 21, 2026

Uh oh!

daosbuild3 commented May 4, 2026

Uh oh!

daosbuild3 commented May 4, 2026

Uh oh!

daosbuild3 commented May 4, 2026

Uh oh!

daosbuild3 commented May 4, 2026

Uh oh!

Uh oh!

daltonbohning May 14, 2026

Choose a reason for hiding this comment

Uh oh!

grom72 May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daltonbohning left a comment

Choose a reason for hiding this comment

Uh oh!

daltonbohning May 18, 2026

Choose a reason for hiding this comment

Uh oh!

grom72 May 21, 2026

Choose a reason for hiding this comment

Uh oh!

phender left a comment

Choose a reason for hiding this comment

Uh oh!

grom72 commented May 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

7 participants

grom72 commented Apr 9, 2026 •

edited

Loading

github-actions Bot commented Apr 9, 2026 •

edited

Loading

grom72 May 14, 2026 •

edited

Loading