Skip to content

DAOS-18640 test: per_server_fault_domain.yaml - Increase space_rb to 10%#18283

Open
shimizukko wants to merge 3 commits into
masterfrom
makito/DAOS-18640-2
Open

DAOS-18640 test: per_server_fault_domain.yaml - Increase space_rb to 10%#18283
shimizukko wants to merge 3 commits into
masterfrom
makito/DAOS-18640-2

Conversation

@shimizukko
Copy link
Copy Markdown
Contributor

@shimizukko shimizukko commented May 19, 2026

The tests in per_server_fault_domain.py is failing due to rebuild timeout after stopping the ranks.

test_rf2_unclean had been fixed with space_rb:5%, but apparently 5% isn't enough for other tests in this file, so increase it.

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-large: false
Test-tag: PerServerFaultDomainTest
Test-repeat: 3

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

@shimizukko
DAOS-18640 test: per_server_fault_domain.yaml - Increase space_rb to 8%
500a23e 
The tests in per_server_fault_domain.py is failing
due to rebuild timeout after stopping the ranks.

test_rf2_unclean had been fixed with space_rb:5%, but
apparently 5% isn't enough for other tests in this
file, so increase it.

Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-large: false
Test-tag: PerServerFaultDomainTest
Test-repeat: 3
Signed-off-by: Makito Kano <makito.kano@hpe.com>
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 19, 2026 •
edited
Loading

Ticket title is 'container/per_server_fault_domain.py:PerServerFaultDomainTest.test_rf2_unclean - wait for rebuild test timeout - DER_NOSPACE'
Status is 'In Review'
Labels: 'ci_master_weekly,request_for_2.8,weekly_test'
https://daosio.atlassian.net/browse/DAOS-18640

@github-actions github-actions Bot added the priority Ticket has high priority (automatically managed) label May 19, 2026
@daosbuild3
Copy link
Copy Markdown
Collaborator

daosbuild3 commented May 19, 2026

Test stage Functional Hardware Large completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18283/1/execution/node/824/log

@shimizukko
DAOS-18640 test: Increase space_rb to 10
d1f7ada 
Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-large: false
Test-tag: PerServerFaultDomainTest
Test-repeat: 4
Signed-off-by: Makito Kano <makito.kano@hpe.com>
@shimizukko shimizukko changed the title DAOS-18640 test: per_server_fault_domain.yaml - Increase space_rb to 8% DAOS-18640 test: per_server_fault_domain.yaml - Increase space_rb to 10% May 20, 2026
@shimizukko shimizukko marked this pull request as ready for review May 20, 2026 10:55
@shimizukko shimizukko requested review from a team as code owners May 20, 2026 10:55
@shimizukko shimizukko requested review from kccain and phender May 20, 2026 10:57
phender
phender previously approved these changes May 20, 2026
View reviewed changes
@shimizukko shimizukko added the waiting-for-merge-approval Waiting for merge approval label May 22, 2026
mchaarawi
mchaarawi requested changes May 22, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@mchaarawi mchaarawi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is it OK to change space_rb to 10?
the default is 5 and changing it to 10 means there is an issue..
on aurora we don't even set to 10 and set to 8. some more information is needed why 10 is required here.

@daltonbohning
Copy link
Copy Markdown
Contributor

daltonbohning commented May 22, 2026

why is it OK to change space_rb to 10? the default is 5 and changing it to 10 means there is an issue.. on aurora we don't even set to 10 and set to 8. some more information is needed why 10 is required here.

After some more investigation here, I'm not sure increasing space_rb will even help:
https://daosio.atlassian.net/browse/DAOS-18640?focusedCommentId=174182

daltonbohning
daltonbohning requested changes May 22, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@daltonbohning daltonbohning left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with Mohamad that we should not need to increase space_rb for this one test. It's probably better to adjust the data generation or size such that we have enough space to begin with. See my comment here: https://daosio.atlassian.net/browse/DAOS-18640?focusedCommentId=174182

@shimizukko
DAOS-18640 test: Reduce IOR block_size to generate less data
bbff592 
Skip-unit-tests: true
Skip-fault-injection-test: true
Skip-func-hw-test-large: false
Test-tag: PerServerFaultDomainTest
Test-repeat: 5
Signed-off-by: Makito Kano <makito.kano@hpe.com>
@github-actions github-actions Bot removed the priority Ticket has high priority (automatically managed) label May 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mchaarawi mchaarawi mchaarawi requested changes

@daltonbohning daltonbohning daltonbohning requested changes

@phender phender phender left review comments

@kccain kccain Awaiting requested review from kccain

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

waiting-for-merge-approval Waiting for merge approval

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@shimizukko @daosbuild3 @daltonbohning @mchaarawi @phender