-
Notifications
You must be signed in to change notification settings - Fork 3.5k
[Test] Stabilize RTX MDL shader-warmup flake in rendering correctness tests #5687
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
hujc7
wants to merge
6
commits into
isaac-sim:develop
Choose a base branch
from
hujc7:jichuanh/stabilize-rendering-mdl-warmup
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
a9f3fac
[Test] Stabilize RTX MDL shader-warmup flake in rendering correctness…
hujc7 08cd6c9
Use sim.render warmup so goldens stay valid
hujc7 c359111
Narrow MDL warmup fix to test_camera_renders_not_empty
hujc7 39e6bb7
Poll camera output until non-zero, max 60 steps
hujc7 3de115a
Add poll-until-nonzero warmup in goldenfile helpers
hujc7 eb23b0c
Apply warmup to registered-tasks rendering test
hujc7 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
13 changes: 13 additions & 0 deletions
13
source/isaaclab_tasks/changelog.d/jichuanh-stabilize-rendering-mdl-warmup.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| Fixed | ||
| ^^^^^ | ||
|
|
||
| * Stabilized ``test_camera_renders_not_empty`` in | ||
| ``test_shadow_hand_vision_presets.py`` by polling the camera output until | ||
| all data-type tensors are non-zero, with a 60-step cap, instead of a | ||
| single ``env.step()``. The test intermittently failed with "Camera output | ||
| is all zeros or all inf" for ``simple_shading_*_mdl`` and | ||
| ``simple_shading_constant_diffuse`` variants on cold-cache CI runners | ||
| because the GPU returned a still-zero framebuffer before the MDL material | ||
| finished compiling. The three goldenfile-comparing helpers in | ||
| ``rendering_test_utils.py`` already use ``flaky(max_runs=3)`` and are left | ||
| untouched. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If all 60 steps complete without any output reaching the threshold, the fixture silently yields and the test fails with the standard "Camera output is all zeros or all inf" assertion error — with no indication that the warmup cap was hit. Adding a
warnings.warnorprinton cap exhaustion would make it immediately clear in CI logs that the problem is slow shader compilation rather than a deeper rendering bug.