Fix JAX inference timing to include GPU sync#992
Open
Abhishek21g wants to merge 1 commit into
Open
Conversation
policy_timing.infer_ms previously stopped the timer at async JAX dispatch, under-reporting GPU work by ~30ms vs the profiler. Sync actions with jax.block_until_ready on the JAX path before recording infer_ms. Fixes Physical-Intelligence#983
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #983 —
policy_timing.infer_msunder-reported GPU inference time because the timer stopped at JAX async dispatch instead of waiting for GPU work to complete.Root cause
On the JAX path,
Policy.infer()measured wall time around_sample_actions()but did not synchronize device work before stopping the timer. JAX returns from jitted calls once work is scheduled, not when kernels finish — reporters saw ~30ms gaps vs the JAX profiler.PyTorch path is unaffected (
.detach().cpu()already forces sync).Fix
Call
jax.block_until_ready(actions)on the JAX path before recordingmodel_time.Verification
On GPU hardware with the policy server running:
uv run examples/simple_client/main.py --env ALOHA_SIM --num-steps 50 # policy_infer_ms should be within ~5ms of JAX profiler kernel time