Skip to content

Fix JAX inference timing to include GPU sync#992

Open
Abhishek21g wants to merge 1 commit into
Physical-Intelligence:mainfrom
Abhishek21g:fix/inference-timing-gpu-sync
Open

Fix JAX inference timing to include GPU sync#992
Abhishek21g wants to merge 1 commit into
Physical-Intelligence:mainfrom
Abhishek21g:fix/inference-timing-gpu-sync

Conversation

@Abhishek21g

Copy link
Copy Markdown

Summary

Fixes #983policy_timing.infer_ms under-reported GPU inference time because the timer stopped at JAX async dispatch instead of waiting for GPU work to complete.

Root cause

On the JAX path, Policy.infer() measured wall time around _sample_actions() but did not synchronize device work before stopping the timer. JAX returns from jitted calls once work is scheduled, not when kernels finish — reporters saw ~30ms gaps vs the JAX profiler.

PyTorch path is unaffected (.detach().cpu() already forces sync).

Fix

Call jax.block_until_ready(actions) on the JAX path before recording model_time.

Verification

uv run pytest src/openpi/policies/policy_test.py::test_jax_async_dispatch_under_reports_without_sync -q
uv run pytest src/openpi/policies/policy_test.py::test_jax_policy_infer_syncs_actions_before_timing -q

On GPU hardware with the policy server running:

uv run examples/simple_client/main.py --env ALOHA_SIM --num-steps 50
# policy_infer_ms should be within ~5ms of JAX profiler kernel time

policy_timing.infer_ms previously stopped the timer at async JAX dispatch,
under-reporting GPU work by ~30ms vs the profiler. Sync actions with
jax.block_until_ready on the JAX path before recording infer_ms.

Fixes Physical-Intelligence#983
@jimmyt857 jimmyt857 removed their request for review July 9, 2026 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The inference timing script is incorrect

1 participant