Reduce delays in webRTC remote display.#7496
Open
ssheorey wants to merge 12 commits into
Open
Conversation
…code, VP9 preference - Remove duplicate redraw on input (WebRTCWindowSystem.cpp) - Draw coalescing (BitmapWindowSystem.cpp) - Input (mouse event) coalescing (BitmapWindowSystem.cpp) - JS requestAnimationFrame coalescing / throttling (webrtcstreamer.js) - Async encoder thread (PeerConnectionManager.cpp/.h) - Data channel low-latency mode (webrtcstreamer.js) - Reduced startup delay (500ms -> 250ms) (WebRTCWindowSystem.cpp) - VP9 codec preference over VP8 (webrtcstreamer.js)
|
Thanks for submitting this pull request! The maintainers of this repository would appreciate if you could update the CHANGELOG.md based on your changes. |
sends zero playout delay in every RTP packet's header extension when adaptation is necessary, drops resolution rather than framerate prevents WebRTC's autonomous resolution scaling; Replaced playoutDelayHint with jitterBufferTarget = 0
There was a problem hiding this comment.
Pull request overview
This PR targets lower end-to-end latency for Open3D’s WebRTC-based remote visualization by reducing redundant redraws, coalescing high-frequency input/render events, and decoupling frame encoding from the render thread. It also hardens ASSIMP glTF export by skipping invalid/unsupported texture maps.
Changes:
- Reduce render/event backlog via redraw and input/draw coalescing in the GUI/WebRTC paths.
- Move WebRTC frame delivery/encoding work off the render thread via an asynchronous encoder thread.
- Improve streaming responsiveness via WebRTC configuration tweaks (field trials, codec preference, jitter buffer settings), plus safer texture export validation.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| python/js/package.json | Adds a Yarn resolution override for license-webpack-plugin. |
| cpp/open3d/visualization/webrtc_server/WebRTCWindowSystem.cpp | Removes duplicate redraw scheduling on data-channel messages; reduces init-frame pacing delay. |
| cpp/open3d/visualization/webrtc_server/PeerConnectionManager.h | Adds state needed for an asynchronous encoder thread (queue/mutex/cv/thread). |
| cpp/open3d/visualization/webrtc_server/PeerConnectionManager.cpp | Adds WebRTC field trials/content hints, bundle policy, async encoder thread, and frame coalescing. |
| cpp/open3d/visualization/webrtc_server/html/webrtcstreamer.js | Coalesces pointer/wheel events via rAF, prefers VP9, tweaks jitter buffer, and changes data-channel options. |
| cpp/open3d/visualization/gui/BitmapWindowSystem.cpp | Coalesces draw events per window and merges high-rate mouse MOVE/DRAG/WHEEL events. |
| cpp/open3d/t/io/file_format/FileASSIMP.cpp | Skips exporting empty/unsupported texture maps; logs warnings for unsupported texture dtypes. |
Comment on lines
+574
to
+579
| const auto& material = w_mesh.GetMaterial(); | ||
| int n_textures = 0; | ||
| if (w_mesh.GetMaterial().HasAlbedoMap()) ++n_textures; | ||
| if (w_mesh.GetMaterial().HasNormalMap()) ++n_textures; | ||
| if (w_mesh.GetMaterial().HasAOMap()) ++n_textures; | ||
| if (w_mesh.GetMaterial().HasAORoughnessMetalMap()) { | ||
| if (HasValidTexture(material, "albedo")) ++n_textures; | ||
| if (HasValidTexture(material, "normal")) ++n_textures; | ||
| if (HasValidTexture(material, "ambient_occlusion")) ++n_textures; | ||
| if (HasValidTexture(material, "ao_rough_metal")) { |
…channel reliability
…faster download. General cleanup.
- Prepend depot_tools to PATH and run win_tools.bat on Windows - Disable protobuf constinit on Apple to fix Xcode 15.4 build
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type
Motivation and Context
WebRTC remote display is slow, even for local viewing. This PR reduces remote rendering latency.
Checklist:
python util/check_style.py --applyto apply Open3D code styleto my code.
updated accordingly.
results (e.g. screenshots or numbers) here.
Description
Also: Check textures for validity before trying to write them to gltf.