Wayland previews#1398
Draft
davidplowman wants to merge 15 commits into
Draft
Conversation
QtGlPreview (QGlPicamera2) renders raw EGL straight onto the widget's
native window: WA_PaintOnScreen plus eglCreateWindowSurface(winId()).
That contract only holds on X11, where winId() is an X window. On Wayland
eglCreateWindowSurface needs a wl_egl_window built from the window's
wl_surface, and no Qt binding exposes that per-window surface to Python
(Qt6 dropped the public per-window native interface; only
QNativeInterface::QWaylandApplication remains). So under a Wayland
compositor QtGlPreview can only run through XWayland, which adds a
per-frame texture-format/copy pass in the compositor.
Add an alternative preview, Preview.QTGL_WL (QtGlPreviewWayland, widget
QGlPicamera2Wl), that renders through Qt's own OpenGL context via
QOpenGLWidget. Qt creates and owns the platform surface (the
wl_egl_window on Wayland, the GLX/EGL drawable on X11), so we never
touch wl_surface and there is no platform-specific code. The only thing
the zero-copy path needs is an EGLDisplay to import the camera dmabuf on,
and inside paintGL() Qt's context is current, so eglGetCurrentDisplay()
returns the right display on either platform. The dmabuf ->
EGLImage(EGL_LINUX_DMA_BUF_EXT) -> GL_OES_EGL_image_external sampling is
reused unchanged.
Because it uses Qt's context, the same widget works on X11 too, but the
existing Preview.QTGL is left as the default so nothing changes for
current users; QTGL_WL is purely additive and opt-in.
Enabling it:
from picamera2 import Picamera2, Preview
picam2 = Picamera2()
picam2.start_preview(Preview.QTGL_WL)
picam2.configure(picam2.create_preview_configuration())
picam2.start()
See examples/preview_qtgl_wayland.py.
Performance (Pi 4, IMX219, labwc, Mesa 25.0.7 / V3D; GPU-busy time from
DRM fdinfo, summed across the preview client and the compositor, over a
20s window):
1920x1080 QTGL (XWayland) 59.8 fps gl 15.29 ms/frame labwc tfu 2.77 ms/frame
QTGL_WL (Wayland) 80.9 fps gl 11.38 ms/frame labwc tfu 0
3840x2160 QTGL (XWayland) 30.1 fps gl 31.16 ms/frame labwc tfu 9.31 ms/frame
QTGL_WL (Wayland) 42.3 fps gl 24.95 ms/frame labwc tfu 0
The native path eliminates the compositor's XWayland texture-format
(tfu) pass entirely; that cost scales with buffer size (~2.8 ms/frame at
1080p, ~9.3 ms/frame at 4K). At the same GPU saturation that freed
budget becomes more delivered frames (+35% at 1080p, +40% at 4K).
Trade-off: QOpenGLWidget renders into an FBO that Qt then composites into
the window (one extra blit) rather than rendering directly to the
surface; this shows up as higher client-side render cost, most visibly at
4K, but is outweighed by dropping the compositor format pass.
Implementation notes:
- Requests a GLES context (samplerExternalOES / GL_OES_EGL_image_external
live in GLES).
- Camera frames arrive on the GUI thread via the existing QSocketNotifier;
render_request() stashes the request and calls update(), and the GL work
happens in paintGL() with Qt's context current.
- Live resize is handled (resizeGL repaints; the viewport, including
aspect-ratio letterboxing, is recomputed from the live widget size every
repaint).
Signed-off-by: Dom Cobley <popcornmix@gmail.com>
…L_WL_DIRECT)
Preview.QTGL_WL uses a QOpenGLWidget, which always renders into an
offscreen FBO that Qt then composites into the window - one extra
full-frame GPU blit per frame. Add Preview.QTGL_WL_DIRECT, which uses a
QOpenGLWindow embedded with QWidget.createWindowContainer(): the
QOpenGLWindow owns its own native (sub)surface and presents directly via
eglSwapBuffers, so there is no in-process blit. This restores the
directness of the original X11 QGlPicamera2 (WA_PaintOnScreen) while
staying native Wayland - on Wayland Qt backs the window with a
wl_egl_window, on X11 with an X drawable. The zero-copy dmabuf ->
EGLImage -> GL_OES_EGL_image_external path is unchanged.
Rendering is done synchronously in render_request() (makeCurrent ->
repaint -> swapBuffers), exactly like the original QGlPicamera2. Deferred
update()/requestUpdate() does not work here: it is throttled to Wayland
frame callbacks, which an embedded subsurface does not reliably receive
(observed ~1 paint/s), so the preview would starve. A QOpenGLWindow, unlike
a QOpenGLWidget, lets us drive its context from outside paintGL, which makes
the synchronous path possible.
Both previews are kept. QTGL_WL_DIRECT is the fastest option;
QTGL_WL remains for embedders that need it, because container/native
windows always stack above sibling widgets and cannot be clipped by
non-rectangular masks. For a viewfinder that fills its area this is fine
(overlays are drawn inside this GL context); apps that float Qt widgets
over the preview should use QTGL_WL.
Enabling it:
from picamera2 import Picamera2, Preview
picam2 = Picamera2()
picam2.start_preview(Preview.QTGL_WL_DIRECT)
picam2.configure(picam2.create_preview_configuration())
picam2.start()
See examples/preview_qtgl_wayland_direct.py.
Performance (Pi 4, IMX219, labwc, Mesa 25.0.7 / V3D; GPU-busy time from
DRM fdinfo summed across the preview client and the compositor; the
client-side render/frame column isolates the blit cost; 20s window):
1920x1080 QTGL (XWayland) 59.5 fps gl 15.33 ms/f client render 1.64 ms/f
QTGL_WL (FBO) 80.0 fps gl 11.39 ms/f client render 3.58 ms/f
QTGL_WL_DIRECT 90.4 fps gl 7.42 ms/f client render 1.02 ms/f
3840x2160 QTGL (XWayland) 30.1 fps gl 31.16 ms/f client render 5.73 ms/f
QTGL_WL (FBO) 42.2 fps gl 25.05 ms/f client render 13.70 ms/f
QTGL_WL_DIRECT 80.0 fps gl 12.07 ms/f client render 3.64 ms/f
Removing the blit cuts the preview client's per-frame GPU render cost from
3.58 to 1.02 ms at 1080p and from 13.70 to 3.64 ms at 4K (a ~2.6 / ~10
ms/frame saving), and roughly doubles 4K throughput versus XWayland
(30 -> 80 fps).
Signed-off-by: Dom Cobley <popcornmix@gmail.com>
For native Wayland, we mustn't set QT_QPA_PLATFORM to "xcb", it should be "wayland" instead. The fix here corrects it for the native Wayland preview windows, but will not work more widely (e.g. for previews within a proper Qt app). So another solution will be needed at some point. Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Caused by not initialising a buffer. Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
The blanket override in picamera2/__init__.py that force everything down the X11 / XWayland route has been removed. previews/qt.py has been amended so that it switches automatically, based on the platform environment variables, to the X11-style or native Wayland widgets. This works for the PyQt6 and PySide6 flavours of the widgets too. Note that it switches between the old X-style version and the "non-direct" Wayland widget because the "direct" version doesn't support overlays and is therefore not a complete replacement. A PySide6 version of the native Wayland widget has been added too. Haven't bothered with a PySide2 variant. Support for the "direct" Wayland widget has been incorporated in the standalone preview by asking for the Preview.QTGL_DIRECT version. On a Wayland platform you will get the more optimised renderer; on an X11 platform you'll just get the old X11 version. In proper Qt apps (rather than the standalone preview), you will get the automatic X11/Wayland selection, but you can pass the "direct=True" parameter when creating the widget. On a Wayland platform, again, you'll get the optimised implementation, and on X11 you'll just get the old one. It's worth noting that there's a difference between Qt 5 and Qt 6 when creating a proper Qt app. Qt 6 is more likely to default to the Wayland backend, so you'll get the Wayland widgets. Qt 5 is likely to default to the X11 backend (even on a Wayland platform), so you'll get those widgets. But you can always override the behaviour by setting the QT_QPA_PLATFORM environment variable before starting the Qt application. Set it to "xcb" to force the X11 backend, or "wayland" for Wayland. Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
But we ignore all the "*" OpenGL imports, as this appears to be standard practice. Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Previously, all three OpenGL preview window types (X11, Wayland, Wayland "direct") were duplicating this code exactly. Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
This can now be shared across all the GLES previews, avoiding code duplication. Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
By creating a _GlRendererMixin class in gl_helpers. Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Although anyone can use this facility, it's really aimed at our test code, so that we can easily check that the correct behaviour is happening underneath. Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
We test that the new native Wayland previews work, and are used as expected, across PyQt5, PyQt6, PySide6, including "direct" and "non-direct" versions. We check both standalone previews and preview windows embedded in Qt applications. We also test that we can force the choice of X11/Wayland implementations by setting QT_QPA_PLATFORM appropriate to either "xcb" or "wayland", as this might be a useful feature in case users experience unexpected regressions. Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
Signed-off-by: David Plowman <david.plowman@raspberrypi.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This takes Dom's native Wayland preview windows and generally munges everything until I think it's in an acceptable state. But not one to commit just yet as I would like to sleep on it for a bit as it does change default behaviour signficantly.