Add video screenshot API - #1553
Open
lestenholm wants to merge 4 commits into
Open
Conversation
Adds a GET /snapshot endpoint that returns a single JPEG frame of the current video feed. A second Rockchip VENC channel is created alongside the existing H.264/H.265 channel to encode on-demand JPEG snapshots from the already-captured raw frame, plumbed up through the native gRPC service into a plain authenticated REST route. Returns 503 when no viewer session is active, since video capture only runs while a WebRTC session is connected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Fix a forward-declaration ordering bug that broke the native build (get_streaming_flag/get_streaming_stopped used before declared). - Split the single /snapshot route into format-specific /screenshot.jpg and /screenshot.png (PNG re-encoded from the hardware JPEG capture in Go, since the capture hardware only produces JPEG). - Add apiAuthMiddleware: accepts either the browser's session cookie or HTTP Basic Auth against the device password (noPassword mode is left open), so both a logged-in browser tab and a headless script/CI runner can use the same URL. - Auto-start video capture on demand when nothing is currently streaming (primary use case is HIL test rigs grabbing a screenshot before/during/ after a run, with no browser session open) and stop it again afterward unless a real WebRTC session or another concurrent screenshot request still needs it, guarded by a small in-flight counter to avoid one request's cleanup cutting off another's capture. Verified end-to-end on hardware: cold-start capture, repeated auto start/stop cycles with no leaked state, and both cookie- and Basic-Auth-gated access. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Shrink the JPEG snapshot channel's SendFrame/GetStream timeouts (2000ms + retry + 200ms -> 150ms each, single attempt) so a slow/failing encode can no longer stall the primary V4L2 capture loop for seconds; a failed attempt is simply skipped and retried against the next frame instead of retried in place. - Shrink video_get_snapshot()'s internal wait from 500ms to 300ms, since it runs under cgoLock (shared by every native call, including UI ticks) and directly bounds how long unrelated native operations get blocked. - Fix captureScreenshot()'s auto-start retry loop to retry on any error until the deadline, not just ErrVideoNotStreaming -- the native side also returns transient per-attempt errors (encode timeout, no frame yet) while capture is spinning up, which need retrying too. Re-verified end-to-end on hardware after these changes: cold-start capture still succeeds (including the slower ~6s cases, which now correctly retry through transient errors instead of failing), and no visible stutter on an active live view while concurrently fetching a screenshot. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Author
|
Ran this through an internal code review before opening for real review; it surfaced 3 legitimate issues, now fixed in fb59c8b:
Re-verified end-to-end on hardware after the fixes: cold-start capture (including slower ~6s cases) still succeeds, and confirmed no visible stutter on an active live view while concurrently fetching a screenshot. |
2 tasks
lestenholm
force-pushed
the
video-snapshot-api
branch
from
August 24, 2026 19:30
fb59c8b to
275a9b6
Compare
Covers /screenshot.jpg and /screenshot.png: rejects requests with no credentials or a bogus session cookie, succeeds via HTTP Basic Auth and via a logged-in browser session (checking status, content-type, and that the body is a real image), and succeeds with no credentials at all in noPassword mode. Verified passing 9/9 against the branch build on real hardware. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
lestenholm
marked this pull request as ready for review
August 24, 2026 19:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1555
Summary
GET /screenshot.jpgandGET /screenshot.png, returning a single frame of the current video feed. JPEG comes straight from a second Rockchip VENC channel added alongside the existing H.264/H.265 encoder channel; PNG is re-encoded from that JPEG in Go (the capture hardware only produces JPEG).apiAuthMiddleware: accepts either the browser's session cookie (so an already-logged-in tab can just load the URL) or HTTP Basic Auth against the device password with the username ignored (curl -u api:<password> ...), so headless/CI callers don't need a login step.noPasswordmode is left open, matching the rest of the local API.ui/e2e/screenshot-api.spec.tscovering both endpoints: rejects no credentials / a bogus session cookie, succeeds via HTTP Basic Auth and via a logged-in browser session, and succeeds with no credentials innoPasswordmode.Test plan
go build ./.../go vet ./.../gofmt -lclean (native cgo lib built via theghcr.io/jetkvm/buildkitDocker image, not compiled locally).jpgand.png), confirmed valid images matching the live HDMI sourcestreamingcorrectly returns to0after each request)ui/e2e/screenshot-api.spec.ts: 9/9 passing against real hardwaremake test_e2e DEVICE_IP=<device>:14 passed / 3 failed / 56 skipped. The 56remote-agentskips are expected — that project needs a general-purpose PC host, not available in this setup. The 3 failures (all 3 OTA suites) are the same pre-existing assertion in a shared helper (runMouseBidirectionalCheck, diffs video-region screenshots assuming a static desktop) — it fails on our test rig's live/animating video source, unrelated to this change; all 3 failures happen on the very last line of each test, after the real OTA-update assertions already passed.internal/native/cgo/video.c(RK_VIDEO_ID_JPEG,VENC_JPEG_PARAM_S,RK_MPI_VENC_SetJpegParam) against the SDK headers — written from memory of the API and confirmed to compile + work on hardware, but worth a second look🤖 Generated with Claude Code