Skip to content

Add video screenshot API - #1553

Open
lestenholm wants to merge 4 commits into
jetkvm:devfrom
lestenholm:video-snapshot-api
Open

Add video screenshot API#1553
lestenholm wants to merge 4 commits into
jetkvm:devfrom
lestenholm:video-snapshot-api

Conversation

@lestenholm

@lestenholm lestenholm commented Aug 22, 2026

Copy link
Copy Markdown

Closes #1555

Summary

  • Adds GET /screenshot.jpg and GET /screenshot.png, returning a single frame of the current video feed. JPEG comes straight from a second Rockchip VENC channel added alongside the existing H.264/H.265 encoder channel; PNG is re-encoded from that JPEG in Go (the capture hardware only produces JPEG).
  • New apiAuthMiddleware: accepts either the browser's session cookie (so an already-logged-in tab can just load the URL) or HTTP Basic Auth against the device password with the username ignored (curl -u api:<password> ...), so headless/CI callers don't need a login step. noPassword mode is left open, matching the rest of the local API.
  • Auto-starts video capture on demand if nothing is currently streaming (primary use case: HIL test rigs grabbing a screenshot before/during/after a run, with no browser session open), and stops it again afterward unless a real WebRTC session or another concurrent screenshot request still needs it — guarded by a small in-flight counter so one request's cleanup can't cut off another's capture.
  • Adds ui/e2e/screenshot-api.spec.ts covering both endpoints: rejects no credentials / a bogus session cookie, succeeds via HTTP Basic Auth and via a logged-in browser session, and succeeds with no credentials in noPassword mode.

Test plan

  • go build ./... / go vet ./... / gofmt -l clean (native cgo lib built via the ghcr.io/jetkvm/buildkit Docker image, not compiled locally)
  • Deployed and exercised end-to-end on real JetKVM hardware:
    • Cold-start capture with no session open (both .jpg and .png), confirmed valid images matching the live HDMI source
    • Repeated auto start/stop cycles with no leaked native state (streaming correctly returns to 0 after each request)
    • Auth: rejected with no credentials, accepted via a valid session cookie, accepted via HTTP Basic Auth
    • No visible stutter on an active live view while concurrently fetching a screenshot
  • New ui/e2e/screenshot-api.spec.ts: 9/9 passing against real hardware
  • Ran make test_e2e DEVICE_IP=<device>: 14 passed / 3 failed / 56 skipped. The 56 remote-agent skips are expected — that project needs a general-purpose PC host, not available in this setup. The 3 failures (all 3 OTA suites) are the same pre-existing assertion in a shared helper (runMouseBidirectionalCheck, diffs video-region screenshots assuming a static desktop) — it fails on our test rig's live/animating video source, unrelated to this change; all 3 failures happen on the very last line of each test, after the real OTA-update assertions already passed.
  • Reviewer: please double check the new Rockchip MPI calls in internal/native/cgo/video.c (RK_VIDEO_ID_JPEG, VENC_JPEG_PARAM_S, RK_MPI_VENC_SetJpegParam) against the SDK headers — written from memory of the API and confirmed to compile + work on hardware, but worth a second look

🤖 Generated with Claude Code

lestenholm and others added 2 commits August 22, 2026 21:05
Adds a GET /snapshot endpoint that returns a single JPEG frame of the
current video feed. A second Rockchip VENC channel is created alongside
the existing H.264/H.265 channel to encode on-demand JPEG snapshots from
the already-captured raw frame, plumbed up through the native gRPC
service into a plain authenticated REST route. Returns 503 when no
viewer session is active, since video capture only runs while a WebRTC
session is connected.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Fix a forward-declaration ordering bug that broke the native build
  (get_streaming_flag/get_streaming_stopped used before declared).
- Split the single /snapshot route into format-specific /screenshot.jpg
  and /screenshot.png (PNG re-encoded from the hardware JPEG capture in
  Go, since the capture hardware only produces JPEG).
- Add apiAuthMiddleware: accepts either the browser's session cookie or
  HTTP Basic Auth against the device password (noPassword mode is left
  open), so both a logged-in browser tab and a headless script/CI runner
  can use the same URL.
- Auto-start video capture on demand when nothing is currently streaming
  (primary use case is HIL test rigs grabbing a screenshot before/during/
  after a run, with no browser session open) and stop it again afterward
  unless a real WebRTC session or another concurrent screenshot request
  still needs it, guarded by a small in-flight counter to avoid one
  request's cleanup cutting off another's capture.

Verified end-to-end on hardware: cold-start capture, repeated auto
start/stop cycles with no leaked state, and both cookie- and
Basic-Auth-gated access.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@CLAassistant

CLAassistant commented Aug 22, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@lestenholm lestenholm changed the title Add video snapshot API Add video screenshot API Aug 22, 2026
- Shrink the JPEG snapshot channel's SendFrame/GetStream timeouts (2000ms
  + retry + 200ms -> 150ms each, single attempt) so a slow/failing encode
  can no longer stall the primary V4L2 capture loop for seconds; a failed
  attempt is simply skipped and retried against the next frame instead of
  retried in place.
- Shrink video_get_snapshot()'s internal wait from 500ms to 300ms, since
  it runs under cgoLock (shared by every native call, including UI ticks)
  and directly bounds how long unrelated native operations get blocked.
- Fix captureScreenshot()'s auto-start retry loop to retry on any error
  until the deadline, not just ErrVideoNotStreaming -- the native side
  also returns transient per-attempt errors (encode timeout, no frame
  yet) while capture is spinning up, which need retrying too.

Re-verified end-to-end on hardware after these changes: cold-start
capture still succeeds (including the slower ~6s cases, which now
correctly retry through transient errors instead of failing), and no
visible stutter on an active live view while concurrently fetching a
screenshot.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@lestenholm

Copy link
Copy Markdown
Author

Ran this through an internal code review before opening for real review; it surfaced 3 legitimate issues, now fixed in fb59c8b:

  1. Snapshot encode could stall the live V4L2 capture loop for up to ~4.2s on a slow/failing JPEG encode (only 3 capture buffers deep) — shrunk the encoder timeouts and dropped the in-place retry, worst case now ~300ms.
  2. Auto-start's retry loop only retried one specific error, so a mid-cold-start timeout could surface as an immediate 500 instead of retrying — now retries any error until the 10s deadline.
  3. The snapshot wait held cgoLock (shared by every native call, including UI ticks) for up to 500ms per attempt — shrunk to 300ms.

Re-verified end-to-end on hardware after the fixes: cold-start capture (including slower ~6s cases) still succeeds, and confirmed no visible stutter on an active live view while concurrently fetching a screenshot.

Covers /screenshot.jpg and /screenshot.png: rejects requests with no
credentials or a bogus session cookie, succeeds via HTTP Basic Auth and
via a logged-in browser session (checking status, content-type, and
that the body is a real image), and succeeds with no credentials at
all in noPassword mode.

Verified passing 9/9 against the branch build on real hardware.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@lestenholm
lestenholm marked this pull request as ready for review August 24, 2026 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a screenshot API for programmatic capture of the video feed

2 participants