Add video input support for VLM inference#2359
Conversation
Adds --videomaxframes, --videofps, --videomaxtokens, --videomintokens, and --ffmpegpath, wires them into load_model_inputs (replacing the previously hardcoded defaults), and prints an effective video config plus a context-budget warning when worst-case video tokens could exceed max context. GUI launcher deferred: the vision fields use a hand-tuned pixel-positioned grid (row/padx offsets, per-field tooltips, save/load/export plumbing) that isn't a trivial copy, so video stays CLI-only for now.
C++ side downsamples fps to spread the frame budget evenly across a video's full duration, falling back to truncation only if needed.
Stream video frames via mtmd_helper_video, tokenize frames and timestamp texts into ordered media chunks, enforce videomaxframes cap and optional videomaxtokens per-frame pixel downscale. Extend the (Attached Video N) placeholder parser and wire videofps/videomaxframes/videomaxtokens/ffmpegpath globals. Guarded by MTMD_VIDEO with graceful skips on all failure paths.
Query mtmd_helper_video_get_info; when estimated frame count exceeds videomaxframes, re-init the stream at a reduced fps so sampled frames cover the full duration. The hard videomaxframes stop remains as a safety net for estimation error; if n_frames is unknown (-1) behavior is unchanged.
|
Sister PR on SillyTavern is here. |
|
What model is this for? as far as i know, there aren't any usable video models yet? |
This was tested against Gemma-4-31b-it. There are several models available that work with this though. |
|
Also tested with Qwen3.6-27B now. Note: Some video formats that work on Gemma seem to have some issues in Qwen, looking to debug now, but if it's an issue in MTMD, I'll leave it alone. |
|
Some videos failed mid-ingest on Qwen3.6 with decode: failed to find a memory slot while others worked, depending on resolution and length rather than format. The cause: for M-RoPE vision models, an image chunk advances the position counter by max(grid_w, grid_h) but occupies grid_w × grid_h KV cells (a 1080p frame: 60 positions, ~2,000 cells). The media budgeting counted positions, so a 32-frame 1080p video needing ~63k cells passed every check at a 57k context and then exhausted the cache during eval. The fix tracks real cell counts (mtmd_input_chunk_get_n_tokens) for all context-fit checks, keeping positions only for n_past math, and caps each video at 75% of context. To fit oversized videos instead of truncating them, frame resolution is calibrated by measurement rather than heuristics: pixels-per-token varies too much across encoders to guess, but mtmd_tokenize runs the model's own preprocessing without invoking the encoder, so probing a frame's true cost is cheap and exact. Since cell cost scales with pixel area, an interpolation search in area space finds the largest resolution fitting the per-frame budget in one or two probes, with a geometric fallback inside encoder clamp ranges where cost stops tracking area. For non-M-RoPE models (Gemma etc.) position and cell counts coincide, so existing checks compute identical results and calibration never triggers at typical contexts. Frames are never upscaled, in-budget videos are untouched, and the worst case is now truncation with a diagnostic message instead of a failed generation. |
2 participants
Closes #1290
Summary
Adds video ingestion for multimodal models. Videos can be attached through the OpenAI-compatible chat completions endpoint (
video_urlandinput_videocontent parts) or the native API (videosarray, mirroringimagesandaudio). Frames are decoded through the vendoredtools/mtmdvideo path (now compiled withMTMD_VIDEOin both the Makefile and CMakeLists), sampled on a frame budget, tokenized with interleaved timestamp text, and evaluated as media chunks alongside existing image and audio handling.7 files changed, +802/−25. No vendored
tools/mtmdfiles are modified.Runtime dependency
Video decoding spawns
ffmpeg/ffprobeas subprocesses; there are no new link-time dependencies. Availability is probed at model load. Without ffmpeg, video requests fail with a log message and every other modality is unaffected.New flags
--videomaxframes--videofps--videomintokens--videomaxtokens--ffmpegpathBehavior notes
duration × videofpsexceeds the frame budget, the sampling rate is lowered to(budget − 1) / durationso frames span the whole clip instead of truncating the tail. Duration comes from stream metadata when present, then container-level ffprobe, then last-packet pts (webm, mkv, and animated webp often expose no stream duration). A hard cap remains as a safety net.[XmY.YYs]timestamp strings are reformatted to[MM:SS], and a(video duration: MM:SS)text chunk is appended after the frames. In testing, Gemma misread the compact vendored notation (reporting an 18s clip as 0.55s); the reformatted output resolved it.(Attached Video N)convention.--maxrequestsizedefault, which rejects requests before any server-side logging. Model load now prints a notice when video is enabled so the failure mode is discoverable.Windows teardown deadlock (why decoding stages a temp file)
The vendored buffer-input API (
mtmd_helper_video_init_from_buf) feeds ffmpeg's stdin from a dedicated thread. On Windows, destroying an undrained video context hangs the server:subprocess.hcreates the child withbInheritHandles=1and leaves the stdin read handle inheritable, so a second spawned ffmpeg inherits a duplicate of the pipe's read end and the feeder thread never receives a broken-pipe error;feeder.join()then blocks forever. This fires on ordinary paths such as re-initializing at an adjusted fps.This PR sidesteps the bug by staging the decoded bytes to a temp file (cleaned up automatically via RAII) and using the file-based
mtmd_helper_video_init, which spawns no feeder and gives ffmpeg seekable input (better moov-at-end mp4 handling). A proper upstream fix would mark the parent's pipe handles non-inheritable (or usePROC_THREAD_ATTRIBUTE_HANDLE_LIST); happy to file that against llama.cpp separately.Pre-existing fix absorbed (offer to split)
mtmd_input_textis a four-field struct; the existing media tokenize call sites used three-field aggregate initialization, leavingtext_lenzero. The vendored tokenizer now reads exactlytext_lenbytes, so markers were lost and image tokenization failed with "number of media markers in text (0) does not match number of bitmaps (1)". This affects images and audio on the current base branch, independent of video. Fixed at all call sites here (12c84fb); I can split it into a standalone PR if preferred.Logging
Vendored decode chatter is routed through the public
mtmd_helper_log_setwith a filter honoring quiet and debug modes (a side effect is tidier image/audio batch timing output). Media generations also trigger llama core logs (set_causal_attn,sched_reserve) that previously ignored--quiet; allama_log_setfilter, installed only when an mmproj is loaded and armed after load completes, applies the same tiers. Text-only model loads are untouched.Out of scope (follow-ups)
Testing
Windows CUDA build. Verified with a Gemma vision mmproj: image and audio regression unchanged; mp4 and animated webp clips sample across their full duration with correct timestamps; long clips downsample instead of truncating; missing-ffmpeg and oversized-payload failure modes are logged; repeated video requests reuse the media cache; log output respects quiet, normal, and debug tiers.