Add video input support for VLM inference#2359

Open

Reithan wants to merge 30 commits into

LostRuins:concedo_experimentalfrom

Reithan:add-video-input-for-vlm-inference

Commits on Jul 23, 2026

refactor media_object is_audio flag into media type enum
Reithan
committed
add video fields to generation_inputs and load_model_inputs contract
Reithan
committed
marshal videos genparam into generation_inputs struct
Reithan
committed
populate video media_objects and skip video eval safely in gpttype_adapter
Reithan
committed
enable MTMD_VIDEO and probe video capability at load
Reithan
committed
accept video content parts in chat completions paths
Reithan
committed
add video input CLI flags and budget warning
Reithan
committed
reword --videomaxframes help as a sampling budget, not a truncation cap
Reithan
committed
implement MTMD_VIDEO frame-streaming eval path in PrepareMediaEmbds
Reithan
committed
downsample video fps to fit frame budget instead of truncating
Reithan
committed
hash media items for cache signature instead of concatenating base64
Reithan
committed
cap attached videos before placeholder numbering and warn on drop
Reithan
committed
enforce videomintokens by upscaling undersized video frames
Reithan
committed
document why fnv1a64_hex is implemented locally
Reithan
committed
keep most recent videos when capping, drop oldest
Reithan
committed
set mtmd_input_text text_len on all media tokenize calls to fix lost marker
Reithan
committed
note maxrequestsize limit on video attachments at model load
Reithan
committed
decode video via temp file to avoid ffmpeg feeder-thread teardown deadlock
Reithan
committed
aim downsampled fps so the last frame lands inside the frame budget
Reithan
committed
probe container duration to downsample videos lacking stream metadata
Reithan
committed
respect quiet mode in video pipeline logging
Reithan
committed
add TODO outlining future video audio-track embedding
Reithan
committed
reformat video timestamps to MM:SS and append duration note for model readability
Reithan
committed
filter llama core log spam during media generations per quiet/debug tiers
Reithan
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add video input support for VLM inference#2359

Add video input support for VLM inference#2359
Reithan wants to merge 30 commits into
LostRuins:concedo_experimentalfrom
Reithan:add-video-input-for-vlm-inference

Commits on Jul 23, 2026

refactor media_object is_audio flag into media type enum

add video fields to generation_inputs and load_model_inputs contract

marshal videos genparam into generation_inputs struct

populate video media_objects and skip video eval safely in gpttype_adapter

enable MTMD_VIDEO and probe video capability at load

accept video content parts in chat completions paths

add video input CLI flags and budget warning

reword --videomaxframes help as a sampling budget, not a truncation cap

implement MTMD_VIDEO frame-streaming eval path in PrepareMediaEmbds

downsample video fps to fit frame budget instead of truncating

hash media items for cache signature instead of concatenating base64

cap attached videos before placeholder numbering and warn on drop

enforce videomintokens by upscaling undersized video frames

document why fnv1a64_hex is implemented locally

keep most recent videos when capping, drop oldest

set mtmd_input_text text_len on all media tokenize calls to fix lost marker

note maxrequestsize limit on video attachments at model load

decode video via temp file to avoid ffmpeg feeder-thread teardown deadlock

aim downsampled fps so the last frame lands inside the frame budget

probe container duration to downsample videos lacking stream metadata

respect quiet mode in video pipeline logging

add TODO outlining future video audio-track embedding

reformat video timestamps to MM:SS and append duration note for model readability

filter llama core log spam during media generations per quiet/debug tiers

Commits on Jul 24, 2026

budget media context by real KV cell counts instead of position advance

cap video frame resolution and frame count by context KV cell budget

calibrate video frame resolution from measured first-frame cell cost

iterate frame resolution calibration via interpolation search on measured cells

make frame resolution calibration bidirectional with bracketed probes

Merge branch 'concedo_experimental' into add-video-input-for-vlm-inference