0.20.11 regression: all vision predictions fail with 'max_length (2147483647) cannot be greater than model context_length'

[elbruno]

Summary

In 0.20.11, every vision prediction that passes an image fails:

max_length (2147483647) cannot be greater than model context_length (32768)

OnnxVisionModel.ResolveVisionInputTokenCount creates a probe generator with
max_length = int.MaxValue. ONNX Runtime GenAI 0.15.x validates max_length against the
model's context_length from genai_config.json and throws, so the probe added by the #44 fix
aborts the request before generation starts.

This affects Fara1.5-9B (declares context_length: 32768) and any model whose config declares
a context_length.

Evidence

Decompiled from ElBruno.LocalLLMs 0.20.11, ElBruno.LocalLLMs.Internal.OnnxVisionModel:

private const int ProbeMaxLength = int.MaxValue;

private static int ResolveVisionInputTokenCount(
    IVisionGenerationRuntime runtime, string prompt, IVisionInputs inputs)
{
    IVisionGenerator probeGenerator = runtime.CreateProbeGenerator(int.MaxValue); // 💥 throws here
    try
    {
        probeGenerator.SetInputs(inputs);
        return ResolveInputTokenCount(name =>
        {
            try { return probeGenerator.GetInputShape(name); }
            catch { return null; }
        }, runtime.CountPromptTokens(prompt));
    }
    finally { probeGenerator?.Dispose(); }
}

CreateProbeGenerator is not wrapped in a try/catch, so the exception propagates out of
GenerateWithImagesCore:

int inputTokenCount = imagePaths.Length != 0
    ? ResolveVisionInputTokenCount(runtime, prompt, inputs)   // ← unguarded
    : runtime.CountPromptTokens(prompt);

Both GenerateWithImagesCore and GenerateWithImagesStreamingCore take this path, so
GetResponseAsync and GetStreamingResponseAsync are both broken.

Note the real generation path is fine — ResolveMaxLength is correct:

internal static int ResolveMaxLength(int maxLength, int inputTokenCount, int? maxOutputTokens) =>
    Math.Max(
        maxOutputTokens.HasValue ? Math.Min(maxLength, inputTokenCount + maxOutputTokens.Value) : maxLength,
        inputTokenCount + 1);

With MaxSequenceLength = 4096, inputTokenCount ≈ 2900, MaxOutputTokens = 128 it yields
3028, comfortably under context_length. Only the probe uses int.MaxValue.

Steps to reproduce

1. ElBruno.LocalLLMs 0.20.11, KnownModels.Fara15_9B, ExecutionProvider.Auto (resolves to CPU).

2. var response = await client.GetResponseAsync(
       [new ChatMessage(ChatRole.User, "Predict the next action.")],
       new VisionChatOptions { ImagePaths = [screenshotPath], MaxOutputTokens = 128, Temperature = 0.1f });

3. The call throws before any tokens are generated:
   max_length (2147483647) cannot be greater than model context_length (32768)

Reproduces with MaxOutputTokens set or unset, and with any MaxSequenceLength, because the
probe ignores both.

Expected

The input-token probe should never make a request the model can reject, and a probe failure
should never fail the user's request — it exists only to refine a token count that already has a
working fallback (runtime.CountPromptTokens(prompt)).

Suggested fix

Two independent changes; either one unblocks callers, and both together are ideal:

1. Clamp the probe length to the model's context length. GenAIConfigParser already parses
   the config, so the value is available via Metadata:

   var probeMaxLength = Math.Min(ProbeMaxLength, metadata?.ContextLength ?? ProbeMaxLength);
   using var probeGenerator = runtime.CreateProbeGenerator(probeMaxLength);

   (If ModelMetadata does not expose context_length yet, adding it would also help consumers
   validate MaxSequenceLength up front.)

2. Make the probe non-fatal, so a future ORT validation change cannot break generation again:

   private static int ResolveVisionInputTokenCount(
       IVisionGenerationRuntime runtime, string prompt, IVisionInputs inputs)
   {
       var fallback = runtime.CountPromptTokens(prompt);
       try
       {
           using var probe = runtime.CreateProbeGenerator(probeMaxLength);
           probe.SetInputs(inputs);
           return ResolveInputTokenCount(name => { try { return probe.GetInputShape(name); } catch { return null; } }, fallback);
       }
       catch (Exception ex)
       {
           _logger.LogDebug(ex, "Vision input token probe failed; falling back to prompt token count.");
           return fallback;
       }
   }

Suggested regression test

An integration test that runs a real image prediction against Fara1.5-9B on CPU would have
caught this. A cheaper unit-level guard: assert that the value handed to CreateProbeGenerator
is <= context_length for a config stub declaring context_length: 32768.

Environment

• ElBruno.LocalLLMs 0.20.11
• Microsoft.ML.OnnxRuntimeGenAI 0.15.1 (CPU)
• .NET 10, Windows 11 x64
• Model: Fara1.5-9B (auto-downloaded, context_length: 32768)
• Consumer: elbruno/ElBruno.MagenticUI

Related

• Vision: max_length ignores image tokens, so MaxOutputTokens always fails with 'input_ids size exceeds max length' #44 (the fix this probe was introduced for — the fix itself is correct, only the probe is wrong)
• ElBruno.LocalLLMs 0.20.11 ships AssemblyVersion 0.20.9.0, breaking BlazorComponents/Rag 0.20.11 at startup #49 (AssemblyVersion mismatch in the same release)

[elbruno]

Additional evidence: the max_length ... cannot be greater than model context_length validation exists in both Microsoft.ML.OnnxRuntimeGenAI 0.14.1 and 0.15.1 native binaries:

$p = "$env:USERPROFILE\.nuget\packages\microsoft.ml.onnxruntimegenai"
foreach ($v in '0.14.1','0.15.1') {
  $d = Get-Item "$p\$v\runtimes\win-x64\native\onnxruntime-genai.dll"
  $t = [System.Text.Encoding]::ASCII.GetString([System.IO.File]::ReadAllBytes($d.FullName))
  "{0}: {1}" -f $v, $t.Contains("cannot be greater than model context_length")
}
# 0.14.1: True
# 0.15.1: True

So this is not an ORT version regression — the probe can never have succeeded against a real model whose genai_config.json declares a context_length, and consumers cannot dodge it by pinning an older GenAI package.

That also suggests the existing unit tests pass because they substitute IVisionGenerationRuntime, so CreateProbeGenerator never reaches native code. A single end-to-end test that loads a real model and predicts from an image would catch this class of bug.

[elbruno]

Fixed in main by d78ddbd (included in the 0.20.12 release line). Vision probing now clamps its probe max_length to the configured model context length and falls back to prompt tokenization if probing fails, preventing the 2147483647 validation error. Verified with the targeted OnnxVisionModel regression suite: 12 passed.