[Runtime] Add SEA-LION v4 32B source runtime#675
Conversation
|
Agent-authored response: Correction round after exact H100 evidence:
This stays draft. Public SMG 1.4.1 versus private production 1.5.0 requires an owner-reviewed skew exception or a published public 1.5.0 image before promotion. |
|
Updated the candidate to the newest OCI Flash-promoted pair: official |
|
Image target updated at
The Validation after the update:
The new fingerprints invalidate prior runtime qualification. OCI image replication and a fresh 14-lane A100-80G/H100/H200/B200 matrix remain required before support is claimed. |
What this PR does
Adds the public OME source bundle for
aisingapore/Qwen-SEA-LION-v4-32B-ITwith a dedicated runtime:ClusterBaseModeland sampleInferenceService.vllm-qwen-sea-lion-v4-32b-it, selected explicitly andautoSelect:false.lightseekorg/smg:1.8.0andvllm/vllm-openai:v0.25.1-cu129, TP2, the artifact's40,960 total-sequence cap, Qwen3 reasoning parsing, and Hermes tool parsing.
The dedicated runtime is required because SEA-LION's checkpoint defaults to non-thinking while base
Qwen/Qwen3-32Bdefaults to thinking. Reusing or editing a generic Qwen3 runtime would either misroute SEA-LIONanswers into
reasoningor change the existing base model's behavior.Runtime arguments
These match the pinned artifact's default and Hermes-style
<tool_call>template.qwen3_xmlis for a differentQwen3-Coder tool grammar.
Relationship to #628
This PR is semantically independent of #628's generic Qwen3-32B runtime: it adds a model-specific AI Singapore
runtime, and this sample pins it explicitly. Both PRs still touch shared kustomizations, and #628 currently needs
rebase/conflict and review resolution before either merge order can be treated as clean. If both land after that
coordination, neither runtime is made a global default by this PR.
Why we need it
The Flash release process makes public OME the source of truth for generic container versions, engine arguments,
formats, size ranges, probes, and validation assumptions. The first exact H100 Runtime CI run at TP2/4/8 proved
the model-specific default mismatch and drove this dedicated contract. Startup, BF16 loading, short benchmark,
substantial context, and explicit thinking passed. The first corrected dispatch reached no GPU because Runtime CI
incorrectly required auto-selection from this intentionally explicit-only runtime. PR #107 now adds the
explicit-runtime admission path, pinned-revision storage, a strict 40,960 boundary probe, and strict tool-loop
semantics before launching a fresh matrix:
Runtime CI PR #107.
Promotion blockers remain:
context, so the full 40,960 boundary must pass exact capacity/performance validation before promotion.
streaming/non-streaming none/auto/required/named/replay semantics on all 14 declared OCI accelerator lanes.
replication still must complete before qualification.
Qwen-SEA-LION-v4.5-27B-ITas its newer version. Confirm that v4-32B should beonboarded instead of, or alongside, that successor.
This draft is source preparation, not proof of production support.
How to test
kubectl kustomize config/modelskubectl kustomize config/runtimespre-commit run --files config/runtimes/vllm/aisingapore/qwen-sea-lion-v4-32b-it-rt.yaml config/models/aisingapore/Qwen-SEA-LION-v4-32B-IT.yaml config/samples/isvc/aisingapore/qwen-sea-lion-v4-32b-it.yaml config/runtimes/kustomization.yaml config/models/kustomization.yamlmake testChecklist
make testpassed after the final dedicated-runtime rename and argument correction