Skip to content

[Runtime] Add SEA-LION v4 32B source runtime#675

Draft
ankrovv wants to merge 4 commits into
ome-projects:mainfrom
ankrovv:codex/qwen-sealion-v4-public-runtime
Draft

[Runtime] Add SEA-LION v4 32B source runtime#675
ankrovv wants to merge 4 commits into
ome-projects:mainfrom
ankrovv:codex/qwen-sealion-v4-public-runtime

Conversation

@ankrovv

@ankrovv ankrovv commented Jul 17, 2026

Copy link
Copy Markdown

What this PR does

Adds the public OME source bundle for aisingapore/Qwen-SEA-LION-v4-32B-IT with a dedicated runtime:

  • Adds the exact ClusterBaseModel and sample InferenceService.
  • Adds vllm-qwen-sea-lion-v4-32b-it, selected explicitly and autoSelect:false.
  • Uses official public lightseekorg/smg:1.8.0 and vllm/vllm-openai:v0.25.1-cu129, TP2, the artifact's
    40,960 total-sequence cap, Qwen3 reasoning parsing, and Hermes tool parsing.
  • Sets SEA-LION's non-thinking default explicitly while allowing request-level thinking opt-in.
  • Registers the model and runtime in their kustomizations.

The dedicated runtime is required because SEA-LION's checkpoint defaults to non-thinking while base
Qwen/Qwen3-32B defaults to thinking. Reusing or editing a generic Qwen3 runtime would either misroute SEA-LION
answers into reasoning or change the existing base model's behavior.

Runtime arguments

--max-model-len=40960
--reasoning-parser=qwen3
--default-chat-template-kwargs={"enable_thinking":false}
--enable-auto-tool-choice
--exclude-tools-when-tool-choice-none
--tool-call-parser=hermes

These match the pinned artifact's default and Hermes-style <tool_call> template. qwen3_xml is for a different
Qwen3-Coder tool grammar.

Relationship to #628

This PR is semantically independent of #628's generic Qwen3-32B runtime: it adds a model-specific AI Singapore
runtime, and this sample pins it explicitly. Both PRs still touch shared kustomizations, and #628 currently needs
rebase/conflict and review resolution before either merge order can be treated as clean. If both land after that
coordination, neither runtime is made a global default by this PR.

Why we need it

The Flash release process makes public OME the source of truth for generic container versions, engine arguments,
formats, size ranges, probes, and validation assumptions. The first exact H100 Runtime CI run at TP2/4/8 proved
the model-specific default mismatch and drove this dedicated contract. Startup, BF16 loading, short benchmark,
substantial context, and explicit thinking passed. The first corrected dispatch reached no GPU because Runtime CI
incorrectly required auto-selection from this intentionally explicit-only runtime. PR #107 now adds the
explicit-runtime admission path, pinned-revision storage, a strict 40,960 boundary probe, and strict tool-loop
semantics before launching a fresh matrix:
Runtime CI PR #107.

Promotion blockers remain:

  1. The dedicated runtime uses the artifact/vLLM ceiling of 40,960. The model author calls 32,768 the native
    context, so the full 40,960 boundary must pass exact capacity/performance validation before promotion.
  2. SMG 1.8.0 and vLLM 0.25.1-cu129 change the full serving fingerprint. Re-run model load/CUDA-graph warmup and
    streaming/non-streaming none/auto/required/named/replay semantics on all 14 declared OCI accelerator lanes.
  3. The exact linux/amd64 images are digest-verified in the Chicago OCI source namespace; approved import and FRA
    replication still must complete before qualification.
  4. The target card advertises Qwen-SEA-LION-v4.5-27B-IT as its newer version. Confirm that v4-32B should be
    onboarded instead of, or alongside, that successor.

This draft is source preparation, not proof of production support.

How to test

  1. kubectl kustomize config/models
  2. kubectl kustomize config/runtimes
  3. pre-commit run --files config/runtimes/vllm/aisingapore/qwen-sea-lion-v4-32b-it-rt.yaml config/models/aisingapore/Qwen-SEA-LION-v4-32B-IT.yaml config/samples/isvc/aisingapore/qwen-sea-lion-v4-32b-it.yaml config/runtimes/kustomization.yaml config/models/kustomization.yaml
  4. make test
  5. Review corrected exact-model Runtime CI rows before promotion.

Checklist

  • Tests added/updated (N/A: config-only source bundle; rendered manifests and existing suites cover it)
  • Docs updated (N/A: model/runtime/sample manifests are the public contract)
  • Full make test passed after the final dedicated-runtime rename and argument correction

@github-actions github-actions Bot added runtime Runtime configuration changes models Model configuration changes config Configuration changes labels Jul 17, 2026
@ankrovv

ankrovv commented Jul 17, 2026

Copy link
Copy Markdown
Author

Agent-authored response:
Human input: none

Correction round after exact H100 evidence:

  • Converted the generic Qwen3 runtime into a dedicated SEA-LION runtime with explicit selection and autoSelect: false.
  • Added the checkpoint-specific non-thinking default and tool-choice-none prompt exclusion.
  • Set the dedicated total-sequence contract to 40,960, matching the artifact and vLLM derivation.
  • Updated the sample and runtime kustomization; this PR no longer supersedes or overlaps [Runtime] Add Qwen3-32B, Qwen3-Embedding-8B, and Qwen3Guard-Gen-8B vLLM runtimes #628 except for a possible mechanical kustomization rebase.
  • Re-ran kustomize, changed-file pre-commit, runtime-selector tests, and full make test; all pass.
  • Independent correctness and simplification reviews found no remaining config issue.

This stays draft. Public SMG 1.4.1 versus private production 1.5.0 requires an owner-reviewed skew exception or a published public 1.5.0 image before promotion.

@ankrovv

ankrovv commented Jul 20, 2026

Copy link
Copy Markdown
Author

Updated the candidate to the newest OCI Flash-promoted pair: official lightseekorg/smg:1.7.0 and vllm/vllm-openai:v0.24.0-cu129. This supersedes the former 1.4.1/0.23 image tuple; no runtime arguments or resource settings changed. Revalidation: changed-file pre-commit PASS, runtime Kustomize render PASS, full make test PASS, independent correctness review found no issues, and simplification review found no scope reduction. GPU/API support remains provisional until the fresh v0.24 Runtime CI matrix passes.

@ankrovv

ankrovv commented Jul 20, 2026

Copy link
Copy Markdown
Author

Image target updated at 60434a9 to the absolute latest official upstream pair requested for this Day-1 runtime:

  • docker.io/lightseekorg/smg:1.8.0
  • docker.io/vllm/vllm-openai:v0.25.1-cu129

The -cu129 suffix is intentional; unqualified vLLM v0.25.1 uses CUDA 13.0. SMG 1.8.0 preserves the image entrypoint python3 -m smg.launch_router, and the vLLM 0.25.1 source audit found every existing SEA-LION startup flag still supported, so this commit changes only the two image tags.

Validation after the update:

  • changed-file pre-commit: PASS
  • kubectl kustomize config/runtimes: PASS
  • full make test: PASS
  • git diff --check: PASS
  • independent correctness review: no findings
  • independent simplification review: no findings

The new fingerprints invalidate prior runtime qualification. OCI image replication and a fresh 14-lane A100-80G/H100/H200/B200 matrix remain required before support is claimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

config Configuration changes models Model configuration changes runtime Runtime configuration changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant