GLM-5.3 sampling example

[sfc-gh-jrasley]

Adds GLM-5.3 sampling job example using 8xH200 GPUs. Tested max context length I can fit within one node without running out of memory as well.

[snowflake-security-bot]

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

[snowflake-security-bot]

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

[sfc-gh-halu]

nit:
there's already glm-sampling.json almost identical, should we consolidate or rename to differentiate?

[sfc-gh-kganesan]

Tried this config on QA6 today (image dev_20260908_114912_ea093491acb, H200) while testing an OpenAI-compatible wrapper over Cortex sampling jobs. The sampling job doesn't come up — vLLM's engine core dies during startup, ~2m45s after job creation:

File "arctic_inference/server/worker.py", line 615, in initialize
    self.llm = AsyncLLM.from_vllm_config(
File "vllm/v1/engine/utils.py", line 1272, in wait_for_engine_startup
    raise RuntimeError(
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

I varied one knob at a time. All four failed identically:

kv_cache_dtype	reasoning_parser	job id	result
fp8 (as in this PR)	glm45	557e40f4-af8d-4839-abcc-a60ab7173073	engine core init failed
fp8_ds_mla	glm45	9a749007-a994-4021-8f1c-09a2819a2e54	same
fp8	removed	f9a59bcd-b9a7-4b6d-91f8-f650af47efca	same
auto (bf16)	glm45	f1f9a0ac-fb56-4148-9200-ffdca8241d48	same

Control on the same image, same hour: Qwen3-8B on 1 GPU reached RUNNING in 4m21s and served generations and tool calls fine. So this is specific to GLM-5.3, not the environment or client.

Likely root cause

GLM-5.3 is GlmMoeDsaForCausalLM (glm5_next) — DeepSeek sparse attention over rope-free (NoPE) MLA: qk_rope_head_dim=0, kv_lora_rank=512, qk_nope_head_dim=256. Upstream vLLM only added glm5_next in vllm-project/vllm#53906, and the vLLM recipe still tells users to run a dedicated image until that ships in a release. If the Cortex image's vLLM predates it there's no model implementation to load and the core dies exactly this way. Worth checking the vLLM and FlashInfer versions baked into the image — NoPE sparse MLA needs FlashInfer >= 0.6.18.

Two things in the config that will still bite once the image supports the arch

1. kv_cache_dtype: "fp8" is wrong for this model. On NoPE MLA, fp8 normalizes into the packed fp8_ds_mla layout, whose cache-write kernel hardcodes DeepSeek's 512 NoPE + 64 RoPE entry shape. GLM's pe_dim=0 violates that guard (pe_dim must be 64 for fp8_ds_mla). Sparse MLA at this geometry is bf16-KV-only today — see GLM-5.3-Flash (glm5_next): no SM120 sparse-MLA path for rope-free MLA (qk_rope_head_dim=0) — three failure modes on RTX PRO 6000 vllm-project/vllm#53963 and [Attention][Kernel] TRITON_MLA_SPARSE: NoPE dim_qk=512 geometry and SM120 support vllm-project/vllm#54031 ("fp8_ds_mla still needs the native FlashMLA kernel, so it stays out of this backend's supported set"). Suggest "kv_cache_dtype": "auto".

2. reasoning_parser: "glm45" is inert on this path. It's a vLLM OpenAI-server frontend argument, not an engine argument, and Cortex sampling goes through /generate rather than vLLM's OpenAI server. On the Qwen3 control the reasoning came back inline in the text (...</think>\n\n391), i.e. nothing strips it. Harmless, but the config implies a capability the endpoint doesn't actually have.

Also worth sanity-checking the sizing: zai-org/GLM-5.3 is 755.6 GB of natively fp8 weights, so 8xH200 at gpu_memory_utilization: 0.85 (~959 GB usable) leaves roughly 25 GB per GPU for KV cache and activations. For comparison, glm-sampling.json on main gives the smaller GLM-5.2-FP8 16 GPUs.

Two platform issues this surfaced (unrelated to this PR, noting for context)

• download-log returns only 3 driver lines. The engine core's stderr — the actual "root cause above" — never lands in the artifact, and Failed core proc(s): {} is empty, so isolating this took four 8-GPU launches instead of one log read.
• When the sampling sub-job fails, the parent job stays INITIALIZING while holding all 8 GPUs and never transitions to FAILED (updated_at freezes at the failure). Every attempt needed a manual cancel to get capacity back; the client meanwhile just blocks against job_ready_timeout, which defaults to 30 minutes.

Happy to re-run any of these once the image question is settled.

[snowflake-security-bot]

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

[snowflake-security-bot]

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.

[snowflake-security-bot]

Snowflake Security Review

Security grade: A — Passed ✅

This PR was classified as LOW risk by the automated pre-screen.