Skip to content

Persistent MXFP8 for experts layers in MoE#1940

Draft
kailash109 wants to merge 7 commits into
NovaSky-AI:mainfrom
kailash109:persistent-mxfp8
Draft

Persistent MXFP8 for experts layers in MoE#1940
kailash109 wants to merge 7 commits into
NovaSky-AI:mainfrom
kailash109:persistent-mxfp8

Conversation

@kailash109

Copy link
Copy Markdown

Building off of #1937 and #1898, introduce persistent MXFP8 quantization for expert layers in MoE

Reuses blockwise casting logic from #1898 to write E8M0 scale factors on 1x32 blocks & deferred load of FP8 weights so optimizer states initialized on FP32 master weights, and pack E8M0 scale factors into uint8 bytes for VLLM ModelOpt-MXFP8 offline quantization. _iter_sync_tensors which calls iter_serialized_fp8_tensors now handles both FP8 (hopper-style) and MXFP8 (blackwell-style).

Todo: I still observe perf degradations from mxfp8 vs the bf16 baseline, need to debug why
image

@erictang000

erictang000 commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

qq: how does the reward track vs the bf16 baseline here? also curious about logprobs mismatch

@kailash109

Copy link
Copy Markdown
Author

Reward curves basically identical
image

and logprob diff seems to be consistently 1.5x that of baseline
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants