From NVIDIA Megatron-LM for visibility by RaymondLi0 · Pull Request #18 · bigcode-project/Megatron-LM

RaymondLi0 · 2023-01-24T20:01:13Z

No description provided.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Yuzhong Wang <yuzhongw@nvidia.com>

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Signed-off-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Kunlun Li <94586211+kunlunl@users.noreply.github.com> Co-authored-by: Kunlun Li <kunlunl@cw-dfw-cs-001-login-02.cm.cluster>

Co-authored-by: William Dykas <wdykas@oci-hsg-cs-001-vscode-03.cm.cluster> Co-authored-by: root <root@nvl72065-T16.cm.cluster> Co-authored-by: root <root@nvl72163-T17.cm.cluster>

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Co-authored-by: Charlie Truong <chtruong@nvidia.com>

Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Signed-off-by: oliver könig <okoenig@nvidia.com>

…apsed duration for saving checkpoint before logging timer (#4263) Signed-off-by: Ankur Srivastava <your_verified_email@domain.com> Co-authored-by: Ankur Srivastava <your_verified_email@domain.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>

…ining script (#4390) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-authored-by: Jon Barker <jbarker@nvidia.com>

Signed-off-by: meg miranda <mmiranda@nvidia.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

…ules (#3779) Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

…ze schedules (#3779) (#4404) Signed-off-by: oliver könig <okoenig@nvidia.com>

…ules (#4411) Co-authored-by: Mikail Khona (NVIDIA) <mkhona@nvidia.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

…ated MIMO (#4801) Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com>

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Co-authored-by: Gao Deng <gdeng@login-lyris02.lyris.clusters.nvidia.com>

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Zhongbo Zhu <zhongboz@nvidia.com>

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

#4852) Signed-off-by: Cory Ye <cye@nvidia.com>

Signed-off-by: jinliangl <jinliangl@nvidia.com>

Co-authored-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>

…ed offloading (#4692)

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

Co-authored-by: Qi Zhang <qizhang@nvidia.com> Co-authored-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> Co-authored-by: tongliu <tongliu@nvidia.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>

…4358) Signed-off-by: Xiaowei Ren <xren@nvidia.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…_skip_fp8_weight_update_tensor (#4874)

…4786) Co-authored-by: Siddhartha Raman S <sraman@login-lyris02.lyris.clusters.nvidia.com> Co-authored-by: Xin Yao <xiny@nvidia.com> Co-authored-by: gautham-kollu <gkollu@nvidia.com> Co-authored-by: Siddhartha Raman S <sraman@login-lyris01.lyris.clusters.nvidia.com>

Signed-off-by: qiyuw <qiyuw@nvidia.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>

#4562) Signed-off-by: qiyuw <qiyuw@nvidia.com>

Signed-off-by: Maanu Grover <maanug@nvidia.com>

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

RaymondLi0 changed the base branch from multi-query-attention to before-merge June 20, 2023 20:12

RaymondLi0 changed the base branch from before-merge to multi-query-attention June 20, 2023 20:12

ko3n1g force-pushed the main branch from ad5a627 to cd71d4c Compare February 18, 2026 09:18

Phlip79 and others added 27 commits April 17, 2026 22:08

Add QK layernorm support for dot-product attention in MambaModel (#4067)

e15ec3c

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Yuzhong Wang <yuzhongw@nvidia.com>

Docs: improve docstrings and comments in example training loop (#4041)

75a2878

Update copy-pr-bot.yaml [skip ci]

86b7218

feat(ckpt): add --async-ckpt-use-cpu-shm argument (#4355)

9978968

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

cp: Fix UT timeout (#4310) (#4373)

664baa8

Signed-off-by: Charlie Truong <chtruong@nvidia.com> Co-authored-by: Kunlun Li <94586211+kunlunl@users.noreply.github.com> Co-authored-by: Kunlun Li <kunlunl@cw-dfw-cs-001-login-02.cm.cluster>

Fix RL reward due to stop token (#4096)

e4d3a4c

FA4 Inference (#4186)

76ac7c2

Co-authored-by: William Dykas <wdykas@oci-hsg-cs-001-vscode-03.cm.cluster> Co-authored-by: root <root@nvl72065-T16.cm.cluster> Co-authored-by: root <root@nvl72163-T17.cm.cluster>

Make param_index_map always use unpacked (full numel) offsets (#4328)

3315c86

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Add activation logging and tokens per expert logging (#3842)

8be1e79

Co-authored-by: Charlie Truong <chtruong@nvidia.com>

Fix RL to once again work with --skip-train (#4249)

98a51eb

Fix Megatron initialization with extra_args_provider (#4327)

afae25b

Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>

Rename MambaModel/MambaStack to HybridModel/HybridStack (#4099)

15e07a2

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

chore(beep boop 🤖): Bump (main) (2026-04-20)

3046182

fix(ci): wrap uv install in retry block (#4387)

9c210f7

Signed-off-by: oliver könig <okoenig@nvidia.com>

refactor(tests): move NCCL env vars from docker launcher to shell tra…

ef1888b

…ining script (#4390) Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Remove packed_attention_mask unused parameter (#3859)

b562151

Co-authored-by: Jon Barker <jbarker@nvidia.com>

Second batch of audit edits (#4115)

859b66a

Signed-off-by: meg miranda <mmiranda@nvidia.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

Replace rampup batch size scheduler with custom step batch size sched…

c9e03d0

…ules (#3779) Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Update copy-pr-bot.yaml [skip ci]

dc87858

revert: replace rampup batch size scheduler with custom step batch si…

a52112d

…ze schedules (#3779) (#4404) Signed-off-by: oliver könig <okoenig@nvidia.com>

Update copy-pr-bot.yaml [skip ci]

0b9bc20

Replace rampup batch size scheduler with custom step batch size sched…

532ad92

…ules (#4411) Co-authored-by: Mikail Khona (NVIDIA) <mkhona@nvidia.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Megatron-FSDP: log mcore detection only after imports succeed (#4400)

e5ec9ab

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

ci(gb200): re-enable tunable_overlap 1-node mr-github test (#4405)

a550e0e

Signed-off-by: oliver könig <okoenig@nvidia.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix local docs building (#4416)

77e2dd4

RL: Onload optimizer after logprobs computation (#4235)

e778967

kamran-nvidia and others added 30 commits May 20, 2026 22:49

MimoOptimizer: fix distributed checkpoint save and load for non-coloc…

68d64f7

…ated MIMO (#4801) Signed-off-by: Kamran Jafari <kjafarisadeg@nvidia.com>

Update copy-pr-bot.yaml [skip ci]

5df3e4e

Route non-Muon params through DistributedOptimizer (#4771)

0044db1

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

ci: Gate optional CI jobs with repository variables (#4907)

990fa69

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Fix tokenizers bug in nightly (#4833)

e69a1ea

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Allow optimizer CG to share the same pool as full-iter CG (#4698)

daec17c

Use sharded_state_dict_default in MLP.sharded_state_dict (#4693)

32a7e46

Co-authored-by: Gao Deng <gdeng@login-lyris02.lyris.clusters.nvidia.com>

Fix MTP recompute crash with packed sequences (#4593)

2b77d32

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Zhongbo Zhu <zhongboz@nvidia.com>

Update PR template (#4904)

c6096d9

ci: Update perf test to output logs for tests to pass (#4906)

0588bf7

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Also persist asymmetrical units for the MXFP8 transpose weight buffer. (

6576040

#4852) Signed-off-by: Cory Ye <cye@nvidia.com>

fix no_shard training convergency and add unittest for no_shard (#3754)

80a2d39

Signed-off-by: jinliangl <jinliangl@nvidia.com>

Move policy epoch stats to the message object (#4533)

e9a3184

Co-authored-by: Teodor-Dumitru Ene <teodord.ene@gmail.com>

Add a knob to throttle the max allowed inflight offload in fine grain…

9a7cd17

…ed offloading (#4692)

refactor(data): consolidate get_batch and enable PP for SFT THD (#4103)

5e4fc93

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

Allow YAML MoE configs to use model specs (#4822)

0d198cd

Move bert and t5 pretrain files (#4820)

547fb17

Update copy-pr-bot.yaml [skip ci]

e27607a

Paged Stashing (#4247)

f007db7

Co-authored-by: Qi Zhang <qizhang@nvidia.com> Co-authored-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> Co-authored-by: tongliu <tongliu@nvidia.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>

make FP4 param gather work with the mixed precisions in NVFP4 recipe (#…

4c63602

…4358) Signed-off-by: Xiaowei Ren <xren@nvidia.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>

fix: Fix multi-node functional test phase sync (#4924)

4db6fa4

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Perf tests (#4917)

686aa8c

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

fix(cuda_graphs): handle TE 2.15 removal of FP8GlobalStateManager.set…

0beaa53

…_skip_fp8_weight_update_tensor (#4874)

Fix paged stashing test submodules lookup (#4925)

6c1bd6e

Fix mxfp8 param gather numerical issue when DP overlap is off (#4800)

5f79118

Signed-off-by: qiyuw <qiyuw@nvidia.com> Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>

[MXFP8/FP4-param-gather] Post processing after forced param AG in eval (

08bad7a

#4562) Signed-off-by: qiyuw <qiyuw@nvidia.com>

ci: Update training script paths in BERT and T5 (#4939)

3fb34c6

Various training utils (#4872)

34560c4

Signed-off-by: Maanu Grover <maanug@nvidia.com>

ci: restore perf test torchrun logs (#4951)

f7f584d

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

From NVIDIA Megatron-LM for visibility#18

From NVIDIA Megatron-LM for visibility#18
RaymondLi0 wants to merge 7014 commits into
bigcode-project:multi-query-attentionfrom
NVIDIA:main

RaymondLi0 commented Jan 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

RaymondLi0 commented Jan 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants