Handle step key correctly in dp_reshardable checkpoint save with --optimizer-cpu-offload#69
Conversation
…-optimizer-cpu-offload` Graft of NVIDIA#2874 (f4502eb): wrap the optimizer `step` in LocalNonpersistentObject in sharded_param_state_dp_reshardable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The graft matches upstream exactly, but I traced the load path with
For 2 the easy fix is skipping (Only cpu-offload / torch AdamW runs are affected — FusedAdam keeps step in param_groups, which is also why no existing test catches this.) |
Grafts NVIDIA#2874 (
f4502eb1c9).Wraps the optimizer
stepinLocalNonpersistentObjectinsharded_param_state_dp_reshardableso it is not persisted into the rank-0common state.
🤖 Generated with Claude Code