Skip to content

feat(refit): add canonical S3 revision publisher - #612

Open
nv-hwoo wants to merge 1 commit into
hwoo/mx-delta-revision-prfrom
hwoo/mx-delta-publish-pr
Open

feat(refit): add canonical S3 revision publisher#612
nv-hwoo wants to merge 1 commit into
hwoo/mx-delta-revision-prfrom
hwoo/mx-delta-publish-pr

Conversation

@nv-hwoo

@nv-hwoo nv-hwoo commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Add the trainer-side publisher for canonical model revisions and exact-base deltas stored in S3. The publisher gathers canonical Hugging Face tensor identities across source ranks, computes deterministic XOR deltas from the installed base, uploads bounded compressed buckets, and publishes the immutable revision manifest through the revision catalog.

This is a parallel effort along the current refit API work. A separate follow-up PRs will merge this into the standard refit API.

Changes

  • Add the engine-agnostic refit API and publisher configuration.
  • Load and validate the launch Hugging Face snapshot as revision 0.
  • Capture a persistent canonical baseline from distributed trainer ranks.
  • Gather rank-local tensor metadata into one canonical model identity.
  • Compute exact-base tensor deltas using XOR.
  • Encode changed tensors into bounded zstd-compressed S3 buckets.
  • Record clean and dirty tensor coverage in a canonical delta index.
  • Upload buckets concurrently with configurable S3 worker and connection-pool limits.
  • Publish the final immutable revision manifest through the catalog.
  • Track publication timing, changed-weight density, and wire-byte metrics.
  • Support metadata-only revision 0 followed by versioned N -> N+1 updates.

Publication Flow

  1. Establish revision 0 from the launch checkpoint.
  2. Capture the canonical trainer baseline.
  3. Gather current model tensors across source ranks.
  4. Validate that the canonical format has not changed.
  5. Compute XOR deltas against the exact installed base.
  6. Upload changed tensors as bounded compressed buckets.
  7. Upload the canonical delta index.
  8. Publish the resulting revision as READY in the catalog.

The publisher owns canonicalization, delta construction, object publication, and revision identity. S3 carries the bytes; the catalog carries only immutable metadata.

Correctness

  • Publication requires the caller-provided base to match the publisher’s current version.
  • Tensor shape, dtype, size, format digest, base digest, and target digest are bound into the published identity.
  • Duplicate tensors gathered from different ranks must have identical target digests.
  • Every canonical tensor is classified as clean or dirty.
  • Every dirty tensor maps to exactly one bucket.
  • S3 object checksums and optional object versions are preserved in revision metadata.
  • Revision 0 carries launch identity without materializing an unnecessary payload.

Validation

  • Publisher API, distributed publication, and S3 payload tests: 23 passed
  • Full focused refit stack: 55 passed
  • pre-commit run --from-ref origin/main --to-ref hwoo/mx-delta-phase3-prs
  • cargo fmt
  • cargo clippy
  • cargo check

@copy-pr-bot
copy-pr-bot Bot deployed to automated-release August 12, 2026 15:29 Active
@copy-pr-bot
copy-pr-bot Bot deployed to automated-release August 12, 2026 15:29 Active
@github-actions github-actions Bot added the feat label Aug 12, 2026
Signed-off-by: Hyunjae Woo <hwoo@nvidia.com>
@nv-hwoo
nv-hwoo force-pushed the hwoo/mx-delta-publish-pr branch from 3d8d01f to 9ff238e Compare August 12, 2026 15:51
@copy-pr-bot
copy-pr-bot Bot deployed to automated-release August 12, 2026 15:51 Active
@copy-pr-bot
copy-pr-bot Bot deployed to automated-release August 12, 2026 15:51 Active
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant