Repository navigation
Download artifacts as parallel byte ranges that resume after a dropped connection - #47
Open
joshuaswarren wants to merge 2 commits into
Open
joshuaswarren wants to merge 2 commits into
joshuaswarren wants to merge 2 commits into
Conversation
…d connection The payload part00 (1.99 GB) came down at 2.9 MB/s on one stream on an M2 Max, and part01 at 103 MB/s minutes later. One slow or dropped connection stalled or restarted a whole part. RangedArtifactDownloader fetches 64 MiB ranges over four sessions, writes them in place, resumes a dropped range from its last byte, and falls back to the single-stream downloader when a server ignores Range. The stager still checks size and SHA-256 per part and for the assembled payload.
joshuaswarren
force-pushed
the
perf/ranged-payload-download
branch
from
October 8, 2026 23:35
bb31f19 to
db8a031
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On an M2 Max, payload part00 (1.99 GB) came down at 2.9 MB/s on one stream for 11 minutes. Part01 then came down at 103 MB/s. Each part is one
downloadTaskwith no Range and no resume, so one slow connection sets the pace, and a dropped one starts the part over.RangedArtifactDownloaderfetches 64 MiB ranges over four sessions and writes them in place. It resumes a dropped range from its last byte, and it falls back to the single-stream downloader when the server ignores Range. A range is accepted only as a 206 for exactly the requested bytes of an object of the pinned size. The stager's size and SHA-256 checks per part and for the whole payload are unchanged.Live, on a Mac Studio on 2026-10-07, part01 of the team-preview payload (1,546,348,246 B) from GitHub release assets, SHA-256 checked against the catalog on every run:
Three new tests run against a local HTTP/1.1 range server and pass three runs in a row:
testRangedDownloadResumesADroppedRangeFromItsLastByte: the second request isbytes=150000-299999.testRangesRunInParallelOverSeparateConnectionsAndProgressReachesTheTotal: 16 ranges, at least two in flight on at least two connections, and progress ends at the size and never goes down.testServerThatIgnoresRangeFallsBackToOneStream.Main's downloader fails the resume case with
NSURLErrorDomain -1005, "The network connection was lost."Checked at bb31f19, on main 27bdf16 (no new commits upstream), on a Mac Studio with macOS 26.6.2, Xcode 27.0 and Swift 6.4:
swift-format lint --strict: exit 0.swift test: exit 0 (UXCore 147, Lifecycle 3, TrustCore 522 with the 1 existing skip, App 8).swift test: exit 0 (136, 3, 522, 5)../test/all: exit 0.Risks:
To go back to one stream and keep the code, set
downloader = ProgressReportingArtifactDownloader()inVerifiedArtifactStager.init(). One line.