Skip to content

Download artifacts as parallel byte ranges that resume after a dropped connection - #47

Open
joshuaswarren wants to merge 2 commits into
omacom:mainfrom
joshuaswarren:perf/ranged-payload-download
Open

joshuaswarren wants to merge 2 commits into
omacom:mainfrom
joshuaswarren:perf/ranged-payload-download

Conversation

@joshuaswarren

Copy link
Copy Markdown
Contributor

On an M2 Max, payload part00 (1.99 GB) came down at 2.9 MB/s on one stream for 11 minutes. Part01 then came down at 103 MB/s. Each part is one downloadTask with no Range and no resume, so one slow connection sets the pace, and a dropped one starts the part over.

RangedArtifactDownloader fetches 64 MiB ranges over four sessions and writes them in place. It resumes a dropped range from its last byte, and it falls back to the single-stream downloader when the server ignores Range. A range is accepted only as a 206 for exactly the requested bytes of an object of the pinned size. The stager's size and SHA-256 checks per part and for the whole payload are unchanged.

Live, on a Mac Studio on 2026-10-07, part01 of the team-preview payload (1,546,348,246 B) from GitHub release assets, SHA-256 checked against the catalog on every run:

MB/s
main, single stream 74.0, 130.4
this branch, ranged 241.3, 165.2, 253.0

Three new tests run against a local HTTP/1.1 range server and pass three runs in a row:

  • testRangedDownloadResumesADroppedRangeFromItsLastByte: the second request is bytes=150000-299999.
  • testRangesRunInParallelOverSeparateConnectionsAndProgressReachesTheTotal: 16 ranges, at least two in flight on at least two connections, and progress ends at the size and never goes down.
  • testServerThatIgnoresRangeFallsBackToOneStream.

Main's downloader fails the resume case with NSURLErrorDomain -1005, "The network connection was lost."

Checked at bb31f19, on main 27bdf16 (no new commits upstream), on a Mac Studio with macOS 26.6.2, Xcode 27.0 and Swift 6.4:

  • swift-format lint --strict: exit 0.
  • Debug swift test: exit 0 (UXCore 147, Lifecycle 3, TrustCore 522 with the 1 existing skip, App 8).
  • Release swift test: exit 0 (136, 3, 522, 5).
  • ./test/all: exit 0.

Risks:

  • It opens four connections to one host instead of one. GitHub and Fastly served every range as 206 here. A publisher that limits connections per client could throttle or refuse. A refusal (a status other than 206 or 200) fails the attempt and does not fall back.
  • I measured the speedup on a warm object from one host, in a handful of runs. I did not reproduce the M2's original case of one connection stuck at 2.93 MB/s.
  • Resume works within one run only. An app restart still restarts an unfinished part.
  • If a server ignores Range after some ranges have started, progress steps back once when the single-stream fallback starts from 0.

To go back to one stream and keep the code, set downloader = ProgressReportingArtifactDownloader() in VerifiedArtifactStager.init(). One line.

…d connection

The payload part00 (1.99 GB) came down at 2.9 MB/s on one stream on an M2 Max,
and part01 at 103 MB/s minutes later. One slow or dropped connection stalled or
restarted a whole part. RangedArtifactDownloader fetches 64 MiB ranges over four
sessions, writes them in place, resumes a dropped range from its last byte, and
falls back to the single-stream downloader when a server ignores Range. The
stager still checks size and SHA-256 per part and for the assembled payload.
@joshuaswarren
joshuaswarren force-pushed the perf/ranged-payload-download branch from bb31f19 to db8a031 Compare October 8, 2026 23:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant