AsyncFileUploadDemo

A small demo of streaming file uploads end-to-end without buffering them in server memory.

The original 2014-era version (src/Web/) showed how to do this on .NET Framework 4.x with ASP.NET Web API 2 and MultipartFormDataStreamProvider. This repo now also contains a modern .NET 10 take (src/WebModern/) that pushes the same idea further: streaming straight off the socket into a destination — disk or Azure Blob Storage — with bounded memory regardless of file size.

If you want the long version of the original story, the blog post is still up: https://www.flapstack.blog/file-uploads-with-web-api-drop-the-3rd-party/.

---

Why streaming uploads matter

The default multipart/form-data story in most web stacks is "let the framework parse the body for you." That parser typically buffers each file — small ones in memory, larger ones spooled to a temp file on disk — before your handler ever sees them. For a demo that's fine. For a real upload service it's two problems:

1. Memory pressure. A handful of concurrent large uploads can blow your process heap or kick everything onto disk.
2. A wasted round trip. If the final destination is another store (S3, Azure Blob, a database), you've now copied every byte to local disk just to copy it back out.

The fix in both versions of this demo is the same shape: read the multipart body as a stream, hand each section's Stream straight to the destination, and never hold the whole file in memory.

---

Two implementations

src/Web/ — original (.NET Framework 4.x, ASP.NET Web API 2)

• Web API 2 controller with a custom MultipartFormDataStreamProvider that names each uploaded file with a GUID and writes it directly under ~/uploads.
• jQuery + Bootstrap + KnockoutJS front-end posting via XMLHttpRequest + FormData.
• IIS / IIS Express hosting via Web.config.
• Built with Cake (build.cake, build.ps1, build.sh).

This is preserved as a historical reference — open src/FileUploadDemo.sln in Visual Studio (Windows) to run it.

src/WebModern/ — modern (.NET 10, ASP.NET Core Minimal API)

A from-scratch rewrite using current .NET conventions:

• Minimal API in a single Program.cs (~90 LOC including helpers).
• MultipartReader directly on HttpRequest.Body — no IFormFile, no temp-file spooling, no ReadFormAsync. Each file section is just a Stream that yields bytes as they arrive from the client.
• Two endpoints sharing one ReadFileSectionsAsync helper that yields (filename, stream) pairs.
• Vanilla JS front-end (wwwroot/index.html) — native multi-file <input>, XHR for upload progress, system-ui CSS. No jQuery, no Bootstrap, no build step.
• Zero new NuGet dependencies beyond Azure.Storage.Blobs for the blob endpoint.

Endpoints

Route	Destination	Strategy
POST /api/upload	Local disk (./uploads/)	Stream each multipart section straight into File.Create(...) via CopyToAsync.
POST /api/upload/blob	Azure Blob Storage	Read fixed 8 MiB blocks from each section into pooled buffers; stage them in parallel (up to 4 concurrent StageBlockAsync calls) against a BlockBlobClient; finalize with CommitBlockListAsync.

The blob endpoint is the more interesting one. It demonstrates the full pattern when bytes must flow through your server (compliance, scanning, transformation) but you still want server-side memory to stay flat as file size grows: bytes flow in from the client and out to Azure concurrently, with at most BlockSize * ParallelBlocks (~32 MiB by default) held in memory per file.

If bytes don't have to go through your server, the genuinely fastest pattern is to mint a short-lived User Delegation SAS and have the browser upload direct to blob via the JS SDK. That's noted but not implemented here — see "What's deliberately not in this demo" below.

---

Running the modern version

Requires the .NET 10 SDK.

cd src/WebModern
dotnet run

Then open the URL Kestrel prints (typically http://localhost:5xxx). You'll see a small form with two submit buttons:

• Upload to disk — writes files under src/WebModern/uploads/.
• Upload to Azure Blob — requires a connection string (see below); otherwise returns 503 with a clear message.

Configuring Azure Blob

Use .NET user-secrets so credentials never enter the repo:

cd src/WebModern
dotnet user-secrets init
dotnet user-secrets set ConnectionStrings:AzureBlob "UseDevelopmentStorage=true"   # Azurite
# or:
dotnet user-secrets set ConnectionStrings:AzureBlob "DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net"

For local dev, Azurite (the official local Azure Storage emulator) is the easiest path. The container name defaults to uploads and can be overridden with the AzureBlob:Container config key.

---

How the streaming path actually works

The core helper, in Program.cs:

static async IAsyncEnumerable<(string FileName, Stream Body)> ReadFileSectionsAsync(
    Stream body, string boundary, [EnumeratorCancellation] CancellationToken ct)
{
    var reader = new MultipartReader(boundary, body);
    for (var section = await reader.ReadNextSectionAsync(ct);
             section is not null;
             section = await reader.ReadNextSectionAsync(ct))
    {
        if (!ContentDispositionHeaderValue.TryParse(section.ContentDisposition, out var cd)
            || !cd.IsFileDisposition()) continue;
        yield return (cd.FileName.Value?.Trim('"') ?? "file", section.Body);
    }
}

MultipartReader reads the multipart envelope and hands you each section as it appears. section.Body is a non-seekable forward-only Stream over the bytes of that file — read it once, in order, no rewinding. That constraint is exactly what makes streaming uploads bounded-memory: there's no place to buffer.

The disk endpoint then does:

await body.CopyToAsync(fileStream, ct);

…and the blob endpoint does block-by-block:

while (true)
{
    var buf = ArrayPool<byte>.Shared.Rent(BlockSize);
    var read = await ReadFullAsync(body, buf, BlockSize, ct);
    if (read == 0) break;
    // Stage this block asynchronously; semaphore caps concurrency.
    await gate.WaitAsync(ct);
    staging.Add(Task.Run(async () => { /* StageBlockAsync, return buffer to pool */ }));
}
await Task.WhenAll(staging);
await blob.CommitBlockListAsync(blockIds, ct);

A few small things worth knowing:

• Block size = 8 MiB. Azure's block ceiling is 4000 MiB; the practical sweet spot is 4–16 MiB. Too small and the per-block overhead dominates; too large and you pay latency on errors and waste memory.
• Parallelism = 4 stages. Plenty for one upload; tune with ParallelBlocks if you measure a bottleneck.
• Block IDs must be fixed-width when committing — Azure enforces it. The demo encodes a counter into a 12-byte buffer, base64'd to a 16-char ID.
• Kestrel limits are removed for the demo (MaxRequestBodySize = null, MultipartBodyLengthLimit = long.MaxValue). In production you'd cap these — the framework's default 30 MB cap exists for good reasons.

---

What's deliberately not in this demo

The README would be longer than the code if every "and then in production…" were implemented. Some omissions called out so they're explicit:

• Auth. Anyone who can reach the endpoint can upload. Put it behind ASP.NET Core authentication / authorization before anything real.
• Antivirus / content scanning. Streaming straight to the destination means you can't easily scan first. Either scan after-the-fact on a queue, or pipe through a scanner mid-stream.
• Server-side type sniffing. The demo trusts the client-supplied extension. For untrusted users, sniff magic bytes server-side and reject mismatches.
• Per-file size limits. Reinstate MaxRequestBodySize and MultipartBodyLengthLimit to something sane.
• Retry tuning. BlobClientOptions.Retry is left at defaults; tune for your network reality.
• Browser → Azure direct (SAS upload). Fastest pattern when bytes don't need to traverse your server: mint a short-lived User Delegation SAS server-side, have the browser POST blocks straight to Azure via @azure/storage-blob. Not implemented here because the stated demo intent is "stream through the server without buffering."
• The old src/Web/. Kept intentionally for historical comparison. Not maintained.

---

Repo layout

src/
  Web/          # original .NET Framework 4.x demo (historical)
  WebModern/    # .NET 10 minimal API + vanilla JS front-end
build.cake      # builds src/Web/ — not used by WebModern
build.ps1
build.sh

The modern project builds with dotnet build / dotnet run. The Cake scripts are only for the original Framework project.