Scalable Download Architecture

Queue-based background ZIP builder for downloading folders of any size

DOWNLOAD FLOW

rendering diagram…

flowchart TB
    A([User - ManageFolders]):::app
    B[POST - Request Download]:::svc
    C[(Downloads table<br/>Status - Pending)]:::data
    D[Download Builder<br/>systemd service]:::svc
    E[Set Status Processing]:::svc
    F[Download files<br/>from Blob Storage]:::data
    G[Write ZIP to disk<br/>ZipArchiveMode.Create]:::svc
    H[Upload ZIP to<br/>gmi-downloads container]:::data
    I[Generate SAS URL<br/>1 hour expiry]:::sec
    J[Set Status Completed]:::svc
    K([Browser downloads ZIP<br/>direct from Blob]):::app
    A --> B
    B --> C
    C -->|poll every 30s| D
    D --> E
    E --> F
    F --> G
    G --> H
    H --> I
    I --> J
    J -->|UI polls every 10s| A
    A --> K
classDef app fill:#1f2630,stroke:#C0894E,color:#F3EFE8;
classDef svc fill:#241c14,stroke:#E2B583,color:#F3EFE8;
classDef data fill:#22201a,stroke:#C0894E,color:#EBD3AE;
classDef ext fill:#1b1f27,stroke:#5A616B,color:#C4C0B8;
classDef sec fill:#1d2733,stroke:#5cc8e0,color:#eafaff;

Queue-based background ZIP builder decouples download from the HTTP request lifecycle.

The Problem

Grade My Investments generates large volumes of report files organized into folders. Users need to download entire folders as ZIP archives. The original implementation downloaded all files from blob storage, zipped them in memory inside the Azure Function, and streamed the result back to the Blazor client. This approach failed for large folders due to several hard constraints:

230s Timeout

Azure load balancer enforces a hard 230-second HTTP timeout. Large folders exceeded this limit during ZIP creation.

1.75 GB Memory

Azure App Service Plan has a 1.75 GB memory limit. In-memory ZIP creation for large folders exceeded this.

Browser Memory

Blazor WASM cannot hold large byte arrays in browser memory, causing crashes for large downloads.

Solution: Queue-Based Background ZIP Builder

The solution decouples the download process from the HTTP request lifecycle. Instead of building the ZIP synchronously during the API call, the system queues the download request, builds the ZIP on disk in a background service, uploads it to blob storage, and provides a direct download link.

1. Request

User clicks "Download Folder"

2. Queue

API creates Download record (Pending)

3. Build

Download Builder picks up & builds ZIP on disk

4. Upload

ZIP uploaded to blob storage

5. SAS URL

Generate time-limited download link

6. Download

User downloads directly from blob storage

Download Builder Console App

A .NET 10 console application that runs as a Linux systemd service on the report VM. It polls the database for pending download requests and processes them in batches.

CLI Commands:

Gmi.DownloadBuilder.Console start    # Start polling loop
Gmi.DownloadBuilder.Console stop     # Graceful shutdown
Gmi.DownloadBuilder.Console status   # Show pending count
Gmi.DownloadBuilder.Console purge-once  # Process one batch

Processing Steps:

Pick up pending downloads from database
Set status to Processing
Download files from blob storage sequentially
Write to ZIP file on disk (not memory)
Upload ZIP to gmi-downloads container
Create File record for billing/tracking
Generate SAS URL and set status to Completed

API Endpoints & Data Model

REST API Endpoints:

Method	Endpoint	Purpose
POST	`/accounts/{id}/downloads`	Request a folder download
GET	`/accounts/{id}/downloads`	List downloads with fresh SAS URLs
DELETE	`/accounts/{id}/downloads/{did}`	Delete download and mark ZIP for purge

Download Status Lifecycle:

Pending Processing Completed

Or on error: Failed

Downloads Table (MySQL):

Downloads
  DownloadID      BIGINT UNSIGNED PK
  AccountID       BIGINT UNSIGNED FK
  FolderID        BIGINT UNSIGNED
  FileID          BIGINT UNSIGNED FK (nullable)
  FolderName      VARCHAR(1024)
  TotalFiles      INT
  ProcessedFiles  INT
  Status          VARCHAR(32)
  DownloadUrl     TEXT (SAS URL)
  BlobPath        VARCHAR(1024)
  RequestedDateTime  DATETIME
  CompletedDateTime  DATETIME

Key Design Decisions

Disk-Based ZIP Building

ZIP files are built on disk using ZipFile.Open with ZipArchiveMode.Create, not in memory. This allows the service to handle folders of any size without memory constraints. The temporary ZIP file is cleaned up after upload to blob storage.

SAS URL Regeneration

SAS URLs are time-limited (1 hour) for security. When the UI requests the download list, the API regenerates fresh SAS URLs from the stored blob path and container name. This ensures users always get valid download links.

Polling Architecture

The Download Builder polls the database every 30 seconds for pending downloads and processes them in batches of 5. The Blazor UI polls the API every 10 seconds while downloads are pending or processing, auto-stopping when all downloads are complete.

Billing Integration

When a ZIP is built, a File record is created in the Files table with FreeStorage = true. This makes download ZIPs visible in the file system but not billed for storage. The File Purge service handles cleanup when downloads are deleted.

Direct Browser Download

Completed downloads open the SAS URL directly in a new browser tab via window.open(). The browser downloads the ZIP directly from Azure Blob Storage, completely bypassing the application server and Blazor memory.

Progress Tracking

The Download Builder updates the ProcessedFiles counter every 10 files during ZIP creation. The UI shows real-time progress (e.g., "245 of 496 files") by polling the API endpoint.

Resilience & Error Handling

The Download Builder is a long-running background service that must remain stable even when external conditions change (e.g., users deleting downloads while they are being processed). Several resilience patterns ensure the service never gets stuck or requires manual restarts.

Scoped DbContext Per Batch

Each processing batch creates a fresh DI scope via IServiceScopeFactory, giving it a new DbContext instance. This prevents a failed batch from leaving stale tracked entities that would poison subsequent batches with cascading DbUpdateConcurrencyException errors.

private async Task ProcessDownloadBatch(int batchSize)
{
    using var scope = _scopeFactory.CreateScope();
    var downloadService = scope.ServiceProvider
        .GetRequiredService<IDownloadService>();
    await downloadService
        .ProcessDownloadBatchAsync(batchSize);
}

Concurrency Handling

When a user deletes a download from the UI while it is being processed, the database row disappears. The repository catches DbUpdateConcurrencyException, detaches the stale entity from the change tracker, and continues processing the remaining downloads in the batch without interruption.

catch (DbUpdateConcurrencyException)
{
    // Row deleted by another process
    _context.ChangeTracker.Entries()
        .Where(e => e.Entity is Download d
            && d.DownloadID == download.DownloadID)
        .ToList()
        .ForEach(e => e.State = EntityState.Detached);
}

Error Status Resilience

When a download fails during processing, the service attempts to mark it as Failed in the database. If that update also fails (e.g., the row was deleted), the error is logged as a warning instead of crashing the batch, allowing remaining downloads to continue processing normally.

catch (Exception ex)
{
    try
    {
        download.Status = "Failed";
        await _downloadRepository
            .UpdateDownloadAsync(download);
    }
    catch (Exception updateEx)
    {
        _logger.LogWarning(updateEx,
            "Could not update download status");
    }
}