Grade My Investments Technology Observatory
GMI · TECHNOLOGY OBSERVATORY // ALL SYSTEMS NOMINAL
ENGINEERED BY LEOPARD DATA

Data Retention & Lifecycle

How Grade My Investments deletes user files safely with a transactional, two-phase purge that never orphans a blob or a database row.

FILE LIFECYCLE
rendering diagram…
flowchart TB
    A([User deletes file]):::app
    B[Set File.IsDeleted true<br/>blob still in storage]:::svc
    C[(Files table<br/>IsDeleted true<br/>PurgeTime null)]:::data
    D[FilePurge worker<br/>batch 50 - every 60s<br/>systemd Restart always]:::svc
    E[RetrieveFilesMarkedForPurgeAsync<br/>IsDeleted true AND PurgeTime null]:::svc
    F[Delete blob from<br/>Azure Storage]:::ext
    G[MarkFileAsPurgedAsync<br/>set PurgeTime]:::data
    H([File fully purged]):::app
    A --> B
    B --> C
    C -->|poll every 60s| D
    D --> E
    E --> F
    F -->|on blob delete success| G
    G --> H
classDef app fill:#1f2630,stroke:#C0894E,color:#F3EFE8;
classDef svc fill:#241c14,stroke:#E2B583,color:#F3EFE8;
classDef data fill:#22201a,stroke:#C0894E,color:#EBD3AE;
classDef ext fill:#1b1f27,stroke:#5A616B,color:#C4C0B8;
classDef sec fill:#1d2733,stroke:#5cc8e0,color:#eafaff;
A user delete only sets a flag; the FilePurge worker removes the blob first and marks the row purged only after the blob delete succeeds.

Two-Phase Deletion

File deletion is split into two distinct phases so that the user-facing action is instant while the expensive, irreversible work happens asynchronously in the background. This avoids blocking the request on Azure Storage calls and guarantees nothing is removed permanently until it can be done consistently.

Phase 1 — Soft Delete

When a user deletes a file, the application simply flips a flag:

  • File.IsDeleted = true
  • The blob still lives in Azure Storage
  • Returns immediately — no storage round-trip

The file disappears from the user's view instantly, but the underlying bytes remain recoverable until the purge worker runs.

Phase 2 — Hard Purge

The FilePurge worker later finalizes the deletion:

  • Removes the blob from Azure Storage
  • Sets File.PurgeTime in the database
  • Marks the row purged — only after the blob delete succeeds

Transactional ordering matters: the database row is updated after the blob is gone, so a failure can never leave the row marked purged while the blob still exists.

The FilePurge Worker

Gmi.FilePurge.Console is a .NET console application that runs the FilePurgeService on a continuous polling loop as a Linux systemd service with Restart=always, so it self-heals after a crash or VM reboot.

Defaults
  • Batch size 50 files
  • Poll interval 60 seconds
  • Restart policy always

CLI Commands

The console exposes a small command surface for operations and manual control:

  • start Begin the continuous polling loop
  • stop Graceful shutdown of the worker
  • status Report worker state and pending counts
  • purge-once Process a single batch and exit

Repository Contract

The worker talks to the database through a focused repository contract. The query selects only files that are soft-deleted but not yet purged, and the mark operation stamps the purge time once the blob is gone.

Method Behaviour Predicate / Effect
RetrieveFilesMarkedForPurgeAsync(limit) Fetch the next batch to purge IsDeleted = true PurgeTime is null
MarkFileAsPurgedAsync(file) Finalize after blob delete Sets PurgeTime

Robust Blob Handling

The purge logic is defensive about the data it receives, because a stored URL may be old, from a different environment, or malformed.

  • Extracts the blob name directly from the stored URL
  • Supports Azure Storage URL format (production)
  • Supports Azurite URL format (local development)
  • Tolerates missing or malformed URLs without crashing the batch

Related: GDPR Account Export

Data lifecycle is not only about removal. A full account export for GDPR compliance reuses the same asynchronous download pipeline that powers folder downloads, packaging a user's data on disk in the background rather than in memory.

See Scalable Downloads for the underlying queue-based builder.

Roadmap