Grade My Investments Technology Observatory
GMI · TECHNOLOGY OBSERVATORY // ALL SYSTEMS NOMINAL
ENGINEERED BY LEOPARD DATA

Disaster Recovery & Business Continuity

Layered continuity controls that protect Grade My Investments data and keep services self-healing — honestly separated into what is implemented today and what is on the roadmap.

CONTINUITY CONTROLS
rendering diagram…
flowchart TB
    subgraph IMPL[Implemented - Central US single region]
        direction TB
        DB[(MySQL prod<br/>35 day backups<br/>point-in-time restore)]:::data
        BLOB[(Blob Storage ZRS<br/>3 availability zones)]:::data
        KV[(Key Vault<br/>soft-delete 90d<br/>purge protection)]:::sec
        SVC[Background services<br/>systemd Restart=always<br/>RestartSec=10]:::svc
        HM[Health Monitor<br/>checks every 60s<br/>emails admins]:::svc
        REC[Idempotent billing<br/>FailedPaymentReconciliation<br/>no double-charge]:::svc
        ROLL[Deployment backups<br/>opt-gmi-Backup-TIMESTAMP<br/>stop-restore-restart]:::svc
    end
    HM -->|watches| SVC
    SVC -->|reads-writes| DB
    SVC -->|reads-writes| BLOB
    SVC -->|secrets| KV
    REC -->|reconciles| DB
    ROLL -->|recovers| SVC
    subgraph ROAD[Roadmap - not yet implemented]
        direction TB
        MR[Multi-region failover<br/>secondary region]:::ext
        HA[Database HA<br/>zone-redundant auto failover]:::ext
        RT[Formal RTO-RPO SLA targets]:::ext
        GEO[Cross-region<br/>geo-redundant backup]:::ext
    end
    IMPL -.->|future| ROAD
classDef app fill:#1f2630,stroke:#C0894E,color:#F3EFE8;
classDef svc fill:#241c14,stroke:#E2B583,color:#F3EFE8;
classDef data fill:#22201a,stroke:#C0894E,color:#EBD3AE;
classDef ext fill:#1b1f27,stroke:#5A616B,color:#C4C0B8;
classDef sec fill:#1d2733,stroke:#5cc8e0,color:#eafaff;
Implemented continuity controls protect data and self-heal services within a single Central US region; roadmap items extend toward multi-region resilience.

This page distinguishes implemented controls running in production today from forward-looking roadmap items that are not yet built. Grade My Investments currently operates in a single region (Central US); there is no secondary region failover today.

Implemented Today

Database Backups & Recovery

The production MySQL database is protected by Azure automated backups:

  • Automated backups Enabled
  • Retention window 35 days
  • Point-in-time restore Supported

Point-in-time restore allows the database to be recovered to any moment within the 35-day retention window, protecting against accidental deletion, corruption, or bad deployments.

Redundant Blob Storage

User files and generated reports live in Azure Blob Storage with environment-appropriate redundancy:

Environment Redundancy Copies
Production Zone-Redundant (ZRS) 3 copies across 3 availability zones
Development Locally-Redundant (LRS) 3 copies in a single datacenter

ZRS in production keeps data available even if an entire availability zone within Central US becomes unavailable.

Secrets Protection

Azure Key Vault guards application secrets against accidental or malicious loss:

  • Soft-delete retains deleted secrets for 90 days
  • Purge protection blocks permanent deletion during the retention window
  • Recoverable even after an erroneous delete operation

Self-Healing Services

All background services run under Linux systemd with automatic restart on failure:

[Service]
Restart=always
RestartSec=10

If a worker process crashes, systemd automatically restarts it after 10 seconds, without manual intervention. This applies to all long-running background services (report builders, download builders, billing, and notification workers).

Health Monitoring & Alerting

  • Health Monitor check interval Every 60 seconds
  • Admin email on failure Enabled

The Health Monitor polls service health every 60 seconds and emails administrators the moment a failure is detected, so issues are caught quickly rather than discovered by users.

Idempotent Billing

Billing is idempotent: a FailedPaymentReconciliation process safely retries failed payments so that a retry can never double-charge a customer. Recovery from a transient payment-provider outage will not produce duplicate charges.

Retries are reconciled against prior attempts before any charge is committed.

Deployment Rollback

Every deployment captures a timestamped backup of the previous release:

/opt/gmi/Backup/<TIMESTAMP>/
Documented rollback procedure:
  1. Stop the affected service
  2. Restore the prior build from the timestamped backup
  3. Restart the service

Implemented Controls Summary

Control Detail Status
MySQL automated backups 35-day retention, point-in-time restore Live
Blob Storage redundancy ZRS across 3 availability zones (prod), LRS (dev) Live
Self-healing services systemd Restart=always, RestartSec=10 Live
Health Monitor 60-second checks, admin email on failure Live
Key Vault protection 90-day soft-delete + purge protection Live
Idempotent billing FailedPaymentReconciliation, no double-charge Live
Deployment rollback /opt/gmi/Backup/<TIMESTAMP>/ stop-restore-restart Live
Region footprint Single region: Central US Single region

Roadmap — Not Yet Implemented

The following items are forward-looking. They are not in place today and are documented here for transparency.

Multi-Region Failover

No secondary region exists today. A future secondary region would allow failover if Central US becomes unavailable.

Roadmap
Database High Availability

The current database tier is Burstable. Zone-redundant automatic failover HA is a future upgrade.

Roadmap
Formal RTO / RPO SLA

No formally documented recovery-time and recovery-point objective SLA targets exist yet.

Roadmap
Geo-Redundant Backup

Cross-region geo-redundant backup is not configured today; backups remain within the single region.

Roadmap