Grade My Investments Technology Observatory
GMI · TECHNOLOGY OBSERVATORY // ALL SYSTEMS NOMINAL
ENGINEERED BY LEOPARD DATA

Platform Resiliency & Fault Tolerance

How Grade My Investments stays reliable through retry logic, graceful degradation, caching, and self-healing infrastructure.

Resiliency Philosophy

Grade My Investments is built on the principle that failures are inevitable — outages are not. Every external dependency (APIs, databases, payment processors, email services) can and will fail transiently. The platform is designed to absorb these failures transparently through layered retry logic, caching, graceful degradation, and self-healing services.

Key metric: A single FMP API rate limit or database connection hiccup does not result in user-visible errors. The platform retries, falls back, or caches — the user never knows.
RESILIENCY LAYERS
rendering diagram…
flowchart LR
    EXT([External Request<br/>FMP - Stripe - Email])
    RET{Transient<br/>Error?}
    BO[Exponential Backoff<br/>3 retries - 1s-2s-4s]
    CACHE[(FMP Cache<br/>5-min in-memory)]
    DB[(MySQL<br/>EF Retry x3)]
    SVC[11 Background Services<br/>systemd Restart=always]
    IDEM{Already<br/>processed?}
    SKIP[Skip - idempotent]
    DONE[Operation<br/>Complete]
    GD[Graceful Degradation<br/>partial result returned]
    EXT --> RET
    RET -- yes --> BO --> RET
    RET -- no - client error --> GD
    RET -- success --> CACHE
    CACHE --> DB
    DB --> SVC
    SVC --> IDEM
    IDEM -- yes --> SKIP
    IDEM -- no --> DONE
    classDef app fill:#1f2630,stroke:#C0894E,color:#F3EFE8;
    classDef svc fill:#241c14,stroke:#E2B583,color:#F3EFE8;
    classDef data fill:#22201a,stroke:#C0894E,color:#EBD3AE;
    classDef ext fill:#1b1f27,stroke:#5A616B,color:#C4C0B8;
    classDef sec fill:#1d2733,stroke:#5cc8e0,color:#eafaff;
    class EXT ext
    class RET,IDEM app
    class BO,GD svc
    class CACHE,DB data
    class SVC,DONE,SKIP app
From rate-limit retry through caching and self-healing to idempotent operations.

1. Exponential Backoff Retry

All external API calls use exponential backoff retry with intelligent error classification. Only transient failures trigger retries — client errors (400, 401, 403, 404) fail immediately since retrying won't help.

FMP Financial Data API
ParameterValue
Max retries3
Backoff schedule1s → 2s → 4s
Retry on429 (rate limit), 5xx (server errors)
Don't retry on400, 401, 403, 404 (client errors)
CoverageAll 23 API endpoints in FmpApiClient + FMP Proxy
SendGrid Email Delivery
ParameterValue
Max retries3
Backoff schedule1s → 2s → 4s (exponential)
Retry on429 (rate limit), 5xx (server errors)
After max retriesMarked as permanently failed with error details preserved
Twilio SMS Delivery
ParameterValue
Max retries3
Backoff schedule1s → 2s → 4s (exponential)
Retry onAny Twilio exception
After max retriesMarked as permanently failed with error preserved
Stripe Payment Processing
ParameterValue
Payment recording retry2 attempts with immediate retry
On complete failureFailedPaymentReconciliation record created for manual resolution
Double-charge preventionPlaceholder payment returned to prevent re-billing

2. Database Connection Resilience

Azure MySQL Flexible Server connections can drop due to idle timeouts, planned failovers, or network blips. Every database context across all 11 background services and the Functions API uses EF Core's built-in retry strategy.

EnableRetryOnFailure(maxRetryCount: 3, maxRetryDelay: 5s)
Services with database retry enabled:
  • Azure Functions API
  • Report Engine
  • Report Generator
  • Claude Analysis
  • Email Sender
  • SMS Sender
  • Schedules Engine
  • Download Builder
  • File Purge
  • Health Monitor
  • Report Comparison
  • Billing Engine

3. FMP API Response Caching

The FMP proxy caches API responses in memory for 5 minutes. This reduces API call costs, prevents rate limiting, and provides faster responses for frequently requested data.

ParameterValue
Cache duration5 minutes
Cache scopePer-endpoint, per-query (e.g., profile?symbol=AAPL)
Cache storageIn-memory ConcurrentDictionary
Stale cleanupEvery 100 requests, entries older than 5 min are evicted
ObservabilityX-Cache: HIT or X-Cache: MISS response header
Benefit10 users viewing AAPL = 1 FMP API call (not 10)

4. Self-Healing Background Services

All 11 background services run as systemd units with automatic restart policies. If a service crashes, it restarts within 10 seconds — no manual intervention required.

Restart=always
RestartSec=10

The Health Monitor service checks all services every 60 seconds and alerts admins via email if any service is down.

5. Idempotent Operations

Critical operations are designed to be safe to retry without side effects:

  • Billing runs: HasSuccessfulBillingRunForPeriodAsync() prevents duplicate monthly charges
  • Report charges: RetrieveJobChargeByJobIDAndAccountIDAsync() checks for existing charge before creating
  • Coupon usage: One-time-use enforcement per account per coupon, checked in a transaction
  • Credit card default swap: Database transaction prevents race conditions
  • Invoice creation: GetInvoiceForAccountAndBillingPeriodAsync() prevents duplicate invoices

6. Graceful Degradation

When external services are unavailable, the platform degrades gracefully rather than failing completely:

  • FMP API down: Reports generate with available data; missing sections show "data unavailable" instead of crashing
  • Claude AI unavailable: AI analysis features show error messages; core report generation continues without AI
  • Stripe down: Free accounts and coupon-covered reports still work; charges are recorded for later billing
  • SendGrid down: Emails queue in database; sender retries on next cycle
  • Blob storage unreachable: Service logs warning and uses placeholder mode for development
  • Dashboard analytics: Each section loads independently via parallel async calls; one section failing doesn't block others

7. Health Check Endpoints

Every web application exposes a lightweight /health.html endpoint for infrastructure monitoring and CI/CD pipeline smoke tests.

AppEndpointPurpose
Client Web/health.htmlPipeline smoke test, load balancer probe
Admin Web/health.htmlPipeline smoke test, load balancer probe
Technology Web/health.htmlPipeline smoke test, load balancer probe
Azure Functions/api/healthAPI availability check

These endpoints return instantly without loading the full Blazor WASM framework, solving B1 tier cold-start timeout issues in CI/CD pipelines.

8. Data Protection & Recovery

  • Database backups: 35-day automated retention with point-in-time restore (Azure MySQL)
  • Blob storage: Zone-Redundant Storage (ZRS) — data replicated across 3 availability zones
  • Deployment backups: Pipeline creates backups of console apps before each deployment
  • Payment reconciliation: If Stripe charges succeed but DB recording fails, FailedPaymentReconciliation table preserves the record for manual resolution
  • GDPR data export: Full account data export available at any time via async ZIP builder

9. API Rate Limiting

Token-bucket rate limiting protects the API from abuse and ensures fair resource distribution:

Client TypeLimitWindow
Authenticated users100 requestsPer minute, per account
Anonymous clients30 requestsPer minute, per IP
Webhooks (Stripe)UnlimitedExcluded from rate limiting
Health checksUnlimitedExcluded from rate limiting

Idle rate limiters are automatically cleaned up every 10 minutes to prevent memory leaks.

Resiliency Stats
  • 11 services with DB retry
  • 23 FMP calls with backoff
  • 5 min FMP response cache
  • 35 day database backup retention
  • 10 sec service auto-restart
  • 3 retry attempts per operation
  • 0 single points of failure
Related Pages