Grade My Investments Technology Observatory
GMI · TECHNOLOGY OBSERVATORY // ALL SYSTEMS NOMINAL
ENGINEERED BY LEOPARD DATA

Background Services

Automated services powering Grade My Investments' core operations

Service Architecture

Grade My Investments' background services run continuously on dedicated virtual machines, handling critical operations like report generation, communications, billing, and system scheduling. These services are deployed via CI/CD pipelines and managed through systemd for high availability and automatic recovery.

Process Topology
rendering diagram…
flowchart LR
  subgraph SRC[Triggers]
    web[Client - Admin - API]:::app
    sched[Schedule Engine<br/>time-based]:::svc
  end
  subgraph Q[MySQL Work Queues]
    q1[(Jobs)]:::data
    q2[(EmailToSend)]:::data
    q3[(SmsToSend)]:::data
    q4[(Downloads)]:::data
  end
  web --> q1
  web --> q2
  web --> q3
  web --> q4
  sched --> q1
  subgraph SVC[systemd Services - Restart always]
    re[Report Engine]:::svc
    be[Billing Engine]:::svc
    ca[Claude Analysis]:::svc
    rc[Report Comparison]:::svc
    es[Email Sender]:::svc
    ss[SMS Sender]:::svc
    dbld[Download Builder]:::svc
    fp[File Purge]:::svc
    hm[Health Monitor]:::svc
  end
  q1 --> re
  q2 --> es
  q3 --> ss
  q4 --> dbld
  subgraph EXT[External and Storage]
    fmp[FMP Market Data]:::ext
    claude[Anthropic Claude]:::ext
    stripe[Stripe]:::ext
    sg[SendGrid]:::ext
    tw[Twilio]:::ext
    blob[(Blob Storage)]:::data
  end
  re --> fmp
  re --> blob
  ca --> claude
  rc --> claude
  be --> stripe
  es --> sg
  ss --> tw
  dbld --> blob
  fp --> blob
  hm -.watch.-> re
  hm -.watch.-> be
  hm -.watch.-> es
  classDef app fill:#1f2630,stroke:#C0894E,color:#F3EFE8;
  classDef svc fill:#241c14,stroke:#E2B583,color:#F3EFE8;
  classDef data fill:#22201a,stroke:#C0894E,color:#EBD3AE;
  classDef ext fill:#1b1f27,stroke:#5A616B,color:#C4C0B8;
Queue-driven background processing — systemd services on a Linux VM, all Restart=always

15

Console Apps

99.9%

Uptime

24/7

Operation

Auto

Recovery

SMS Sender Service

Handles all outbound SMS communications including trade alerts and notifications via Twilio integration. Processes queued SMS messages from the database in configurable batches.

Key Features:
  • Queue-based message processing from database
  • Retry logic with 3-attempt maximum per message
  • Configurable batch size and polling interval
  • Delivery status tracking (sent/failed)
  • CLI commands: start, stop, status
Architecture
Default Interval 30 seconds
Default Batch Size 10 messages

Email Sender Service

Manages email delivery for reports, transactional emails, and automated notifications through Twilio SendGrid. Supports To, CC, and BCC recipients.

Key Features:
  • HTML and plain text support via SendGrid
  • Multi-recipient parsing (To, CC, BCC)
  • Retry logic with 3-attempt maximum per email
  • Email address validation before sending
  • CLI commands: start, stop, status
Architecture
Default Interval 30 seconds
Default Batch Size 10 emails

Report Engine Service

Core report orchestration service that monitors the job queue and dispatches report generation work. Polls for new and processing jobs, manages job lifecycle, and coordinates with downstream services including charge calculation and notification delivery.

Key Features:
  • Job queue monitoring with status-based filtering
  • Charge calculation and coupon code processing
  • Job notification and messaging integration
  • Detailed startup diagnostics and connection logging
  • CLI commands: start, stop, status
Architecture
Processing Mode Continuous loop
Registered Services 20+ DI registrations

Report Generator Alpha Service

Single-execution report generator invoked per job ID. Fetches financial data from the FMP API, generates comprehensive investment reports, and uploads results to Azure Blob Storage. Each run produces a dedicated Serilog log file for full traceability.

Key Features:
  • Per-job execution with dedicated log files
  • FMP Financial Data API integration (with mock mode)
  • Azure Blob Storage upload for generated reports
  • Serilog structured logging with file and console sinks
  • Exit code reporting (0 = success, 1 = failure)
Architecture
Execution Model Single run per job
Data Source FMP API

Billing Engine Service

Comprehensive billing service with a two-pass billing workflow. Pass 1 generates a read-only billing preview report. Pass 2 processes actual credit card charges via Stripe. Supports both bulk billing runs and single-account charge/retry operations.

Key Features:
  • Two-pass billing: preview report then charge
  • Single-account billing with alternate card support
  • API mode for remote execution via Azure Functions
  • CSV report generation for billing and charge results
  • CLI commands: start, run-once, billing-report, charge, charge-account
Architecture
Default Check Interval 1 hour
Payment Provider Stripe

Schedules Engine Service

Report generation schedule processor that continuously scans for due schedules and creates jobs for the Report Engine to pick up. Loads all active schedules on startup and checks for due items each cycle, with admin error notifications on failure.

Key Features:
  • Active schedule loading from database on startup
  • Due schedule detection and job creation
  • Admin email notifications on processing errors
  • Periodic GC to prevent long-term memory growth
  • CLI commands: start, stop, status, load
Architecture
Default Scan Interval 1 second
Error Recovery Admin alerts + retry

File Purge Service

Automated file cleanup service that processes files marked for deletion in the database. Purges files from Azure Blob Storage in configurable batches, with support for single-run mode and continuous operation.

Key Features:
  • Batch-based file purging from Azure Blob Storage
  • Configurable batch size and polling interval
  • Single-run mode via purge-once command
  • Status reporting with file details
  • CLI commands: start, stop, status, purge-once
Architecture
Default Interval 60 seconds
Default Batch Size 50 files

Download Builder Service

Scalable background service that builds ZIP downloads for large folders asynchronously. Decouples the download process from HTTP requests to handle folders of any size without timeouts or memory constraints.

Key Features:
  • Queue-based download processing
  • Disk-based ZIP building (not in-memory)
  • Progress tracking per download
  • SAS URL generation for direct blob downloads
  • Automatic retry on failure
  • Scoped DbContext per batch (IServiceScopeFactory)
  • CLI commands: start, stop, status, purge-once
Architecture
Default Interval 30 seconds
Default Batch Size 5 downloads

Claude Analysis Console Service

AI-powered file analysis service integrating with Anthropic's Claude API (Sonnet, Opus, Haiku models). Users right-click files or folders in ManageFolders and select "Analyze with Claude" to get intelligent document analysis with cost estimation shown before processing.

Key Features:
  • Queue-based processing with batching (5) and concurrency (3)
  • Supports PDF, CSV, XLSX, XLS, DOCX, DOC, TXT, MD, JSON, XML
  • Image analysis: PNG, JPG, GIF, BMP, WEBP
  • Retry logic (3 attempts) with rate limiting
  • Cost estimation with token-based billing (50% markup)
  • Results saved as DOCX alongside source files
  • Claude Console - interactive chat about files
Architecture
Batch Size 5 jobs
Concurrency 3 threads

Report Comparison Console Service

Processes report comparison jobs — downloads Excel files from both report runs via blob storage, calls the Claude API with a structured comparison prompt, and generates a DOCX comparison analysis covering 11 sections including executive summary, movers & shakers, and actionable recommendations.

Key Features:
  • Queue-based processing with batching (3) and concurrency (2)
  • Downloads both Excel file sets from blob storage
  • Sends to Claude with structured comparison prompt
  • Generates DOCX with 11-section comparison analysis
  • Stale request recovery (60 min timeout)
  • Custom user questions included in output
Architecture
Batch Size 3 jobs
Concurrency 2 threads

SEO Page Builder Service

Generates the free public ticker grade pages (/grade/{TICKER}) that power the SEO acquisition funnel. Polls a refresh job queue (admin button or quarterly self-check), pulls FMP fundamentals for ~2,500 curated tickers, computes grades with the same engine the report dashboard uses, writes Claude summaries only when a grade changes, renders self-contained static HTML, and uploads the site to blob storage for the deploy pipeline to publish.

Key Features:
  • Job-queue driven with orphaned-run recovery after restarts/deploys
  • Per-ticker failure isolation — a bad ticker keeps serving its previous snapshot
  • Cost-aware AI: summaries regenerate only on grade changes
  • Incremental progress — admin sees grades land in chunks of 50 during a run
  • Deterministic output: unchanged data renders byte-identical pages

Health Monitor Service

Continuously monitors system health on the report VM -- disk usage, memory consumption, CPU load, database connectivity, queue depths, external API reachability (FMP, Anthropic, SendGrid, Azure Blob), and background service status via systemctl. Records metrics to the database and sends critical alerts to admins when thresholds are breached.

Key Features:
  • Disk usage monitoring with 80%/90% alert thresholds
  • Memory usage tracking via /proc/meminfo (Linux)
  • CPU load average from /proc/loadavg
  • Database connectivity test with response time measurement
  • Queue depth monitoring: pending jobs, emails, SMS, stale requests
  • External API reachability checks (FMP, Anthropic, SendGrid, Azure Blob)
  • Background service status checks via systemctl
  • Log file size monitoring with large-file warnings
  • Critical error email alerts to admin notification list
Architecture
Default Check Interval 600 seconds (10 min)
Alert Thresholds 80% warn / 90% critical

PyEasy Report Generator

Investment report generation tool that fetches financial data from the FMP API, processes configurable stock lists, and produces individual symbol reports, global statistics, and ML-enhanced reports with forecasting and sentiment analysis. Outputs Excel workbooks to a local directory.

Key Features:
  • FMP API financial data integration
  • Configurable stock lists and generation targets
  • Individual symbol Excel reports
  • Global statistics aggregation across stock lists
  • ML-enhanced reports with forecasting and sentiment analysis
  • Parallel symbol data processing
Architecture
Execution Model Single run
Output Format Excel workbooks

Scale Test Seeder DEV/TEST

Test data seeder that creates synthetic accounts, folders, and files via the Azure Functions API. Supports seeding, cleanup, and billing setup operations for scale testing scenarios with configurable account counts and data volumes.

Key Features:
  • Synthetic account, folder, and file creation via API
  • Cleanup command to remove all seeded test data
  • Billing seed: enable billing, add Stripe test cards, create charges
  • Template file downloads from existing source accounts
  • CLI commands: seed, cleanup, billing-seed
Architecture
Execution Model Single run
Target Azure Functions API

Scale Test Runner DEV/TEST

Load testing tool that simulates concurrent MAUI app user sessions against the Azure Functions API. Launches configurable numbers of virtual users, each performing realistic API call sequences with configurable think times, and produces live statistics and a final summary report.

Key Features:
  • Configurable concurrent virtual user sessions
  • Realistic API call patterns with think time simulation
  • Live statistics printed every 10 seconds
  • Optional Claude analysis endpoint testing
  • Account auto-discovery by prefix convention
  • Duration-based test execution with final summary
Architecture
Default Concurrency 10 users
Default Duration 60 seconds

CI/CD Deployment Process

Automated Deployment Pipeline

All background services are automatically deployed to the report VM through our comprehensive CI/CD pipeline, ensuring consistent, reliable, and zero-downtime deployments.

1. Code Commit
Push to repository triggers pipeline
2. Build & Test
Automated build and testing
3. Package
Create deployment artifacts
4. Deploy
Deploy to report VM
Deployment Features:
  • Blue-green deployment strategy
  • Automatic rollback on failure
  • Health checks and validation
  • Configuration management
  • Service dependency handling
Deployment Statistics
Deploy Time 4.5 min
Success Rate 99.7%
Weekly Deploys 25+
Zero Downtime 100%

SystemD Service Management

Linux SystemD Integration

After deployment, all background services are registered and managed through Linux SystemD, providing robust process management, automatic startup, and system-level monitoring.

SystemD Benefits:
  • Automatic service startup
  • Process monitoring
  • Crash recovery
  • Resource limiting
  • Dependency management
  • Logging integration
  • Security isolation
  • Performance monitoring
Service Configuration Example:
[Unit]
Description=Gmi SMS Sender Service
After=network.target

[Service]
Type=simple
User=gmi
WorkingDirectory=/opt/gmi/sms-sender
ExecStart=/opt/gmi/sms-sender/SmsSender
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
Service Status
SMS Sender Active
Email Sender Active
Report Engine Active
Billing Engine Active
Schedules Engine Active
File Purge Active
Download Builder Active
Claude Analysis Active
Report Comparison Active
Health Monitor Active
System Uptime

127 Days

Last restart: System update

Monitoring & Operations

Performance Monitoring
  • CPU and memory usage
  • Processing throughput
  • Queue depth monitoring
  • Response time tracking
Error Handling
  • Automatic error detection
  • Alert notifications
  • Log aggregation
  • Failure analysis
Maintenance
  • Scheduled maintenance windows
  • Rolling updates
  • Backup and recovery
  • Capacity planning