Health Monitoring

Continuous system health checks, alerting, and real-time dashboard visibility

Health Monitoring Architecture

Grade My Investments' Health Monitor is a dedicated console application running on the report VM as a systemd service. It continuously tracks system health across multiple dimensions: disk usage, memory consumption, CPU load, database connectivity, background service status, queue depths, stale job detection, log file sizes, and external API connectivity (FMP, Anthropic, SendGrid, and Azure Blob Storage). By default the monitor runs every 600 seconds (10 minutes), configurable via the --interval command-line option. Each cycle collects metrics, persists them to the systemhealthchecks database table, and evaluates alert thresholds. When critical conditions are detected, the system writes a CriticalErrorLog record and sends email alerts to all administrators via the AdminNotificationEmailService. The AdminWeb application provides a real-time dashboard that auto-refreshes every 5 seconds, giving operators instant visibility into the health of the entire platform. Additionally, the Azure Functions API exposes dedicated health endpoints for external probing.

10-Minute Cycle

Health checks run every 600 seconds by default (configurable)

10+ Metric Categories

Disk, memory, CPU, database, services, queues, APIs, and more

Threshold Alerts

Warning and critical thresholds trigger database logs and email alerts

Live Dashboard

AdminWeb dashboard auto-refreshes every 5 seconds

HEALTH MONITOR DATA FLOW

rendering diagram…

flowchart TB
    subgraph COLLECT[Collect - every 600s]
        VM[Report VM<br/>Disk - Memory - CPU]
        SVC[Systemd Services<br/>gmi-reportengine etc]
        QD[Queue Depths<br/>Jobs - Emails - SMS]
        APIS[External APIs<br/>FMP - Anthropic - SendGrid - Blob]
    end
    subgraph STORE[Store]
        DB[(systemhealthchecks<br/>MySQL)]
        ERR[(criticalerrorlogs<br/>MySQL)]
    end
    subgraph EVAL[Evaluate Thresholds]
        WRN{Warning<br/>exceeded?}
        CRIT{Critical<br/>exceeded?}
    end
    subgraph ALERT[Alert]
        DASH[AdminWeb Dashboard<br/>auto-refresh 5s]
        EMAIL[Admin Email<br/>SendCriticalErrorNotification]
    end
    VM --> WRN
    SVC --> WRN
    QD --> WRN
    APIS --> WRN
    WRN -- yes --> DB
    WRN -- critical --> CRIT
    CRIT -- yes --> ERR --> EMAIL
    DB --> DASH
    classDef app fill:#1f2630,stroke:#C0894E,color:#F3EFE8;
    classDef svc fill:#241c14,stroke:#E2B583,color:#F3EFE8;
    classDef data fill:#22201a,stroke:#C0894E,color:#EBD3AE;
    classDef ext fill:#1b1f27,stroke:#5A616B,color:#C4C0B8;
    classDef sec fill:#1d2733,stroke:#5cc8e0,color:#eafaff;
    class VM,SVC svc
    class QD,APIS ext
    class DB,ERR data
    class WRN,CRIT app
    class DASH,EMAIL sec

Collect-store-evaluate-alert cycle from VM metrics to admin dashboard.

Metrics Collected

Disk Usage

Monitors the root filesystem on the report VM using DriveInfo. Reports total capacity, used space, free space, and percentage used.

Root drive tracking (/ on Linux, C:\ on Windows)
80% warning threshold
90% critical threshold
Total / used / free breakdown in bytes

Memory Usage

Reads /proc/meminfo on Linux to determine total, available, and used memory. On macOS, uses sysctl and vm_stat as a fallback. Calculates usage percentage and flags high-memory conditions.

Reads /proc/meminfo (Linux) or sysctl + vm_stat (macOS)
Total / available / used in bytes
85% warning threshold
95% critical threshold

CPU Load

Reads /proc/loadavg on Linux to capture the 1-minute load average. On macOS, uses sysctl -n vm.loadavg. Reports a normalised percentage based on CPU core count.

1-minute load average
Normalised to CPU core count as percentage
Cross-platform (Linux + macOS)
Reported in AdminWeb dashboard

Database Connectivity

Calls CanConnectAsync() on the MySQL database context to verify connectivity. Measures response time in milliseconds and flags connection failures or slow responses.

EF Core CanConnectAsync test
Response time measurement (ms)
Critical alert on connection failure
Warning when response exceeds 5,000 ms

Process Info

Collects system-level process information including active process count, the Health Monitor's own memory footprint, and system uptime. On Linux reads /proc/uptime; on other platforms uses process start time.

Active process count
Monitor process memory (WorkingSet64)
System uptime in days

Background Services

Queries systemctl is-active on Linux to check each registered service. Reports each service as active or inactive and warns if any are not running.

gmi-reportengine
gmi-emailsender
gmi-smssender
gmi-filepurge
gmi-schedulesengine
gmi-claudeanalysis
gmi-reportcomparison
gmi-healthmonitor

Queue Depths

Queries the database to count pending work items across all background processing queues. Also counts stale jobs that have been stuck in "Processing" status for more than 60 minutes.

Pending report generation jobs
Pending Claude analysis jobs
Pending report comparison jobs
Unsent emails and SMS
Warning at 20+ pending report jobs
Warning at 50+ unsent emails
Critical alert on any stale jobs (> 60 min)

External API Connectivity

Probes external APIs to verify network reachability and measures response times. Uses a shared HttpClient with a 10-second timeout for all connectivity checks.

FMP API -- /stable/quote?symbol=AAPL (lightweight single-quote endpoint)
Anthropic API -- /v1/models (401 expected without auth = reachable)
SendGrid API -- /v3/ base endpoint
Azure Blob Storage -- blob.core.windows.net
Response time measured for each API
Critical alert when FMP or Anthropic unreachable

Logs and Table Stats

Scans the /opt/gmi directory tree for .log files and reports total size and any individual files exceeding 50 MB. Also queries row counts for key database tables to track data growth.

Total log file size in bytes
Warning on log files > 50 MB
Jobs table row count
Files table row count
EmailsToSend table row count

Alert System

When a health check detects a metric that exceeds a configured threshold, the Health Monitor adds a warning string to the check record. At the end of each cycle, if any warnings begin with "CRITICAL", the overall health status is set to "Critical" and the alert pipeline fires. Alerts flow through two channels simultaneously: a CriticalErrorLog database record for persistence and audit, and an email notification sent to every address on the admin notification list via AdminNotificationEmailService.SendCriticalErrorNotificationAsync().

Threshold Configuration

Metric	Warning	Critical
Disk Usage	80%	90%
Memory Usage	85%	95%
Database Response	> 5,000 ms	Connection failure
Services	Any service not active
Pending Report Jobs	> 20 in queue
Unsent Emails	> 50 in queue
Stale Jobs	Any job stuck > 60 min
FMP API	Non-success status	Unreachable
Anthropic API	Unreachable
SendGrid API	Unreachable
Azure Blob Storage	Timeout or unreachable

Alert Channels

CriticalErrorLog Table

When the overall status is "Critical", a CriticalErrorLog record is created with ServiceName = "HealthMonitor", ErrorType = "System Health Critical", the warning messages as ErrorMessage, and a context string summarising disk, memory, CPU, and database metrics. These records are surfaced in the AdminWeb critical-error dashboard.

Admin Email Alerts

An email is sent to every address on the admin notification email list via SendCriticalErrorNotificationAsync(). The email includes the service name, error type, warning messages, and a context line with current disk, memory, CPU, and database connection status.

Azure Functions Health Endpoints

In addition to the background Health Monitor service, the Azure Functions API exposes three health-related HTTP endpoints. These can be called by external monitoring tools, load balancers, or the AdminWeb dashboard.

HealthCheck

GET /api/health (anonymous)

Performs live checks of MySQL database connectivity and Azure Blob Storage health, and verifies Stripe configuration. Returns an overall Healthy/Unhealthy status with per-check latency measurements.

Database CanConnectAsync + latency
Blob Storage IsHealthyAsync + latency
Stripe API key + webhook config check
Returns 200 OK or 503 Service Unavailable

ReadinessProbe

GET /api/ready (anonymous)

A lightweight readiness check that returns a simple "Ready" status with a UTC timestamp. Suitable for load balancer readiness probes that only need to confirm the Azure Functions host is responding.

Minimal overhead -- no external calls
Always returns 200 OK

GetSystemHealthData

GET /api/management/system-health?historyCount=20 (admin only)

Returns the latest SystemHealthCheck record and a configurable number of recent history records. Requires admin (Support role) authentication. This is the endpoint consumed by the AdminWeb dashboard.

Latest health check with all metrics
Configurable history count (default 20)
Admin role required (Support level)

AdminWeb Health Dashboard

The AdminWeb application includes a dedicated System Health page that provides real-time visibility into the current state of the report VM. The dashboard calls the GetSystemHealthData API endpoint and auto-refreshes every 5 seconds via a System.Threading.Timer, giving operators a live view without manual page reloads.

Dashboard Sections

Overall status card (Healthy / Warning / Critical) with machine name and timestamp
Disk usage card with progress bar and total / used / free breakdown
Memory usage card with progress bar and total / used / available breakdown
CPU load card with percentage and Normal / Elevated / High indicator
Database card with connected/disconnected status and response time
System info: uptime, active processes, monitor memory
Background services with colour-coded active/inactive badges
Queue depths: pending report, Claude analysis, comparison, email, SMS
Stale jobs table with critical highlighting
External API connectivity: FMP, Anthropic, SendGrid, Azure Blob with status and response time
Database and log stats: table row counts, total log size, large file warnings
Health check history table (last 20 records) with colour-coded rows

Sample Dashboard Layout

Disk Usage (/) 45%

Memory 62%

CPU Load 12.5%

Database 12 ms

FMP API 245 ms

Database Tables

Health monitoring data is stored in two MySQL tables. The systemhealthchecks table holds every metric snapshot recorded by the Health Monitor service. The criticalerrorlogs table captures alert events when critical thresholds are breached.

systemhealthchecks

One row per health check cycle. Stores the full snapshot of all metrics collected during that cycle.

Column	Description
`SystemHealthCheckID`	Primary key (auto-increment)
`MachineName`	Hostname of the monitored machine
`ServiceName`	Always "HealthMonitor"
`DiskTotalBytes`	Total disk capacity in bytes
`DiskUsedBytes`	Used disk space in bytes
`DiskFreeBytes`	Free disk space in bytes
`DiskUsedPercent`	Disk usage percentage
`MemoryTotalBytes`	Total system memory in bytes
`MemoryAvailableBytes`	Available memory in bytes
`MemoryUsedBytes`	Used memory in bytes
`MemoryUsedPercent`	Memory usage percentage
`CpuUsagePercent`	CPU load normalised to core count
`DatabaseConnected`	Boolean connectivity result
`DatabaseResponseTimeMs`	Connection response time (ms)
`ActiveProcessCount`	Number of running processes
`ProcessMemoryBytes`	Monitor process memory usage
`SystemUptimeDays`	System uptime in days
`ServiceStatuses`	Semicolon-delimited service=status pairs
`PendingReportJobs`	Queued/processing report jobs
`PendingClaudeAnalysisJobs`	Pending Claude analysis jobs
`PendingReportComparisonJobs`	Pending report comparison jobs
`PendingEmails`	Unsent emails in queue
`PendingSms`	Unsent SMS in queue
`StaleReportJobs`	Report jobs stuck > 60 min
`StaleClaudeAnalysisRequests`	Claude requests stuck > 60 min
`StaleReportComparisonJobs`	Comparison jobs stuck > 60 min
`FmpApiReachable`	FMP API connectivity result
`FmpApiResponseTimeMs`	FMP API response time (ms)
`AnthropicApiReachable`	Anthropic API connectivity result
`AnthropicApiResponseTimeMs`	Anthropic API response time (ms)
`SendGridApiReachable`	SendGrid API connectivity result
`SendGridApiResponseTimeMs`	SendGrid API response time (ms)
`BlobStorageReachable`	Azure Blob Storage connectivity result
`BlobStorageResponseTimeMs`	Azure Blob Storage response time (ms)
`JobsTableRowCount`	Row count of Jobs table
`FilesTableRowCount`	Row count of Files table
`EmailsToSendTableRowCount`	Row count of EmailsToSend table
`TotalLogFileSizeBytes`	Total size of all .log files
`LargeLogFiles`	List of log files exceeding 50 MB
`HealthStatus`	Healthy / Warning / Critical
`Warnings`	Semicolon-delimited warning messages
`CreatedDateTime`	UTC timestamp of the check

criticalerrorlogs

One row per critical error event. Used by the Health Monitor and other system components to record critical errors that require administrator attention.

Column	Description
`CriticalErrorLogID`	Primary key (auto-increment)
`ServiceName`	Originating service (e.g. "HealthMonitor")
`ErrorType`	Error category (e.g. "System Health Critical")
`ErrorMessage`	Human-readable error description (max 4,000 chars)
`StackTrace`	Stack trace if available (max 8,000 chars)
`JobId`	Related job ID (nullable)
`AccountId`	Related account ID (nullable)
`AdditionalContext`	Extra context (max 500 chars)
`IsResolved`	Whether the issue has been acknowledged
`ResolutionNotes`	Notes on how the issue was resolved
`ResolvedDateTime`	UTC timestamp when resolved
`CreatedDateTime`	UTC timestamp of the error

Architecture Flow

The Health Monitor follows a collect-store-evaluate-alert pattern. Data flows from the report VM and external APIs through the monitor service into MySQL, and from there into the AdminWeb dashboard (via the Azure Functions API) and the email alert pipeline.

Report VM

Disk, Memory, CPU, Services, Processes, Logs

Health Monitor

Collects metrics + probes external APIs every 600s

MySQL Database

systemhealthchecks + criticalerrorlogs

AdminWeb Dashboard

Auto-refresh 5s

Email Alerts

Admin notification list

External APIs probed each cycle:

FMP
/stable/quote?symbol=AAPL

Anthropic
/v1/models

SendGrid
/v3/

Azure Blob
blob.core.windows.net