The pioneering .NET Console application that processes jobs, fetches symbol data, generates comprehensive Excel and JSON reports, and provides advanced ML.NET-powered forecasting capabilities.
ALPHA PROCESSING PIPELINE
rendering diagram…
sequenceDiagram
participant DB as Database
participant ALPHA as ReportGeneratorAlpha
participant FMP as FMP API
participant MLNET as ML.NET Engine
participant EXCEL as EPPlus - Excel
participant BLOB as Azure Blob
DB->>ALPHA: Job status NewReadyForProcessing
ALPHA->>DB: Mark PickedUpAndProcessing
ALPHA->>DB: Load symbol list
ALPHA->>FMP: Fetch quotes - history - financials
FMP-->>ALPHA: Market data for all symbols
ALPHA->>MLNET: Run forecasting models
MLNET-->>ALPHA: Revenue - cash flow - health scores
ALPHA->>EXCEL: Generate xlsx workbook
ALPHA->>EXCEL: Generate HTML report
EXCEL-->>ALPHA: Files ready
ALPHA->>BLOB: Upload xlsx - html - json
BLOB-->>ALPHA: Upload confirmed
ALPHA->>DB: Mark CompletedWithSuccess
ALPHA->>DB: Send completion notification
Four sequential stages from job acquisition through Azure Blob upload and completion notification.
Report Generator Alpha Engine
First of Its Kind: Report Generator Alpha is the original .NET Console application that established Grade My Investments' report generation capabilities. It reads jobs from the database, processes symbol lists, and outputs professional Excel and JSON reports.
Console Application Architecture
> Gmi.ReportGeneratorAlpha.Console.exe
> Checking for pending jobs...
> Found Job ID: 12345
> Loading symbol list: Tech Portfolio (45 symbols)
> Fetching market data...
> Generating Excel report...
> Generating JSON data file...
> Uploading to Azure Blob Storage...
> Updating database records...
> Job 12345 completed successfully!
Core Workflow
Job Processing: Polls database for jobs with status "NewReadyForProcessing"
Symbol List Retrieval: Fetches associated symbol lists from database
Data Collection: Retrieves market data for all symbols
Excel Generation: Creates comprehensive workbook with multiple sheets
JSON Export: Outputs raw data in machine-readable format
Azure Storage: Uploads files to blob storage with proper folder structure
Database-Driven Processing
Job Processing Pipeline
Query database for jobs with JobStatusTypeID = 1 (NewReadyForProcessing)
Update job status to JobStatusTypeID = 2 (PickedUpAndProcessing)
Load associated symbol lists from jobsymbollists table
Retrieve symbol details from symbollistitems table
Fetch real-time quotes for all symbols in the list
Retrieve historical price data
Collect fundamental metrics (P/E, market cap, dividends)
Calculate technical indicators
Process data in parallel for performance
Create Excel workbook using EPPlus library
Generate multiple worksheets with formatted data
Add charts and conditional formatting
Export raw data to JSON format
Create file metadata records
Upload Excel file to Azure Blob Storage
Upload JSON file to Azure Blob Storage
Insert file records in files table
Update job status to JobStatusTypeID = 4 (CompletedWithSuccess)
Record completion timestamp and send notifications
Performance Characteristics
Processing Metrics
Metric
Performance
Notes
Symbols per Second
10-15
With full analysis
Concurrent Threads
8-16
Based on CPU cores
Memory Usage
2-4 GB
For 500 symbol report
Report Generation Time
30-60 sec
100 symbols average
Data Points Processed
1M+
Per large report
Linux Deployment & Logging
Console Application Invocation on Linux
Report Generator Alpha is invoked on-demand by the report engine when jobs are queued for processing. Running as a .NET console application on Ubuntu 20.04, it provides detailed logging output for monitoring and debugging with structured logging, timestamp precision, and comprehensive error tracking.
Log Features
Timestamps: Millisecond precision with ISO 8601 format
Log Levels: Debug, Info, Warning, Error, Critical
Context: Job IDs, symbol counts, processing stages
Performance: Execution timing and memory usage
File Locations: Output paths and blob storage URLs
Report Engine Integration
On-Demand Execution: Invoked by the report engine when jobs are queued
Console Application: Runs as standalone .NET console process
Process Lifecycle: Starts, processes job, outputs logs, then exits
Database Polling: Checks for jobs with "NewReadyForProcessing" status
Log File Output: Structured logging to file system for monitoring
Log Analysis Features
Processing Stages
Job acquisition and validation
Symbol list loading and processing
Market data fetching with retries
Report generation and formatting
File upload and database updates
Error Handling
API rate limit exceptions
Network connectivity issues
Data validation failures
File system I/O errors
Database connection timeouts
Performance Metrics
Symbols processed per minute
Memory consumption tracking
Database query execution times
File upload speeds and sizes
End-to-end job completion times
Report Engine Invocation
# Report engine invokes the console application
./Gmi.ReportGeneratorAlpha.Console
# Manual execution for testing
dotnet Gmi.ReportGeneratorAlpha.Console.dll
# Check for running processes
ps aux | grep ReportGeneratorAlpha
ML.NET Forecasting Integration
Machine Learning Capabilities: Report Generator Alpha now incorporates Microsoft's ML.NET framework to provide advanced forecasting and prediction capabilities directly within the report generation process.
Forecasting Features
Financial Forecasting
Revenue Forecasting: Predicts future quarterly revenue using historical patterns
Cash Flow Prediction: Projects free cash flow trends up to 4 quarters ahead
Debt Analysis: Forecasts debt levels and debt-to-equity ratios
