Grade My Investments Technology Observatory
GMI · TECHNOLOGY OBSERVATORY // ALL SYSTEMS NOMINAL
ENGINEERED BY LEOPARD DATA

Job Architecture

Comprehensive workflow and processing architecture for report generation jobs in the Grade My Investments platform.

Job Processing Overview

The Grade My Investments Job Architecture defines how report generation requests are processed from initial submission through completion and delivery. The system handles both immediate and scheduled report generation with robust error handling and status tracking.

Jobs are processed through a multi-stage pipeline that includes validation, queuing, processing, file generation, storage, and notification delivery.

JOB PIPELINE
rendering diagram…
flowchart LR
    subgraph SUBMIT[Submission]
        A([Client Web App]):::app
        B([Admin Scheduler]):::app
    end
    subgraph QUEUE[Queue - MySQL]
        C[(jobs table)]:::data
        D[(emailqueue)]:::data
        E[(smsqueue)]:::data
    end
    subgraph PROCESS[Processing]
        F[Validate Request]:::svc
        G[Report Engine]:::svc
        H[Fetch FMP Data]:::ext
        I[Generate Files<br/>xlsx - json - docx]:::svc
    end
    subgraph DELIVER[Delivery]
        J[(Azure Blob Storage)]:::data
        K[SAS URL]:::svc
        L([Email - SMS Notify]):::app
    end
    A --> F
    B --> F
    F --> C
    C --> G
    G --> H
    H --> G
    G --> I
    I --> J
    J --> K
    C --> D
    C --> E
    D --> L
    E --> L
classDef app fill:#1f2630,stroke:#C0894E,color:#F3EFE8;
classDef svc fill:#241c14,stroke:#E2B583,color:#F3EFE8;
classDef data fill:#22201a,stroke:#C0894E,color:#EBD3AE;
classDef ext fill:#1b1f27,stroke:#5A616B,color:#C4C0B8;
classDef sec fill:#1d2733,stroke:#5cc8e0,color:#eafaff;
Report generation job lifecycle from submission through delivery.

Job Lifecycle

1
Job Submission

User submits report request via Client Web App or scheduled through Admin interface

2
Validation & Queuing

Azure Functions validate request parameters and add job to processing queue

3
Job Processing

Report Engine picks up job, fetches financial data, and generates analysis

4
File Generation

Multi-format output generation (Excel, JSON, DOCX) with charts and tables

5
Storage & Delivery

Files stored in Azure Blob Storage and notifications sent via email/SMS

Job Types & Processing

On-Demand Jobs
  • • Immediate processing
  • • Real-time status updates
  • • Priority queue placement
  • • Interactive progress tracking
Scheduled Jobs
  • • Recurring report generation
  • • Batch processing optimization
  • • Off-peak execution timing
  • • Automated delivery

Job Data Model

Field Type Purpose
JobID Primary Key Unique job identifier
AccountID Foreign Key Links to user account
JobType Enum On-demand or scheduled
Status Enum Queued, Processing, Completed, Failed
SymbolListID Foreign Key Symbols to analyze
Parameters JSON Analysis parameters and settings
Priority Integer Queue priority (1-10)
EstimatedDuration TimeSpan Predicted processing time
ActualDuration TimeSpan Actual processing time

Architectural Decisions

Claude AI Recommended Database Queue Architecture

During Grade My Investments' development, we consulted Claude AI about implementing a Kafka-like message queuing mechanism for report generation and communication messages. Claude AI analyzed our requirements and recommended using the database as a queue instead of implementing Apache Kafka.

Decision Analysis
Apache Kafka Approach

Considered but not implemented:

  • High-throughput distributed messaging
  • Complex cluster management
  • Additional infrastructure overhead
  • Over-engineered for our volume
Why Not Kafka:
  • Grade My Investments processes 10-100 jobs/day, not thousands/second
  • Additional operational complexity
  • Infrastructure costs and maintenance overhead
  • Unnecessary for current scale requirements
Database Queue Approach

Claude AI's recommendation - implemented:

  • MySQL database tables as message queue
  • FIFO processing with status tracking
  • Built-in ACID transactions
  • Perfect fit for our volume and complexity
Why Database Queue:
  • Architecturally appropriate for 10-100 jobs/day
  • Leverages existing MySQL infrastructure
  • Simple, reliable, and maintainable
  • No additional moving parts or complexity
Database Queue Implementation

Following Claude AI's recommendation, Grade My Investments implements queuing through MySQL database tables:

FIFO Processing
  • Job Queue: jobs table with status tracking
  • Email Queue: emailqueue for notification delivery
  • SMS Queue: smsqueue for text notifications
  • Processing Order: First-In-First-Out with priority support
Reliability Features
  • ACID Transactions: Guaranteed data consistency
  • Status Tracking: Queued → Processing → Completed
  • Error Handling: Failed jobs marked for retry
  • Atomic Operations: No partial state corruption
Performance Benefits
  • Low Latency: Direct database queries
  • Simple Scaling: MySQL performance optimization
  • Monitoring: Standard SQL queries for status
  • Maintenance: Familiar database operations
Decision Impact

Claude AI's recommendation proved architecturally sound for Grade My Investments' specific requirements:

  • Right-Sized Solution: Perfect fit for 10-100 daily jobs
  • Operational Simplicity: No new systems to manage
  • Cost Effective: Leverages existing database infrastructure
  • Reliable Delivery: MySQL's proven durability and consistency
  • Future Scalability: Can migrate to Kafka if volume increases 100x
  • Team Familiarity: Standard SQL operations and monitoring

Report Traffic Pattern

The Report Traffic Pattern feature provides real-time visibility into the shared report generation queue. Inspired by aviation traffic patterns, users can see where their report sits in the queue, how many reports are ahead, and estimated time to completion.

Queue Visibility
  • Live view of all queued reports
  • Queue position for each report
  • Symbol count per report
  • Anonymized data for other users
ETA Calculation
  • ~13 API calls per symbol
  • 3,000 FMP API calls/minute limit
  • ~230 symbols processed/minute
  • Cumulative wait time estimates
Auto-Refresh
  • 20-second polling interval
  • Available in all 3 apps
  • Post-generate dialog shows ETA
  • Shared queue across all users
API Endpoint

GET /reports/traffic-pattern -- Returns the full queue state including all pending and processing jobs with symbol counts and ETAs. Authenticated users see their own reports highlighted. Other users' data is anonymized.

Response includes: queue items, currently processing report, user's queue positions, total estimated wait time, and rate limit information.

Error Handling & Recovery

Retry Mechanisms
  • Exponential Backoff: Increasing delays between retry attempts
  • Circuit Breaker: Prevents cascade failures from external APIs
  • Dead Letter Queue: Failed jobs moved for manual inspection
  • Partial Recovery: Resume processing from last successful checkpoint
Monitoring & Alerts
  • Status Tracking: Real-time job status updates in database
  • Performance Metrics: Processing time and throughput monitoring
  • Failure Notifications: Immediate alerts for critical failures
  • Resource Monitoring: CPU, memory, and disk usage tracking
Processing Performance
  • Multi-threaded execution
  • Rate limiting compliance
  • Resource optimization
  • Parallel file generation
  • Memory-efficient streaming
Job Status Flow
Queued
Processing
Generating
Uploading
Completed
Processing Metrics
  • Average Job Time: 3-8 minutes
  • Queue Capacity: 100 concurrent jobs
  • Success Rate: 99.2%
  • Retry Success: 95.8%
  • Peak Throughput: 50 jobs/hour
Output Formats
  • Excel (.xlsx)
  • JSON Data
  • CSV Export
  • DOCX Reports (Claude Analysis, Report Comparison)