Grade My Investments Technology Observatory
GMI · TECHNOLOGY OBSERVATORY // ALL SYSTEMS NOMINAL
ENGINEERED BY LEOPARD DATA

Scale Testing

Validating platform capacity with concurrent virtual users, automated data seeding, and real-time performance measurement.

SCALE TEST WORKFLOW
rendering diagram…
flowchart LR
    subgraph PREP[Preparation]
        MOCK[Mock FMP Client<br/>synthetic financial data]:::svc
        SEED[Data Seeder<br/>accounts - folders - files]:::svc
        ENV[Env Setup<br/>P1V2 - rate limit raised]:::ext
    end

    subgraph RUN[Load Run Phases]
        P1[Baseline<br/>10 users - 2 min]:::app
        P2[Moderate<br/>20 users - 5 min]:::app
        P3[Heavy<br/>50 users - 10 min]:::app
        P4[Claude AI<br/>10 users - 5 min]:::sec
    end

    subgraph METRICS[Metrics Collected]
        RT[Real-Time Stats<br/>every 10 sec]:::data
        SUM[Final Summary<br/>P50 - P95 - P99]:::data
        CSV[CSV Export]:::data
    end

    MOCK --> SEED
    SEED --> ENV
    ENV --> P1
    P1 --> P2
    P2 --> P3
    P3 --> P4
    P4 --> RT
    RT --> SUM
    SUM --> CSV

    classDef app fill:#1f2630,stroke:#C0894E,color:#F3EFE8;
    classDef svc fill:#241c14,stroke:#E2B583,color:#F3EFE8;
    classDef data fill:#22201a,stroke:#C0894E,color:#EBD3AE;
    classDef ext fill:#1b1f27,stroke:#5A616B,color:#C4C0B8;
    classDef sec fill:#1d2733,stroke:#5cc8e0,color:#eafaff;
Three-tool scale test lifecycle from data seeding through load running to metrics analysis.

Why Scale Test?

Before launching a SaaS platform, you need to know it can handle real-world load. Scale testing answers critical questions: How many concurrent users can the infrastructure support? Where are the bottlenecks? Will the database hold up under pressure? Will API rate limits protect the system or choke legitimate traffic?

Grade My Investments uses a purpose-built scale testing framework consisting of three tools that work together to seed synthetic data, simulate concurrent user sessions, and measure performance metrics with percentile-level precision.

Key principle: If the platform performs well under load on a smaller dev environment, it will handle production workloads on larger infrastructure with confidence.

Scale Testing Toolchain

The scale testing framework is composed of three dedicated .NET console applications, each handling a specific phase of the testing lifecycle:

Mock FMP Client

Generates realistic synthetic financial data so report generation runs without consuming real API credits. Deterministic per symbol for repeatable tests.

MOCK_FMP=true
Data Seeder

Creates synthetic test accounts, folders, and files via the API. Supports configurable account counts, folder depth, and file density.

ScaleTest.Console seed
Load Runner

Simulates N concurrent virtual users performing realistic workflows against the API. Captures latency percentiles and error rates in real time.

ScaleTestRunner.Console run

Virtual User Simulation

Each virtual user runs as an independent concurrent session, executing a realistic workflow loop that mimics how real users interact with the platform. The simulation includes configurable "think time" delays between actions to model human browsing behavior.

Workflow Per Virtual User
1
Select Random Account

Pick from the pool of seeded test accounts to distribute load across data partitions.

2
Browse Folder Tree

GET /accounts/{"{id}"}/folder-tree — Fetches the full hierarchical folder structure with nested files.

3
List Files in Folder

GET /folders/{"{id}"}/files — Retrieves file metadata for a randomly selected folder.

4
Download File (20% probability)

GET /files/{"{id}"}/download — Downloads binary file content from Azure Blob Storage.

5
Think Time

Configurable random delay (default 500ms–2000ms) simulating human interaction pauses.

6
Request ZIP Download (20% probability)

POST /accounts/{"{id}"}/downloads — Requests a ZIP package of an entire folder.

7
Repeat

Loop continuously until the test duration expires. Each iteration is independent.

Test Phases

Scale tests are run in progressive phases, each increasing concurrency to identify the point where performance degrades:

Phase Concurrent Users Duration Purpose
Baseline 10 2 minutes Establish performance baselines and verify test harness
Moderate 20 5 minutes Normal operating load for early platform adoption
Heavy 50 10 minutes Stress test to find bottlenecks and breaking points
Claude AI 10 5 minutes Test Claude AI analysis endpoints under concurrent load

Performance Metrics Collected

The load runner captures detailed performance metrics for every API call, providing both real-time visibility and comprehensive post-test analysis:

Real-Time Stats (Every 10 Seconds)
  • Total request count
  • Requests per second (RPS)
  • Average latency
  • Error count and error rate percentage
Final Summary Report
  • Per-endpoint breakdown (count, avg, P50, P95, P99, min, max)
  • HTTP status code distribution
  • Error categorization (rate limits, server errors, timeouts)
  • CSV export for further analysis
Latency Percentiles

Averages can be misleading — a few slow requests can hide behind a fast average. That's why Grade My Investments tracks P50 (median), P95 (95th percentile), and P99 (99th percentile) latencies per endpoint. P99 tells you what your slowest 1% of users experience.

Metric What It Measures Target
P50 Median response time — half of requests are faster than this < 300ms
P95 95th percentile — only 5% of requests are slower < 1,000ms
P99 99th percentile — captures worst-case user experience < 2,000ms
Error Rate Percentage of non-200/201 responses < 1%

Automated Data Seeding

Realistic scale testing requires realistic data. The Data Seeder creates synthetic accounts, folder hierarchies, and uploaded files via the same API endpoints that real users would use. This exercises the full request pipeline including authentication, validation, database writes, and blob storage uploads.

Seeder Capabilities
Parameter Default Description
--accounts 10 Number of synthetic test accounts to create
--folders-per-account 5 Folder hierarchy depth per account
--files-per-folder 3 Files uploaded per folder (uses real template files)
--prefix scaletest Naming prefix for easy identification and cleanup
Idempotent design: The seeder can be run multiple times safely. If accounts already exist, they are reused rather than duplicated. A matching cleanup command removes all seeded data by prefix.
Billing Test Data

A dedicated billing-seed command extends the seeded accounts with billing data for end-to-end payment testing:

  • Enables and validates each test account
  • Adds Stripe test credit cards (Visa ****4242)
  • Creates randomized report generation charges ($3–$75 per charge)
  • Creates storage charges for billing cycle testing

Mock Financial Data Provider

The report engine's largest external dependency is the Financial Modeling Prep (FMP) API, which provides market data and financial statements. For scale testing, a mock implementation generates synthetic but realistic financial data without making real API calls.

Mock FMP Features
Deterministic

Same symbol always returns same data. Uses symbol-based RNG seeding for repeatable tests.

Realistic Ranges

Prices $15–$500, realistic margins (30–60% gross), proper balance sheet ratios.

Latency Simulation

Adds 10–50ms random delays to simulate real API response times.

Full Coverage

All 13 FMP endpoints mocked: prices, financials, balance sheets, cash flows, metrics, analyst data.

Infrastructure Preparation

Before running scale tests, the target environment is prepared to isolate performance measurements from configuration artifacts:

Step Action Reason
1 Upgrade App Service to P1V2 Remove compute ceiling so we measure database/storage limits, not CPU
2 Raise API rate limit Prevent rate limiter from masking real performance bottlenecks
3 Disable admin notification emails Prevent email flood from thousands of test operations
4 Seed test data Create realistic data volume for meaningful results
5 Run progressive test phases Gradually increase load to find the breaking point
6 Restore original settings Return environment to normal operating configuration

Concurrent Users to Platform Users

Scale test "concurrent users" are far more aggressive than real users. The test runner uses 500ms–2,000ms think times, while real users typically pause 30–60 seconds between actions. This means each concurrent test user represents approximately 15–30 real platform users.

Concurrent Test Users Real Platform Users Assessment
10 100–200 Comfortable
20–25 200–500 Maximum clean capacity
30 300–600 Starting to see pressure
50 500–1,000 Requires connection pool tuning
Quick Facts
  • 3 dedicated testing tools
  • 50+ concurrent virtual users tested
  • 37,000+ API requests per test run
  • Percentile-level latency tracking
  • Automated data seeding and cleanup
  • Real-time stats every 10 seconds
  • CSV export for analysis
Technology Stack
  • .NET 10 Console applications
  • HttpClient Concurrent sessions
  • ConcurrentBag Thread-safe metrics
  • System.CommandLine CLI framework
  • CancellationToken Graceful shutdown
CLI Examples
# Seed 50 test accounts
dotnet run -- seed \
  --accounts 50 \
  --folders-per-account 5 \
  --files-per-folder 5

# Run 50 concurrent users for 10 min
dotnet run -- run \
  --concurrency 50 \
  --duration 600 \
  --think-time-min 500 \
  --think-time-max 2000

# Set up billing test data
dotnet run -- billing-seed \
  --charges-per-account 3

# Clean up test data
dotnet run -- cleanup
Bottlenecks Found

Scale testing identified these bottlenecks before they could impact production:

  • MySQL connection pool — exhaustion at 50 concurrent users
  • API rate limiting — default limits too restrictive for bulk operations
  • Storage charge rounding — sub-penny charges violating DB constraint
  • Invoice tracking — EF Core entity tracking conflict on paid status

All issues were identified and fixed during scale testing, before reaching production.

Projects
  • Gmi.ScaleTest.Console
  • Gmi.ScaleTestRunner.Console
  • Gmi.RestClient.FinancialDataProvider
  • Gmi.BillingEngine.Console