The Test That Convicted the Original

Parallelizing 38,000 machine-learning fits across cores without giving back the flat-memory win — and how the safety gate built to prove the refactor harmless instead proved the original code was quietly wrong

The Setup

After the report build cache shipped, a whole-market report (5,489 symbols) ran at a flat ~500 MB — but its ML phase ran one symbol at a time, on purpose: sequential was the memory play. The result was an hour-plus of SSA model fitting using roughly half of a 2-vCPU machine, with the other half idle. Every symbol's seven computations — four statement forecasts, a health score, leverage spike detection, and a sentiment model over ~1,250 daily bars — touch only that symbol's data. The phase is embarrassingly parallel. The question was never whether to parallelize; it was how to do it without surrendering what the build cache had just won.

ARCHITECTURE

rendering diagram…

flowchart LR
    Q[(Shared queue<br/>5,489 symbols)]
    subgraph W1[Worker 1]
      C1[Own MLContext<br/>recycled per 250]
    end
    subgraph W2[Worker 2]
      C2[Own MLContext<br/>recycled per 250]
    end
    subgraph WN[Worker N = cores]
      CN[Own MLContext<br/>recycled per 250]
    end
    G{Memory governor<br/>pause starts below<br/>1 GB available}
    M[Merge lock<br/>seven result sets]
    Q --> W1 --> M
    Q --> W2 --> M
    Q --> WN --> M
    G -.throttles.-> W1
    G -.throttles.-> W2
    G -.throttles.-> WN

N workers pull symbols from one shared queue — work-stealing, because per-symbol ML cost varies 3x and pre-divided slices would leave workers idle at the end. Each worker owns its own MLContext; results merge under one lock that costs microseconds against ~1 second of fitting.

Four Decisions That Made It Safe

One queue, not partitions. Symbols differ ~3× in ML cost (data depth, sentiment series length). Work-stealing from a shared queue keeps every worker busy to the last symbol instead of one worker finishing its slice early and watching the others.
One MLContext per worker. Rather than trusting library documentation about thread safety on a pinned package version, each worker constructs its own context — a structural guarantee instead of a documentation promise. Workers also recycle their context every 250 symbols, carrying forward the memory-hygiene lesson the whole-market run taught.
A memory governor that throttles starts, never kills work. Before each symbol, workers check machine-wide available memory against a 1 GB floor and pause if it's low. An in-flight fit always completes — the governor can delay results, never corrupt them. Worst case, throughput degrades to yesterday's sequential speed: the feature's failure mode is its own absence.
Native threading pinned to one. The SSA fits call into Intel MKL, which brings its own thread pool. N managed workers × MKL's threads would oversubscribe 2 cores into a slowdown, so OMP_NUM_THREADS=1 makes parallelism purely the workers' job. (This platform has scars here — ask the macro engine about libiomp5.)

The Parity Gate Caught a Real Bug — in the OLD Code

The merge rule was severe on purpose: run the same universe through the sequential path and the parallel path and demand bit-identical output across all seven result sets. Any difference means a shared-state leak. SSA forecasting is deterministic for a given series and seed, so this should be trivially achievable.

It wasn't — and the reason was a genuine defect that predates the refactor entirely. The sentiment model's training pipeline consumes the shared MLContext's random stream (its train/test split draws from it), which means each symbol's sentiment forecast depended on which symbols happened to be fitted before it. Reorder the symbols, get different forecasts. The sequential engine had been doing this all along — invisibly, because nobody had ever demanded the output be a pure function of the input.

The fix is structural: sentiment now fits with a fresh seeded context per symbol, making every forecast a deterministic function of that symbol's data alone. The test built to prove the new code harmless instead convicted the old code — which is the best possible outcome of a safety gate, and the whole argument for writing them severe.

What It Buys

	Sequential	Parallel, 2 vCPU	Parallel, 8 vCPU
ML phase, whole market (~5,500 symbols)	60–90 min	~35–50 min	~10–15 min (est.)
Output vs sequential	baseline	bit-identical	bit-identical
Rollback	—	one config value (`MaxMlParallelism=1`)

The deeper win is that a VM upgrade now buys ML speed linearly — parallelism turns core count into wall-clock, which changes the hardware calculus for whole-market runs. And because the governor defends the memory floor, scaling up the workers can never quietly trade away the flat-memory guarantee that made this safe to attempt at all.