Macro Data Scale
The honest numbers behind the Macro Dashboard's data estate — what gets processed, what gets stored, and why the whole thing fits in less storage than a single phone photo. This page is about a different kind of engineering flex: the big-data problem that wasn't, because design made it unnecessary.
The Footprint, Measured (Production, Aug 2026)
| Store | Contents | Blobs | Size |
|---|---|---|---|
gradestudy/ | Per-company raw cache for the grades × regimes study — 484 companies' monthly returns + reconstructed grade spans | 488 | 3.5 MB |
series/ | Full history for all 38 series — every observation the government has ever published for them | 38 | 3.0 MB |
history/ + root | Daily snapshot archives + the live dashboard snapshot | 4 | 0.8 MB |
backtest/, playbook/, lab/, commodities | Calibration artifacts, sector-regime tables, the Macro Lab pack, commodities blob | 11 | 0.5 MB |
| Total blob estate | — | 541 | 7.8 MB |
And the MySQL side of the macro feature — the deliberately tiny "queryable sliver" — holds 291 rows across three tables: the engine run log, the immutable regime-change event history, and share links. Twenty-nine years of economic history, a four-regime classification timeline, a 484-company study, and a production instrument's full operational record: under eight megabytes and three hundred database rows.
Small at Rest, Busy in Motion
The estate is small because the processing does the work. Every engine run — at least daily at 6:30 ET, plus minutes after each government release — refetches the complete history of all 38 series: 151,224 observations, re-validates them, recomputes every transform (YoY, real variants, rolling z-scores), reclassifies all 355 months of regime history, rewalks the grade bands, refreshes 21 SSA forecasts, rebuilds the sector playbook from ~27 years of ETF returns, and atomically swaps one snapshot. The grades × regimes study adds a heavier on-demand sweep: ~1,500 API pulls across 484 companies, joined into 104,983 point-in-time stock-month observations — then cached, so re-running the entire study costs almost nothing.
Full refetch on every run sounds wasteful until you price it: 151K observations is single-digit megabytes. That affordability buys something precious — the entire class of incremental-sync bugs cannot exist here. No watermarks, no missed-revision windows, no "did we backfill after the outage?" Every run rebuilds the truth from the source, and government revisions flow through automatically because there is no cached copy to go stale.
The Judgment Call: Restraint as Architecture
The reflexive architecture for "economic data platform" is a data lake, an incremental ETL pipeline, a warehouse, and an orchestration layer — resume-driven infrastructure for a problem this product does not have. The macro engine's actual requirements were: correct, explainable, always fresh, cheap to operate solo. Three design rules delivered all four:
- Blob-wholesale, MySQL-sliver: bulk truth lives in blob storage at ~2¢/GB-month; the database holds only what needs querying (run log, event history). MySQL is the expensive resource — 291 rows is the entire bill.
- Derive, don't store: transforms, z-scores, regime classifications, and playbook tables are recomputed from raw observations every run. Stored derivations are cache-invalidation debt; recomputed ones are always consistent with their inputs.
- Full refetch as idempotency: when the whole dataset costs megabytes, "always rebuild everything" is not waste — it is the elimination of state-synchronization as a failure category.
Growth math, for honesty: the daily archive adds roughly a quarter-megabyte per day — call it ~90 MB per year. At that rate the macro estate reaches one gigabyte sometime around 2037. The storage line on the Azure bill rounds to pennies; the engineering time not spent operating a data platform rounds to the entire feature list shipped instead.
Related reading: Macro Dashboard Architecture (the storage split and the release-aware scheduler) · Applied Data Science (what the processing computes) · Calibrate in Flight (the 354-month validation harness) · Lean Cloud Economics (the same philosophy, platform-wide).