Calibrate in Flight

Forty-Nine Decisions Before the First Line of Code

The feature started as a living design document that accumulated 49 numbered decisions — engine placement, storage economics, regime methodology, legal posture, failure modes, monetization — each recorded with its reasoning at the moment it was made. The doc wasn't ceremony; it was the build plan. When implementation started, every architectural argument had already been had, so the code went down in ordered slices against a spec that couldn't drift: the document decided, the code obeyed.

One decision mattered more than the rest: ship the instrument on draft settings, then calibrate against history on the production system — publicly. The alternative (tune in private until the backtest flatters, then launch) produces prettier stories and worse instruments. Ours ships with its test results printed on the label, failures included.

The Backtest: 354 Months, No Mercy

The backtest harness is not a notebook — it's a permanent mode of the production engine, running the exact classifier and scorer against every month from 1997 forward. The draft bands failed honestly: they warned before zero of the window's three recessions, because coincident factors (a healthy labor market, calm credit) paid bonuses that drowned out the yield curve's leading signal. So four candidate configurations swept the same 354 months on the same harness, and the winner — sharpened curve penalties, capped coincident bonuses — froze as bands v1.0 the same day. The findings we publish, warts first:

• 2001: warned, five months early. The grade crossed into warning territory in November 2000.
• 2008: deteriorated ~20 points but bottomed at 52 — above the warning line. Printed, not tuned away.
• 2020: unwarnable. No macro series sees a pandemic. Pretending otherwise would be a tell.
• One false-alarm era (2022–24) — the deep inversion that fooled essentially every curve model on earth fooled ours too.
• The regime axes needed no tuning: months split 84 / 96 / 87 / 87 across the four regimes, naturally balanced.

Why publish a scorecard this modest? Because the modern era holds only three recessions — a sample that supports behavioral claims ("the gauge behaves the way its famous ingredients behave"), never prophetic ones. An instrument that overclaims its backtest is an instrument you should not trust with the next decade.

War Story: The Forecast That Failed Silently — and Gracefully

The dashboard's ML projections (singular-spectrum forecasts at 3/6/12-month horizons) shipped with a deliberate degradation rule: a series whose model can't train ships with no forecast, never a fabricated one. That rule worked so well it briefly hid a real bug. In production, every SSA training call was throwing DllNotFoundException: MklImports — ML.NET's math kernel package ships libMklImports.so for Linux but not its Intel OpenMP dependency (libiomp5.so). The engine logged warnings, shipped snapshots without forecasts, and nothing looked broken — the charts simply lacked their dashed projection lines.

A user-level question — "how do I get to the projections?" — triggered the audit that found the gap. The fix was surgical: pull Intel's official OpenMP runtime from its intel-openmp wheel, install the single shared library system-wide (so pipeline redeploys can't wipe it), verify the linkage with ldd, restart. Next run: all 21 forecast-enabled series carrying projections with 80% confidence bounds.

Two lessons kept: graceful degradation needs a loud counterpart — a "feature silently absent everywhere" check, not just per-item fallbacks (the dashboard now discloses staleness and absence in the UI, and the runbook documents the dependency); and native-dependency gaps are platform-specific — the same code no-ops on macOS ARM and worked nowhere until the VM got its one missing .so file.

The Grades × Regimes Study — Backtesting Our Own Grades

With the regime timeline frozen, a second study became possible: how did stocks of each GMI grade actually perform in each regime? The methodology is where the engineering honesty lives, because financial backtests die by look-ahead bias and silent survivorship:

• No look-ahead, mechanically enforced: a fiscal year's reconstructed grade takes effect the month after its actual SEC filing date. A March-filed 10-K never grades January. This rule has its own golden tests.
• Fundamentals-only reconstruction: historical grades recompute the published formula from each year's audited statements. The analyst-price-target factor has no point-in-time archive, so it scores the formula's own neutral band for every historical year — disclosed, not fudged.
• Survivorship, disclosed in the product: the universe is today's S&P 500 membership, which means the failures aren't in the sample and absolute return levels run optimistic. The study's claim is the relative comparison between grade buckets inside the same survivor universe — and the UI prints exactly that caveat.
• Ungraded months are dropped, never defaulted — a stock with no filed history in a month simply doesn't count that month, in any bucket.

The output ships on the dashboard as "Quality Across Regimes" — grade buckets × regimes, with sample sizes on every cell. And the first production run delivered a lesson in why you run the study before writing the conclusion: the raw grid shows the tiny low-grade cohort (87 companies, crisis-concentrated months) posting the biggest averages — the survivor-rebound artifact in its purest form, printed rather than hidden. The claim the data actually supports is narrower and stronger: A-graded companies earned positive market-relative returns in every regime, and in the Slowdown regime — the hardest tape — they were the only meaningful cohort that did (A +0.17%/mo vs SPY while B and C ran −0.33 and −0.45). Marketing copy drafted before the run claimed "quality compounds through all four regimes"; the copy was corrected to match the data the same night, which is the whole discipline in one sentence.

Why "Calibrate in Flight" Is a Feature, Not a Compromise

Every calibrated instrument faces the same choice: tune in private and present the result as if it were born perfect, or ship the hypothesis, test it where users can watch, and freeze what survives. The second path costs some polish on day one and buys the only thing that matters long-term: a version-stamped instrument whose every number can be audited back to a published decision. Bands v1.0 is frozen; the harness that judged it is permanent; any v1.1 must beat it in public, on the same 354 months, and ship as a diff. That loop — instrument, gauntlet, freeze, repeat — is the engineering pattern this page exists to document.