One Contract, Two Estates

Two Systems, One Owner, Opposite Postures

Grade My Investments is a revenue product with a batched release discipline: it grades stocks, reads the macro economy, and publishes what it finds. Alongside it sits Plutus, a private paper-trading laboratory in its own repository, built to test whether those grades and that macro regime can actually drive a trading decision over a thirty-year replay. Plutus trades nothing real yet and publishes nothing; its whole output is a tear sheet that says whether a strategy would have beaten the index, and how honestly that claim was earned.

Plutus needs data it does not have and cannot afford to buy. GMI has been accumulating exactly that data as a byproduct of running its own business: balance sheets, income statements and cash flows for roughly 5,500 US symbols going back up to 35 years; 40-odd government series with decades of history from a free source; a deterministic regime classifier with a published formula; and a paid market-data tier that reaches endpoints a hobby key is refused.

The obvious answer is a database grant. The rest of this page is about why that answer was rejected, what replaced it, and the one defect the replacement caught on its second day.

Why the Database Was the Wrong Boundary

Direct SQL access won on exactly one axis — initial effort — and lost on every other. Two problems were fatal on their own:

• The meaning is not in the schema. The statement cache stores its dataset as a one-byte integer whose names live in a C# enum. The regime is a byte whose names live in a switch expression. The grade is a computation, not a column. A consumer reading those tables directly re-implements GMI's business logic, and when GMI adds a sixth dataset, that consumer does not fail — it silently misreads. Drift with no error is the failure mode this whole design exists to avoid.
• The network cost was disproportionate. Production MySQL sits inside a virtual network with public access disabled. Reaching it from another estate means VNet peering or private endpoints — an infrastructure project, an address-range collision risk, and a permanent operational dependency, bought entirely to avoid writing code on the side that already knows what the bytes mean.

The two adopted options split by size, not preference. A regime state is kilobytes, daily, and every field is a computed value whose meaning belongs to GMI — an endpoint is exactly right. Thirty-five years of statements for 5,500 symbols is on the order of 770,000 periods and a few gigabytes of JSON; over HTTP that is paginated requests against a function timeout, billed per execution. As gzipped line-delimited JSON it is a handful of files, written once, read once.

The Constraint That Decides Everything: Point in Time

GMI's data stores are read-through caches, and caches mutate. Profiles are refreshed on a thirty-day policy. Sentiment expires. Grades are recomputed on every report run. When a company restates a filing, the cached statement is overwritten and a counter is incremented — the original as-reported figures are gone. All of that is correct for a live dashboard and poison for a backtest.

Anything that queries a mutating store gets a different answer depending on when it runs. Two backtests with the same manifest must reproduce bit-identically, and a live query breaks that silently. Worse, it imports lookahead: a 2019 backtest reading a statement restated in 2023 is trading on information that did not exist in 2019. Macro data has the same disease in a subtler form — GDP is revised for years after its first print, and a backtest run on final-revision figures knows things nobody knew at the time.

So the contract has three jobs, all requirements rather than features:

• Freeze. Export artifacts are immutable and dated; API responses are frozen by the consumer on ingest. An endpoint is as live as a query — it is not a snapshot.
• Filter. Periods that are no longer point-in-time honest — restated statements, live-overwritten grade columns — are excluded or split out on GMI's side, where the flag's meaning is maintained.
• Stamp provenance. Every value says where it came from, when it was fetched, and — for regime data — whether it was lived or reconstructed. That distinction reaches the tear sheet as a caveat, and it is the single most important word in this design.

Three Routes, Deliberately Coarse

The API is three versioned, read-only routes under one prefix, gated by a dedicated function key that opens nothing else. Every other endpoint on the platform still demands a signed-in user or an administrator token; the key cannot reach them. The routes are coarse on purpose — "give me the whole artifact" rather than "give me this field filtered by that predicate" — because the consumer is the fast-moving system and GMI is the batched one. If Plutus had to ship a GMI change every time it wanted a field, its velocity would be gated by GMI's release rhythm. Slicing happens on the consumer's side, where changes are free.

Six rules apply to every payload, and they are most of the value of building an API at all:

• Enums resolve to names, always. "regime": "Slowdown", never "regime": 4. This is what stops the consumer duplicating the enum and drifting from it.
• Every payload carries an as-of time and a formula version. The consumer stamps what it ingests; it cannot stamp what it was not told.
• Empty and error are distinguishable. No matching rows is a 200 with an explicit zero count; failure is a non-2xx. Never a 200 with a silently empty array — a consumer that cannot tell "no data" from "broken" will backtest straight through a hole in its history.
• Routes are versioned. When a shape changes, both versions run until the consumer has moved.
• Read-only by construction. GET only; no route accepts a body.
• Point-in-time fields are mandatory. Anything with a knowability lag carries both dates.

The Defect the Design Caught on Day Two

The regime route shipped labelled lived, because the table it reads is the engine's record of what it observed. Reading the engine's own code the next morning, to answer a different question, turned up a command called backfill: it seeds that same table from the frozen backtest artifact — one row per historical regime flip since 1997 — with the detection time set to the moment the backfill ran. The rows sit beside the lived ones, and the route was serving them with an invented knowability date and a provenance stamp that said they had been watched happen.

That is precisely the confusion the provenance rule exists to prevent, and it is why the rule needed code rather than intent. The route now identifies rows by the engine run that wrote them, withholds anything from a backfill run, and reports the number withheld — so the ribbon the consumer sees is observed history only, and the fact that something was excluded is visible without opening a database.

The same read answered the question it set out to answer. The reconstructed ribbon that the export was going to have to compute already existed: the engine's calibration harness replays the frozen formula month by month from 1997 by truncating every input series to each as-of month, and writes the result as a versioned artifact. The export's job became copying and re-stamping it, not recomputing anything — and the artifact's own disclosure that it was replayed over current-vintage inputs travels with the copy, as a caveat the consumer must repeat.

The Export: A Background Job Whose Failure Is Loud

The bulk side is a background service in the same shape as the platform's other scheduled jobs — a daemon under systemd on the report VM, configured through the same secrets chain, deployed and restarted by the same pipeline. It writes gzipped line-delimited JSON to a dedicated container that holds nothing but what the export writes. The first slice carries the two artifacts that unblock regime-conditioned backtesting:

• The reconstructed regime ribbon. One line per month from 1997, each stamped reconstructed, monthly, and the frozen formula version. The job refuses to run under unfrozen calibration bands, and refuses an artifact whose bands differ from the deployed configuration — a stale artifact after a recalibration is an error, not an export.
• Macro history with real vintages. For every series in the engine's catalog, every observation as it read in every real-time window, not just today's revision. The government data source publishes its own archive of past prints, reachable with one extra parameter on the endpoint GMI already called; each line carries the window in which that value was the published one. For any as-of date, the point-in-time value is the row whose window contains it. That is the only way to keep a 2009 backtest from reading a 2012 revision.

What turns a silent export failure into a loud one is the manifest. Every run writes one: each artifact with its line count, byte count, and a SHA-256 over the gzipped bytes as written; the grain of every artifact, stated so an ingest never blends the monthly reconstructed ribbon with the daily lived one; the provenance and vintage policy of each; source watermarks; every error; and a complete flag that is false if anything failed. The consumer refuses an incomplete run. And it never has to, because of the pointer.

The container's root holds a single latest.json, a copy of the newest complete manifest, and the consumer reads nothing else to find a run. An incomplete run writes its own manifest for diagnosis, leaves the pointer alone, and exits non-zero. "Latest" is therefore the last good run by construction; the consumer's refusal is a belt over braces. The manifest also carries the publishing cadence, so the ingest's staleness alarm takes its threshold from the publisher rather than from a constant that can drift. GMI keeps the current run and the previous complete one and prunes the rest — GMI publishes, the consumer retains.

Exposure Prevented by Construction, Not Configuration

Under the rejected design, GMI's customer records, payment history, login history and support tickets would have been kept away from the consumer by a grant list — one line of DDL between another system and everything. Under the adopted one, that data is not reachable by any credential the consumer holds, because the consumer has no database access and the export container contains only what the export writes.

• The API key opens one route prefix. The three routes are declared at the function-key authorization level so the platform host itself checks the key before the code runs; the authentication middleware then lists exactly those three as key-only, so it does not also demand a user token. The key is not the platform's own service key, so it cannot satisfy any other endpoint's service-to-service check.
• The container credential is read-and-list only, a shared access signature bound to a stored access policy on that one container — revocable by deleting the policy, without rotating the storage account's keys. A short-lived delegated signature was considered and declined: a credential meant to sit in a vault should not expire in a week.
• The consumer holds nothing that can write. "Never writes to GMI" is not a policy that someone could violate; it is the absence of a capability.
• Nothing in the trading or backtest path speaks to GMI at all. A scheduled ingest job is the only thing that touches either surface, and everything downstream reads the consumer's own frozen copy. GMI being down never stops a trading cycle; a stale copy degrades a signal and says so.

And the coupling itself moved somewhere safer. Under direct SQL, a GMI schema change was a consumer outage discovered at runtime. Now the API and the export live in the same repository as the migration, written against the real enums, and break at compile time — in the commit that made the change, on the side that made it.

What It Unlocks for Test Trading — and What It Does Not

Two findings in the inventory changed existing consumer decisions outright. A plan to compute a dollar-strength basket by hand from currency pairs was retired by the discovery that GMI already fetches the Federal Reserve's official broad dollar index, free and longer than any commercial one. And the question of how to get a regime history deep enough to backtest against moved from "open" to "tractable engineering" once the classifier turned out to take an explicit as-of date.

What it does not fix is stated just as plainly. Delisted and dead-company history remains the consumer's problem, because GMI's caches are built from symbols GMI looked up. The reconstructed ribbon was replayed over revised inputs and says so. And a snapshot route can never answer a historical question, however tempting the field names look.

The Rule That Keeps It From Rotting, and What AI Changed

The whole surface is three routes and one export, and keeping it that small is a design requirement rather than an accident. The smell test is written into the interface document that both repositories carry verbatim: if a fourth endpoint is being added within a month, the boundary is in the wrong place, and the answer is to export more and query less. The contract's regression tests live on GMI's side, beside the code they protect, against a real engine snapshot — so a shape change fails GMI's own build before it can reach the consumer.

The API, its correction, the export console, the vintage fetch, the pipeline wiring, the systemd unit, and fifty-two tests were built across two working sessions. Throughput is the least interesting part of that. What changed is which questions got asked before building. Under ordinary pressure the database grant ships, because it is an afternoon's work and the drift it invites is someone else's problem next year. The inventory, the point-in-time analysis, the options table, the provenance rule, and the morning spent reading the engine's own code to find a backfill command that would have quietly poisoned a tear sheet — that is the work that gets cut first, and it is the work that determined whether any of this can be trusted.

When that analysis costs hours instead of weeks, the correct boundary becomes the cheap one. The result is not a faster integration. It is one where the coupling between two systems went from a hundred queries that fail quietly to a single contract that fails at compile time.