The SEC Data Store: Government Filings as the Primary Fundamentals Source

The Question, and the Honest Money Answer

"Would it be cheaper to compile the government data from 10-Qs and other financial documents of public companies, versus pulling from FMP? Can we have a government data store in the GMI database like we have for FMP — maybe even government data as the primary and FMP as the secondary?"

GMI's fundamentals come from Financial Modeling Prep (FMP) under a commercial contract: $0.005 a call against a $50 monthly floor that already includes the first 10,000 calls. Statements are immutable once filed, so GMI's provider cache keeps them permanently. The first estimate, reasoned from the contract alone, said SEC-sourced statements would save $0 in a normal month: whatever the steady state costs, the floor absorbs it.

The design made a measurement (Phase 0) its first step instead of trusting that estimate. The FMP cache already counts every real call by dataset in fmpcachedailystats, so the answer was a query:

The measurement changed the conclusion. Both months were far above the 10,000-call floor. In a quiet month the floor still absorbs everything and the SEC saves nothing, but heavy months, the ones with full regrades, deep statement fetches and universe growth, are where the money actually goes, and statements are more than half of it. The 59% is a ceiling, not a forecast: key metrics stay on FMP (they carry price-based fields only FMP has), and a call is avoided only for an approved symbol whose read would otherwise have missed the cache.

Allan's decision (Q6) was to build it even if the savings came to $0, for four reasons that do not depend on the invoice. Phase 0 still ran, as a measurement, not a gate.

Why Build It Anyway: The Strategic Case

1. Licensing of public pages. FMP's contract requires an FMP-approved EULA for public display and counts anonymous viewers of FMP-derived data toward its pricing tier. EDGAR data is a US government work: no licence, no display restrictions. Grades computed from SEC statements on the public /grade/{TICKER} pages carry none of those terms. Prices on the page still come from FMP, so this reduces the exposure rather than removing it, and the page says which numbers came from where.
2. Provenance. "Computed from Apple's 10-K filed 2025-10-31 (accession 0000320193-25-000079)" turns the auditable-arithmetic promise into a link anyone can click and check against the filing on EDGAR.
3. Point-in-time honesty. Every SEC fact carries the date it was filed, and an amendment is a new fact rather than an overwrite. GMI can answer "what did the filings say as of date X", which fixes restated-data look-ahead in grade history and in the Plutus backtests.
4. Vendor independence. When an FMP contract ends, FMP raw data must be destroyed and the destruction certified. The SEC store is GMI's to keep.
5. An independent cross-check. Two sources for the same number let GMI catch provider errors. The TSM reported-currency bug of August 2026, where Taiwan-dollar statements read as US dollars produced a $50-trillion market value, is exactly the class of error a second source exposes. The normalizer records each filer's reporting currency (TWD for TSMC).

The engineering judgment here is in the framing: a data project justified only by a vendor invoice dies when the invoice is small. This one was scoped so the strategic case carries it, and the money is reported honestly as a by-product, including when it is zero.

What EDGAR Publishes, and Using It Politely

The SEC's fair-access policy asks for a descriptive User-Agent with a contact address and at most 10 requests a second. gmi-secsync declares Grade My Investments inquiries@leoparddata.com, runs at 4 requests a second by default (hard-capped at 8, deliberately below the limit), backs off 2 s, 8 s, then 30 s on 403, 429 or 5xx, and counts every request in its run accounting. Being a good citizen of a free public service is cheap, and getting blocked by it is not.

The limits are real. Structured XBRL starts around 2009–2011, so the SEC offers roughly 15 years of history against FMP's 30+. Foreign private issuers file in the IFRS taxonomy, unevenly. And the SEC publishes no prices, analyst estimates, price targets, earnings calendar, news, ETF holdings or company descriptions. That list is why this was never going to be "replace FMP": it decides how the two sources divide the work.

Ingestion: A Service of Its Own

Decision (Q5): a new systemd service, gmi-secsync, on the report VM, rather than one more pass inside the schedules engine. A 1.4 GB bulk load or an EDGAR outage should never stall the engine that runs reports and alerts. It deploys from the master pipeline exactly like gmi-notifier. It never calls FMP and never serves users: its only job is to fill the SEC store and grade it.

• Ticker map. The first production sync, on October 15, 2026, read 10,444 SEC ticker rows and kept 6,066 filers in scope: listed on Nasdaq, NYSE or Cboe (OTC only behind SecData:IncludeOtc), 10-K/10-Q filers including banks and insurers, plus 20-F/40-F filers in ifrs-full.
• One-time bulk load. bulk-load streams companyfacts.zip to disk, then reads it one filer entry at a time, so memory is bounded by the largest single filer rather than the archive. It fetches each filer's submissions once for SIC and fiscal-year end, and has a --dry-run that counts facts kept against facts seen without writing a row, which was used to measure storage on the small MySQL server before the real load. It is a manual command: the daemon never starts it on its own.
• Nightly sync (07:30 UTC): refresh the ticker map, read the daily master index for each day since the watermark, fetch submissions + companyfacts for filers that filed a 10-K, 10-Q, 20-F, 40-F or an amendment, re-normalize them, then re-gate the whole universe, because the FMP side of the comparison moves too. Capped at SecData:NightlyMaxCiks (2,000) a night.
• Only what is used. A filer's companyfacts holds thousands of concepts. The store keeps only those a mapping rule references (active or not), plus the cover-page share count. A rule that starts using a new concept needs a back-fill (backfill-concepts, below); normalize re-derives from stored facts alone.

The one-time bulk load ran against production on October 15, 2026: the 1,345 MB archive downloaded in 27 seconds, then 5,899 filers loaded in 56 minutes with zero failures. Of 69,052,964 facts in those filers' files, the store kept 15,151,564 (21.9%), inserting 15,126,491 after de-duplication, and normalized them into 488,939 periods, using 5,900 SEC requests. On disk: secfacts 4.43 GB, secnormalized 2.40 GB. The burstable MySQL server peaked near 54% CPU and never drew down its CPU credits, so the app stayed responsive throughout.

Console commands: daemon, bulk-load, nightly, sync-cik, refresh-tickers, seed-rules, normalize, reconcile, status, and since October 16 gap-fill, backfill-concepts and apply-debt-suggestions.

Storage: Append-Only Facts, Versioned Projections

One small schema detail did real work. An instant fact (a balance-sheet value) has no period start, and MySQL unique keys treat every NULL as distinct, so a unique key over the period would never deduplicate instant facts and every re-load would duplicate them. A generated PeriodStartKey column gives instants a concrete value the key can collide on. The migration is idempotent and hand-applied, like every GMI migration, and publictickergradehistory gained one nullable DataSource column.

Normalization: About 80% of the Work

Downloading EDGAR is easy. Turning it into the same income statement FMP returns is the project. Companies tag the same line differently, banks have no revenue line, insurers earn premiums, foreign filers use another taxonomy, and nobody files a fourth quarter.

Mapping rules are data, with a small expression language

Each GMI field has an ordered chain of expressions. The first one that yields a value wins, and every value records which expression produced it. The language is deliberately tiny: A | B (first available), A + B (sum), !A (required, or the whole expression yields nothing), max(), neg(), and @field to reference another normalized field.

revenue         10  max(RevenueFromContractWithCustomerExcludingAssessedTax, Revenues)
revenue         20  SalesRevenueNet | RevenueFromContractWithCustomerIncludingAssessedTax | ...
netIncome       10  NetIncomeLoss | NetIncomeLossAvailableToCommonStockholdersBasic | ProfitLoss
grossProfit     20  !@revenue - !@costOfRevenue
capitalExpenditure  neg(PaymentsToAcquirePropertyPlantAndEquipment | PaymentsToAcquireProductiveAssets)
freeCashFlow        !@operatingCashFlow + @capitalExpenditure

Revenue starts with max() for a reason found in real filings: a company with lease or other non-ASC 606 income tags both the 606 subset and the all-in total, and the first-available rule would pick the smaller one. Outflows are negated because FMP reports capex and dividends as negatives.

Why data and not code: the rules are seeded from SecDefaultRules into secmappingrules, and from then on the table is the source of truth, editable from AdminWeb. Seeding only fills missing slots, so an admin's edit is never overwritten by a deploy. Tuning a chain for one awkward filer becomes a row edit plus normalize, not a release. Per-CIK overrides are tried first.

Bank and insurance sets list only what differs; any field they do not define falls back to us-gaap. Decision Q4 put all four in v1 scope, because the reconciliation gate makes breadth a workload risk rather than a grade-drift risk: anything that doesn't reconcile simply stays on FMP.

Periods are the other half

• Fiscal calendars are the filer's own. Non-December years and 52/53-week years are keyed on each filing's own report period, not on calendar quarters.
• Q4 is never filed. It is derived as FY minus the 9-month year-to-date figure from the Q3 10-Q (else FY − Q1 − Q2 − Q3). Shares and per-share values need their own arithmetic. Balance-sheet values (instants) for Q4 come straight from the 10-K.
• 10-Q cash flows are year-to-date. Q2 and Q3 flows come from differences of YTD figures.
• Amendments by filed date. The latest periodic report (since normaliser 1.2, never a proxy statement or 6-K) wins as of any date, while a citation points to the first filing that reported the value, so provenance names the document where a number first appeared.
• A measured, accepted difference: Q4 EPS and Q4 weighted shares differ from FMP by 0.5–0.9%, because FMP uses full-year weighted shares for Q4. Neither is a grade input, so the difference is documented rather than "fixed" into FMP's convention.

The totalDebt Investigation

GMI's grade reads three ratios FMP's key metrics supply. Only the fundamentals-only ones are recomputed from SEC statements; price-based ones (market cap, enterprise value, earnings yield) stay FMP's, because prices do. Each formula was pinned against real FMP payloads (a March 2026 Magnificent 7 report job) for Apple, Microsoft and NVIDIA, across every fiscal year in the payload, and matches to about 15 significant digits:

Presence mirrors FMP. If FMP left a ratio null, the SEC side leaves it null too. That sounds like throwing away data, but GMI's grade builder falls back to the latest quarterly balance sheet when debtToEquity is missing. Filling the gap from SEC would silently move the grade onto a different code path, which is exactly what this project promised not to do.

Then decision Q3 said "match FMP's debt definition exactly". Nobody documents what FMP's totalDebt is, so it was reverse-engineered from real FMP payloads against the XBRL:

Microsoft and NVIDIA match the same composition exactly, on every annual and quarterly balance sheet in the fixture. The catch is the word face. FMP omits any component a filer buries in "accrued expenses and other" or "other liabilities", and companyfacts carries no presentation data, so no single XBRL chain can reproduce FMP for every filer:

The resolution, and what was rejected. Chasing a cleverer universal formula was rejected: the information needed is not in the data. Instead the default chain is the full composition, the gate blocks the filers that disagree (so they stay on FMP and no grade moves), and the candidate compositions ship as inactive variant:totalDebt rules. The reconciler names the variant that would match, and an admin can apply it as a per-CIK override with one click. It is never applied automatically. (When the list reached 343 such suggestions, a human-run bulk command took over the clicking; see Clearing the Work List in Bulk, below.)

The Reconciliation Gate: Nobody's Grade Changes Silently

Decision Q2 set the bar: a symbol becomes SEC-approved only when every grade input agrees with FMP within 0.5% for the periods the grade uses, and the real grade engine, run once on FMP data and once on the SEC-overlaid twin, produces the identical letter. The gate does not re-implement grading; it calls the same DashboardSummaryBuilder.BuildSymbolSummary the product uses.

The inputs it compares are exactly what the grade reads:

• Annual revenue for the latest two years (revenue growth) and net income for three (growth plus the persistent-loss cap)
• Latest quarterly equity and total debt (the fallback path for D/E and ROE)
• Key-metrics ROE, current ratio and D/E, only where FMP supplied them
• Debt and current ratio are skipped for Financial Services, which the published recipe does not score on them

The first evidence, on real FMP payloads for the Magnificent 7:

All seven letter grades were identical. A gate that only compared letters would have approved all seven, and four of them would have been sitting on inputs that disagree with FMP by more than half a percent, one market move away from a different grade. The letter check catches today's drift; the input check catches tomorrow's. That is why the gate blocks on inputs, not just on the letter.

The work list. Every disagreement lands on the AdminWeb SEC Data Store page with its evidence (period, field, SEC value and the concept and accession it came from, FMP value, relative difference) and four actions: fix a mapping rule, mark the symbol FMP-only, accept with a recorded reason, or reset. FmpOnly and AcceptedOverride are sticky against the nightly re-gate, and an override cannot be saved without its reason.

The first universe-wide reconciliation (20 minutes, 5,297 symbols) approved 1,107 (21%) for SEC-primary: 926 of 3,396 US filers (27%), 181 of 1,147 banks and insurers (16%), and 0 of 754 foreign filers. The other 4,190 stay FMP-primary, each with the field and period that blocked it. The leading blockers were total debt (1,086 symbols, the "face of the balance sheet" pattern above), return on equity (762), net income (648), revenue (593) and current ratio (414). A clean zero for foreign filers is itself a finding: it points at something systematic rather than 754 separate mapping mistakes, so it went first on the work list (it turned out not to be currency; see below). A strict gate that approves a fifth of the universe on day one is the design working: nobody's grade moved, and every remaining symbol has a specific, fixable reason.

Foreign Filers, 0 to 230: It Wasn't Currency

Problem. Zero of 754 foreign filers passed. A clean zero looks like one systematic bug, and after the TSM bug, reporting currency was the obvious suspect.

How it was found. Currency was checked first and ruled out: ASML in euros and Novo Nordisk in Danish kroner match FMP to the unit on every annual figure. The zero turned out to be four unrelated causes that together covered the whole segment:

Result. Foreign SEC-primary went from 0 to 230. The lesson is about diagnosis: a suspiciously round failure count invites a single explanation. Testing the favourite suspect first, on real filers, is what stopped a currency "fix" that would have changed nothing.

Clearing the Work List in Bulk, Without Self-Filling It

Problem. After the foreign fixes, 1,323 symbols were SEC-primary. Two patterns dominated what was left, and neither was a mapping mistake.

FMP zero = SEC blank. When FMP reports 0 for a line the filer never tags (a holding company's or a bank's "revenue"), the overlay serves FMP's 0 on both sides, so what a user sees is identical. That one comparison was blocking 196 symbols on its own (634 rows). It is now recorded as a Match with a note. A non-zero FMP value with nothing at the SEC still blocks.

343 debt compositions nobody was going to click. For about 343 US filers the gate had already named a variant totalDebt composition that matches FMP (META-style "excluding finance leases", TSLA-style "excluding operating leases", AMZN-style "noncurrent only"). Applying them one click at a time in AdminWeb wasn't practical. Allan's decision was a command, apply-debt-suggestions, with the gate as its referee:

1. Write each suggestion as a per-filer override (priority 1, UpdatedBy = auto-suggest, visible and revertible on the Rules tab), never over an existing override.
2. Re-normalise those filers and re-run the full gate: every input within 0.5% and the identical letter.
3. Roll back every override whose filer still disagrees on totalDebt or debt-to-equity, and re-normalise it back.

It has a --dry-run that lists the picks by composition, and it is not nightly: a new match is applied only when a person runs the command again. That keeps the alerts-flood rule intact. A human starts it, and the gate, not the command, decides what stays.

Result. 343 kept, 0 rolled back. By that evening SEC-primary stood at 1,728 of 5,297 symbols (October 16, 2026), up from 1,107 the day before.

The SEC's Own API Has Gaps

Problem. 263 US symbols failed because FMP's latest quarter was "not in the SEC filings". For 141 of them the quarter genuinely wasn't in the store.

How it was found. The natural assumption was late filers, and it was wrong. Every sampled company had filed its June-quarter 10-Q, with XBRL, in late July or early August 2026 (Abbott, Pentair, Huntsman, CubeSmart, SL Green, Oshkosh and others). The SEC's companyfacts API, and the bulk zip built from it, silently omits some filings. Measured against the SEC's quarterly Financial Statement Data Sets (FSDS), about 250 listed filings were missing across four quarters:

That overturned the foreign-filer write-off above: TSMC and Toyota were never an "IFRS taxonomy lag we cannot fix". The same filings are in FSDS.

Decision: fill from the SEC's other official dataset, not from FMP. gap-fill <quarter> downloads one quarter's FSDS zip (~65 MB, one request), picks the periodic filings by known filers whose accession has no facts in secfacts, converts their rows to facts, then re-normalises and re-reconciles only those filers. FSDS is not a drop-in copy. Three quirks had to be handled:

• Dates are rounded to month end. Apple's quarter ending 2026-06-27 arrives as 2026-06-30, and a 52/53-week filer's period would then never line up with its own history. Each date is snapped back to the filer's nearest known exact period end (from stored facts, plus the filing's exact reportDate from submissions, one request per filer). Durations become exact starts the same way.
• Per-share units arrive as bare currency (USD) and are restored to USD/shares, so EPS stays non-additive.
• Dimensional and co-registrant rows are dropped, because companyfacts carries consolidated facts only, and the two paths must mean the same thing.

The proof. A unit test on real FSDS rows shows that Apple's June 10-Q via FSDS reproduces the companyfacts copy fact for fact: values, units, exact starts and ends. Abbott's missing quarter normalises to equity of $51.11B and net income of $0.928B. The production gap-fill is part of a combined run that has not happened yet, so its effect on the count is not claimed here.

Debt Shapes, and Concepts That Were Never Ingested

Problem one: no debt at all. For 98 US filers the latest quarter was stored but totalDebt was empty: the borrowings-plus-leases composition found nothing. Sampling them against FMP found four shapes, each equal to FMP on a real filer. They became fallbacks:

They run only when priority 10 finds nothing, so no filer that already had a value can move. The Apple, Microsoft and NVIDIA to-the-dollar tests still pass.

Problem two: a rule can't read what was never stored. Ingestion keeps only the concepts some rule references, which is what keeps the store small. The flip side is that a rule on a new concept has no facts behind it. backfill-concepts finds every referenced concept with no stored facts, re-reads the bulk zip for just those concepts across all known filers (full history, INSERT IGNORE), then re-normalises and re-reconciles only the filers that gained facts. The storage trade-off stays, and now it has a cheap way back.

Problem three: 949 mismatches with no candidate. 120 were sampled: 60 stratified by the direction and size of the gap and 60 at random, each checked against its full companyfacts at the reconciled quarter end. The finding was not about leases. It was borrowings tagged with concepts the chain never looks at: revolvers, notes and loans payable, convertible and senior notes beside LongTermDebt. The main chain found the leases, so it had a value, and the fallbacks never fired. Dropbox is typical: revolver 27.1 + 2,582.4, convertible 690.7, leases 389.3 + 285.9 = FMP's 3,975.4 ($m).

Eight shapes became eight inactive variant:totalDebt suggestions (#60 to #130), estimated to cover about 314 of the 931 US symbols. Each is pinned by a test on one real filer, asserting that the default chain disagrees with FMP and that the expected variant is the first one within 0.5%. What was deliberately not fixed matters as much:

• FMP-side anomalies (~16 in the sample, e.g. SNAP, BOX), where FMP omits disclosed borrowings. Matching them would copy FMP's error into the store.
• Company-extension tags (~25, e.g. Lennar's floor-plan notes). companyfacts carries only standard taxonomies; FSDS does carry them, a possible next step.
• One-off compositions below a ~5% bar (ORCL, QCOM, WY), and small unexplained residuals under ~5%.
• Rejected rules: DebtInstrumentCarryingAmount (a per-instrument disclosure) and "leases only" (which would only reproduce FMP's anomalies).

Honest caveat: for totalDebt the gate checks only the latest quarter, while an applied override governs every period of that filer, so a composition that matches by coincidence is caught only when a later quarter disagrees. The round-two variants and their new concepts have not been applied in production yet.

Normaliser 1.2: When a Proxy Statement Overrode the Audit

Problem. At 21:20 UTC on October 16, the US segment's worst blockers were net income (~406 symbols), return on equity (~395), current ratio (~232) and revenue (~160).

How it was found. Instead of guessing from a handful of filers, every blocking comparison was pulled from production read-only, companyfacts was downloaded for the 2,711 filers involved (at most 2 requests a second), and the normaliser's choices were replayed offline against FMP's value in each row. 217 currently approved filers were replayed too, as a regression check; none regressed. The uncomfortable result: most failures were ours, not FMP's. About 82% of the ROE failures came from that year's net income being off.

IFRS banks get their own rule set. A bank's IFRS Revenue is net of interest expense, while FMP's revenue for Canadian banks is the gross figure. ifrs-full-bank has one rule, revenue = !Revenue + !InterestExpense, and falls back to ifrs-full for everything else, the way us-gaap-bank falls back to us-gaap. Royal Bank FY2025: 66,605 + 70,825 = 137,430m against FMP's 137,357m (0.05%); CIBC, BMO and Inter&Co land within 0.22%. Detection needs interest income and a deposit or customer-loan line, plus a bank SIC for listed filers, because Wipro tags DepositsFromBanks. TD, HSBC, UBS and others still fail: FMP's bank revenue is not one composition, and no rule was bent to chase it.

Suggestions, not fixes, for the rest. Where FMP simply chose a different net-income or revenue line (to common vs including minority interests; ASC 606 only vs all-in), the reconciler now suggests a variant:netIncome or variant:revenue, but only if it matches FMP on the disagreeing row and on every other compared year, because a per-filer override changes every year. They are never applied automatically. One further change, treating a current ratio as a match when the filer has an unclassified balance sheet and the overlay keeps FMP's value on both sides, is written up as a proposal and waits for Allan's decision.

The offline replay estimates that normaliser 1.2 clears every blocking input for about 400 more symbols. That is an estimate, not a result: the identical-letter rule still decides, and the production run has not happened yet.

The Seam: One More Decorator, Zero Consumer Changes

Every GMI consumer already reads fundamentals through one interface, IFmpApiClient, wrapped by the caching decorator. The SEC store slots in as one more decorator:

consumer ─► SecFirstFundamentalsClient ─► CachingFmpApiClient ─► FmpApiClient
                 └► SEC store (MySQL): statements for SEC-approved symbols

It returns the same DTOs (IncomeStatement, BalanceSheet, CashFlowStatement, KeyMetrics), so grading, Investor Legends, reports, Earnings Setup and the Plutus export did not change a line. It is wired into the Functions API, both report generators, the SEO page builder (public pages and Legends), the notifier and the schedules engine.

• Overlay, not replacement. FMP's rows are the base and SEC fields overwrite them (field-level fallthrough). FMP periods the SEC lacks, such as pre-XBRL years, pass through (year-level fallthrough). The period set always equals FMP's, so a 22-year Legends request still gets 22 years.
• Outside the cache, on purpose. Putting SEC values into the FMP cache was rejected. The cache hashes payloads to detect restatements; SEC values in it would look like FMP restating itself, and the cache would stop being a faithful copy of what FMP said.
• Everything else passes straight through. Unapproved symbols, prices, quotes, estimates, targets, calendar, news, holdings, profiles, and any fault on the SEC side.
• Not even registered while off. With SecData:PrimaryEnabled false (the default everywhere), the decorator is not in the container at all: every host's IFmpApiClient is the object it was before, and tests prove fundamentals reads are byte-identical. The decorator checks the switch again itself, so a hand-wired host can't bypass it. New provenance properties are omitted from JSON when null, so cached FMP payloads and their restatement hashes did not change either.

As of October 16, 2026, both switches (SecData:PrimaryEnabled and SecData:ServeWithoutFmpCall) are still off in production: the store is loaded and reconciled, and turning it on waits for a human review of the work list.

How SEC and FMP Divide the Work

Two switches, and why the first one saves nothing

SecData:PrimaryEnabled turns on the overlay: FMP is still called (or read from cache) and SEC values are laid over it. That buys licensing, provenance and the cross-check, and it saves $0 by design, because the FMP call still happens. The money switch is SecData:ServeWithoutFmpCall: when the FMP cache would have missed and the SEC store holds at least the requested number of periods, the read is answered from the SEC store alone. It covers statements only, never key metrics, and the trade-off is written down: statement fields the mapping doesn't cover (otherExpenses, ebitda) come back empty on those reads. Every grade input is mapped. Both switches default off.

Splitting them was deliberate. The overlay can be judged on correctness alone; the SEC-only mode can be judged on the savings panel's real numbers before anyone decides the trade-off is worth it.

A Lesson Applied Up Front: No Alerts From a Data-Source Switch

GMI learned this one the hard way. When pay-as-you-go alerts launched on September 29, 2026, a detector's watermark started at zero and replayed the entire macro-regime history as if it were new, while free alerts defaulted on. Dozens of emails reached each affected account before the services were stopped. The fixes were a first-run baseline, a freshness guard, a flood breaker, every alert defaulting to off, and a rule: tests must cover RunAsync on a populated table, not just the per-row function.

A source switch is the same hazard in a new costume: thousands of symbols changing their data provider in one night looks, to a change detector, like thousands of fundamentals changes. So it was designed out before the first line of SEC code shipped:

• publictickergradehistory.DataSource records NULL for FMP (existing rows are unchanged) and SEC for SEC-sourced grades.
• The grade-change analyzer treats a change of source between consecutive rows like a formula change: no grade, score or factor events. The weekly digest drops a week-ago grade from the other source.
• There is nothing to announce anyway: the gate only switches a symbol whose letter is identical.
• It is proven at GradeHistoryDetector.RunAsync level, the layer the flood escaped through.

The same lesson shaped every default: gate decisions are never self-filled. Variant rules are suggestions an admin applies, overrides need a reason, the bulk load is a manual command, and both serving modes and the Plutus point-in-time artifacts ship off.

Point-in-Time: What Did the Filings Say As Of Date X?

Because secfacts is append-only and every secnormalized row is a version with KnownFrom/KnownUntil set by filing date, an as-of view is a filter, not a recomputation. Commercial fundamentals feeds typically give you today's restated numbers for every past year, and a backtest on restated data quietly knows the future.

• Grade history, as originally filed. Each fiscal year is graded from the statements as of that year's own 10-K, so a later restatement cannot rewrite what the grade was. Same formula, same AiGradeFormulaVersion.
• Investor Legends. StatementsAsOf hands the Legends inputs as known on a date, the basis for look-ahead-free backtests of the screens.
• Plutus. Decision Q7 shipped the point-in-time export with this phase: fundamentals-pit and grades-history-pit stream every version row to the sister trading laboratory, behind PlutusExport:IncludeSecPointInTime (default off). An admin endpoint, management/sec-store/as-of/{symbol}?date=, answers the same question for one company.

Honest gap: the Legends precompute uses SEC-first current data today; the as-of backtest path is exposed but not yet run by the weekly pass.

Survivorship-Free Research: Loading the Companies That Died

Problem. Allan wanted to run research studies across all companies, cheaply, from the SEC store. But the store kept only filers in today's SEC ticker map. A study over that set sees only companies that survived to October 2026: it is survivorship-biased, the classic way a backtest flatters itself. companyfacts.zip already holds every XBRL filer, dead or alive. The fix was to stop throwing the dead ones away, for studies, and never for the app.

Decision: an operating-company filter built only from what is in the zip. Fetching submissions for every extra filer would have cost about 50 minutes of SEC requests and a second resumability problem, so the filter uses zero extra requests. A not-listed filer is kept only if it has US GAAP or IFRS facts from a 10-K/10-Q/20-F/40-F, carries no investment-company concepts, passes deliberately narrow name heuristics, and has at least one mapped fact. Banks and insurers are recognised from their concepts instead of SIC codes, which are not in the zip.

Measured before it was loaded. The dry run reports listed and study-only filers separately, with the facts each would add, before a row is written. On that estimate the database storage was raised from 32 to 64 GB beforehand, rather than discovering the ceiling mid-load. The load is additive and resumable: it skips every filer already loaded, so a re-run after an interruption carries on, and a third run inserts nothing.

Result. 9,923 delisted operating companies loaded as study-only data, 12,522,749 facts, normalised with the same mapping rules as the listed universe. The study view, vw_sec_study_fundamentals, now covers 15,332 companies and about 972,000 company-periods, each row a point-in-time version with known_from/known_until, so a study can be both survivorship-free and look-ahead-free.

Never in the app. A filer's ListingStatus (Listed, Delisted or Unknown) is the boundary, enforced on every path rather than in one place: the gate's ticker map holds Listed filers only, serving requires a Listed filer (so a sticky approval left by a company that later delisted is ignored), admin decisions refuse non-listed symbols, coverage counts Listed only, the nightly sync refreshes Listed only, and the Plutus export leaves study-only rows out unless asked. A nightly guard also refuses to delist anything if more than max(100, 5%) of listed filers vanish in one refresh, because a truncated SEC response must not delist the universe.

Honest gaps: the filter is heuristic, "Delisted" means "not in GMI's universe today" rather than exchange history, and delisted companies have no prices in GMI (prices come from FMP by ticker), so studies are fundamentals-only for now: growth, margins, leverage, survival and restatements.

Where It Shows Up in GMI

Every surface uses one wording, generated by a single SecProvenanceText helper, e.g. "Computed from Apple's 10-K filed 2025-10-31 (accession 0000320193-25-000079)". One source of truth for the sentence means a reader sees the same claim on the web, on the phone, in a report and on a public page.

Operating It: The AdminWeb SEC Data Store Page

• Master-switch state, read-only. Flipping it is a configuration change, not a button.
• Sync status and coverage: share of the universe SEC-primary, split US / financials / foreign.
• The work list with evidence and actions: recheck, fix mapping rule, mark FMP-only, accept with reason, reset.
• Mapping-rule editor over secmappingrules, including one-click application of a suggested variant as a per-CIK override.
• Run accounting from secsyncruns: SEC requests, facts, periods, failures per run.
• Sources & Savings, below.

Its endpoints live under management/sec-store/*, never admin/: the Azure Functions host reserves that prefix and silently 404s anything routed under it while still listing the function as registered, a trap GMI has hit twice. MAUI parity was deliberately not built for an admin-only operations page, the same call as the FMP Cache Operations page.

Savings, counted honestly

Only a replaced real FMP call counts as money. "Would have missed" is decided by the FMP cache's own read predicate (CachingFmpApiClient.WouldServeFromCache) over the same watermark row, not by a second, drifting copy of the rule. Hosts that call FMP uncached count every pass-through as a live call. The counters batch in memory and flush every 30 seconds as one upsert per key. With SEC-only serving off, the panel reports $0 and says why: a savings panel that flattered the project would be worse than none.

Operations: Restart-Safe, Off by Default, Scoped

Every deploy re-ran the night. The daemon kept its "last run day" only in memory, so every restart after 07:30 UTC ran a nightly pass and a full-universe reconcile (~20 minutes, ~5,300 symbols) again. On a day of several promotions that is real load on a small MySQL server. On startup the daemon now reads today's run history and decides (unit-tested):

A one-filer sync-cik now records its own run kind, so it is never mistaken for the night's pass, and if the run history can't be read the daemon falls back to the old behaviour rather than doing nothing.

Quarterly gap-fill, behind a switch that ships off. With SecData:FsdsAutoGapFill on, each night becomes sync, then gap-fill, then the full reconcile, so one reconcile also re-gates every filer the gap-fill changed. It targets the most recent completed quarter from the 5th of January, April, July and October, logs a 404 as "not published yet" and retries the next night, and fills each quarter once (dry, skipped, partial and failed runs don't count, so a partial run retries just its failures). It is a new automated outbound behaviour, so, per the alerts-flood rule, it stays off until Allan flips it; the manual gap-fill verb works either way.

Re-process only what a fix touches. A full normalize covers 15,989 filers and takes about an hour. normalize and reconcile now take --cik, --ciks-file and --segment US|Financial|Foreign, combined as a union, so an IFRS rule fix is normalize --segment Foreign then reconcile --segment Foreign. The exception is normaliser 1.2 itself, which changes fact selection for every filer and therefore needs the full run.

Engineering Notes: Two Bugs the First Production Deploy Found

The 64 unit tests ran against real SEC filings and real FMP payloads, but on an in-memory provider. Production found two things they couldn't, both on October 15, 2026, and both fixed the same day.

1. Every SEC save failed. Replacing a filer's normalized rows is one unit: delete the old rows, insert the new ones in chunks, commit. The production MySQL context runs EF Core's retrying execution strategy, which refuses a user-initiated transaction outright ("MySqlRetryingExecutionStrategy does not support user-initiated transactions"), because it cannot retry half of a transaction it doesn't own. The in-memory provider has no execution strategy, so no test could see it. The fix runs the whole unit inside db.Database.CreateExecutionStrategy().ExecuteAsync(...), clearing the change tracker at the top so a retry starts clean. A retry now replays the entire delete-and-insert, which is the only correct thing to retry.

2. Every service stop was recorded as a crash. The daemon links a CancellationTokenSource to Ctrl+C and to ProcessExit. But ProcessExit fires after Main returns, by which time the source has been disposed, so Cancel() threw ObjectDisposedException and systemd logged every clean stop as a SIGABRT. Harmless to data, corrosive to operations: a service that "crashes" on every deploy trains people to ignore crashes. The fix is a three-line SafeCancel() that swallows exactly that exception.

How This Fits the Cost Picture (Wave 27)

The SEC store covers the statements share of FMP spend. The rest (prices, profiles, analyst recommendations and targets, the earnings calendar) is Wave 27's, built alongside it to make a full-market report as cheap as possible.

• Bulk turned out not to be on the plan. The FMP contract prices a Bulk query at $1.00 (one dataset for every symbol, against about $27 per-symbol across the universe), and the design leaned on it. A live probe on October 15 got HTTP 402, "not available under your current subscription", from every bulk endpoint. The bulk code stays, behind availability flags that default off, and the estimator now prices runs on honest per-symbol terms.
• A cost estimator and a $50 per-run cap. Every report run is estimated before its first FMP call, from real cache coverage. Over the cap, only an Admin-role override proceeds, and the report engine re-checks as the backstop for every host. The estimator knows about this store: a statement read is not counted as an FMP call when SEC-only serving is on, the symbol is approved and the store holds enough periods.
• The universe comes from FMP's company screener: 2,048 NYSE, 3,751 NASDAQ and 247 AMEX rows, filtered to US common stock.
• The arithmetic. A report symbol makes 12 FMP reads, so each read type costs about $27 across the universe and a cold full-universe run is about $297. Each SEC-approved symbol served without FMP removes six statement reads, $0.03, from a cold run. What remains is dominated by prices and analyst data, which no government source publishes.

With the first reconciliation's 1,107 approved symbols, SEC-only serving would take about $33 off a cold full-universe run, from roughly $297 to $264; at the 1,728 approved on October 16 the same $0.03 a symbol comes to about $52. Those are what the switch would save: ServeWithoutFmpCall is still off in production, so the realised saving today is $0. The two waves together answer the original question precisely: the SEC can take over the statements, and everything else needs a cheaper question to FMP rather than a different provider.

The Decisions, and What Was Rejected

Known gaps, stated plainly: the WASM clients' own direct FMP calls (quotes and search) are not decorated, the cache warmer intentionally is not, the health monitor does not list gmi-secsync yet, and the /stable key-metrics ROE formula is assumed to equal the older payload's. If it uses average equity, the gate, which compares against live FMP values, blocks those symbols and they stay on FMP. No grade moves.