The $1,600 Question That Costs $10

Keeping live news warm for 5,500 stocks — and why the winning design was not a cheaper version of the obvious one, but a different question entirely

The Cost-Shape Trap

The platform caches stock news in MySQL and refreshes it on demand: open a screen, and any symbol not checked in the last 30 minutes gets a batched API refresh. That read-through design is frugal — you pay only for what someone looks at — but it has a cold-start smell: a list nobody has opened recently has no news until the first viewer pays the fetch-and-wait.

The obvious fix is a background sweep: refresh every tracked symbol on a timer. Priced out, the obvious fix is a trap. Per-symbol news costs one API call per 50 symbols, so a 5,500-symbol universe is 110 calls per sweep — $0.55 each time. Sweep every 15 minutes around the clock and the bill is ~$1,600 a month, for a feature that mostly re-downloads articles it already has. The cost scales with symbols × frequency, and both multipliers only ever grow.

Ask a Different Question

The sweep asks the provider 5,500 small questions: "any news for AAPL? for MSFT? for…" But the provider also exposes the answer to one big question: "what was published anywhere in the market since I last asked?" — a market-wide wire, newest first, every article already tagged with its symbol. The whole US market publishes a few hundred stock articles per 15-minute window; one page of the wire holds 250. Draining it is 1–3 calls per poll, flat, regardless of universe size.

Same articles, same freshness, same destination rows in MySQL. The only thing that changed is the shape of the question — and with it, the cost function: from O(symbols × frequency) to O(frequency). Tracking twice as many stocks now costs zero additional dollars. That is the entire feature.

ARCHITECTURE

rendering diagram…

flowchart LR
    FMP[FMP market-wide wire<br/>news published anywhere<br/>newest first]
    P[News poller<br/>every 15 min, market hours<br/>1-3 calls per poll]
    DB[(MySQL news cache<br/>immutable rows, URL-hash dedup<br/>kept forever)]
    S["@FEED sentinel<br/>fresh = wire is active"]
    R1[ClientWeb / AdminWeb / MAUI<br/>news screens and drawers]
    R2[Backtesting consumer<br/>reads the archive]
    FMP --> P --> DB
    P --> S
    S -.suppresses per-symbol fetching.-> R1
    DB --> R1
    DB --> R2

One poller drains the market-wide wire into the MySQL cache; every reader — three apps and the trading-research consumer — reads MySQL and never touches the API. The @FEED sentinel tells read-through the wire is active, so screen loads stop spending per-symbol calls; if the poller ever stops, the sentinel goes stale and on-demand fetching resumes by itself.

A Cost Guarantee You Can Read in the Code

The budget promise ("about ten dollars a month") is enforced structurally, not aspirationally — three independent bounds, any one of which caps the bill on its own:

A polling window. Weekdays, 13:00–21:30 UTC — wide enough to cover US market hours in both daylight-saving regimes plus the pre-open and closing-bell news bursts. Nights and weekends spend nothing: 34 polls a day, not 96.
A per-poll page cap. Paging stops at overlap — the moment a page inserts zero new rows, the wire has been drained back to the previous poll. A hard three-page ceiling bounds even a news hurricane at 3 calls.
A daily call budget. A counter refuses further polls past 100 calls in a UTC day. Worst case is therefore arithmetic, not hope: 100 × ~21 trading days × $0.005 ≈ $10.50/month ceiling; typical days run near half that.

Two quieter decisions matter as much. The poll schedules its next due time before doing any work, so a failing API or database cannot hot-loop the poller into burning budget every scan tick. And the handoff to the existing read-through path is a sentinel row in the fetch-state table, refreshed on every successful poll: while it is fresh, screens trust the wire and spend nothing; the moment the poller goes quiet for 30 minutes, the old on-demand behavior resumes — the feature's failure mode is the system it replaced, a property borrowed deliberately from the ML memory governor's design.

The Archive Is the Second Product

Cached articles are immutable rows deduplicated by a SHA-256 of their URL, and they are kept forever — by decision, not neglect. A second system (a morning-trading research platform sharing the database) will backtest against this archive, and a backtesting corpus is only as good as its completeness. That reframes the poller: it is not just a freshness feature, it is the collection instrument for a permanent market-news record. On-demand fetching archives only what someone happened to view; the wire archives everything published from the day it turned on — for the same flat ten dollars. Two products, one poll loop.

What It Buys

	Per-symbol sweep	Market-wide wire
Cost, 15-min freshness, 5,500 symbols	~$1,600/mo	~$5–10/mo (capped at $10.50)
Cost of doubling the universe	2×	$0
Cold-list first view	5–15 s fetch wait	instant — cache is always warm
Backtesting archive coverage	only what was viewed	everything published
If the poller dies	—	read-through resumes automatically

The lesson generalizes past news: when a recurring cost scales with universe × frequency, look for the API shape where the provider aggregates for you. The cheapest call is usually not a smaller version of the expensive one — it is a different question.