Flying the System
Seven commits, a live schema migration, and a production incident diagnosed in three minutes — on a running platform, with no maintenance window and no staging environment. Why that is a design property rather than a risk appetite.
No Maintenance Window, No Staging Environment
Grade My Investments has no scheduled downtime. It also, deliberately, has no staging environment — the duplicate development environment was decommissioned in August 2026, and production became the place changes get proven. Continuous integration builds from the main branch and deploys straight to production; a change is verified locally, then promoted, and the next thing that happens is real traffic touching it.
That is a stage-appropriate decision, not a universal recommendation. At demo scale a second full environment cost real money every month and produced a weaker signal than the alternative: shipping changes whose worst outcome is the behavior that already existed. The rest of this page is about what "whose worst outcome" has to mean in the code for that trade to be sound — and one afternoon that exercised every part of it.
A Worked Example: One Afternoon
On 31 August 2026 a new subsystem was designed, built, shipped, instrumented, and repaired between lunch and dinner. The platform served traffic throughout; scheduled report jobs ran on their normal cadence; nothing was rolled back.
| Time (UTC) | What happened |
|---|---|
17:05 | Retention decision recorded: cached news is kept permanently as a backtesting corpus |
17:43 | Market-wide news wire poller promoted to production |
17:55 | Storage inventory and five-year projection published, measured against the live database |
18:21 | First production poll: 750 articles fetched, 622 new after dedup, 422 symbols, 3 API calls. The news cache went from 692 articles to 1,314 in a single poll |
18:59 | Admin status card shipped — writing to an accounting table that did not exist yet |
21:00 | The admin screen is opened. The wire card reports "status unavailable"; the cache statistics card fails once |
21:03 | Telemetry read, cause identified, migration applied by hand to production |
21:04 | The next scheduled poll files the first accounting row — 2 calls, 500 articles fetched, 26 new, 307 symbols — and the card lights up |
21:04 | Schema snapshot re-dumped from production and promoted, keeping the repository honest |
21:31 | Platform statistics across the technology site re-measured and corrected |
21:36 | A project that had been building and deploying for months, but was missing from the solution file, is added |
The interesting entry is 18:59. Code that writes to a missing database table
was deployed to production on purpose, two hours before the table existed, and the
platform's response was a log line.
Every Change Degrades to the System It Replaced
The property that makes an afternoon like that ordinary is not test coverage, and it is not confidence. It is that each change is built so its failure mode is the behavior that existed before it. Three mechanisms carry that weight:
- Schema changes are additive and land first. New tables and nullable columns only — never a rename, a drop, or a narrowing of a type that running code still reads. An additive migration is invisible to the deployed version, which means the two can be applied in either order without a coordinated release. When the order slips, as it did here, nothing breaks that was not already new.
- New writes are fenced. The poller records its accounting in its own error boundary, separate from the work it is accounting for. A missing table produced one warning per poll and zero interruption: articles kept flowing into the archive for two hours while the bookkeeping had nowhere to go.
- Readers degrade instead of failing. The status card renders independently of the rest of its page and treats an unavailable status as a state to display, not an exception to propagate. A visitor saw a card that said it had no data yet — not a broken screen, and not a broken neighbor.
The same reasoning governs the feature itself. While the poller is alive it refreshes a sentinel row that tells the on-demand path to stand down; if the poller ever stops, the sentinel goes stale within thirty minutes and per-symbol fetching resumes on its own. There is no failure of the new subsystem that produces anything worse than the system that preceded it — which is the same design the memory governor and the report build cache are built on.
flowchart TB
C[Change verified locally<br/>promoted to main branch]
M{Schema change needed}
A[Apply ADDITIVE migration first<br/>new table or nullable column only]
D[Deploy code that tolerates<br/>the dependency being absent]
R{At runtime: dependency present}
N[New behavior<br/>accounting rows written<br/>status card renders data]
O[Previous behavior<br/>one warning logged<br/>card renders 'unavailable']
W[Work itself continues<br/>articles keep flowing]
C --> M
M -->|yes| A --> D
M -->|no| D
D --> R
R -->|yes| N
R -->|no| O
O --> W
O -.self-heals when the dependency lands.-> NThe Part That Actually Broke
Two cards on one screen failed at the same moment, which is exactly the situation where a plausible story beats a true one. The tempting conclusion — the new deployment took the page down — was wrong, and the telemetry said so in about three minutes.
Application Insights records requests and exceptions separately, so the two failures
could be separated by cause rather than by proximity. The wire endpoint returned
500 twice with Table 'fmpnewspolllog' doesn't exist — the
known, expected consequence of the pending migration. The cache statistics endpoint
returned 500 once with a MySQL command timeout after 32 seconds, then
returned 200 in six seconds fifty-two seconds later, unchanged and
unattended. One was a missing migration; the other was a cold worker meeting a heavy
aggregate query, and it had already healed itself before anyone looked.
Both conclusions were reached without touching production state. That matters more than the speed: the reflex to restart a service, clear a cache, or roll back a deployment is how a transient blip becomes an outage. Evidence first, then the smallest action the evidence actually supports — here, applying one migration and refreshing a page.
The Standing Discipline
Flying without a net is only sane if the same small rules are followed every time. These are the ones this platform actually runs on:
| Rule | What it prevents |
|---|---|
| Migrations are additive, and applied before the code that depends on them | A deployment that cannot run against the schema in front of it |
| New writes get their own error boundary | Bookkeeping failures taking down the work being booked |
| New reads render a state, never throw a page away | One unavailable panel breaking the screen around it |
| Background loops schedule their next run before doing work | A failing dependency turning into a hot loop that burns budget or quota |
| Recurring spend is bounded structurally, not by intention | A runaway process discovered on an invoice rather than in code review |
| Counters that matter live in a table, not in memory | Losing the evidence every time a process restarts |
| The committed schema snapshot is re-dumped from production after every migration | A repository that quietly disagrees with the database it describes |
| Diagnose from telemetry before changing system state | Turning a self-healing blip into a real incident |
None of these are exotic. What is unusual is applying all of them to every change, including the small ones — which is precisely the part that used to be uneconomical.
What AI Assistance Actually Changed
The obvious claim about AI-assisted development is speed, and speed is the least interesting part. Seven commits in an afternoon is a throughput number; it says nothing about whether the afternoon was wise.
The real shift is in what becomes affordable. The defensive scaffolding described on this page — the fenced write, the degrading card, the sentinel that hands control back, the accounting table that survives a restart, the structural spend ceiling, the twelve tests that prove the poller stops paging at overlap and refuses to exceed its budget — is the work that gets cut first under deadline pressure. It is unglamorous, it does not demo, and it is exactly what determines whether a change can be shipped to a live system without a maintenance window.
When that scaffolding costs an hour instead of a week, it stops being a luxury and starts being the default — and the ability to ship continuously into production follows from it rather than from courage. The afternoon above is not evidence that moving fast is safe. It is evidence that a system built to fail backwards can be modified while it runs, and that building systems that way is now within reach of one person.