Production Postmortems

Real incidents, real dates, real root causes. Published because platforms that only tell you about their good days are telling you nothing at all

THE INCIDENT DISCIPLINE

rendering diagram…

flowchart LR
    INC[Incident detected<br/>often by dogfooding] --> DIAG[Root-cause diagnosis<br/>live probes, logs, schema checks]
    DIAG --> FIX[Immediate fix<br/>shipped same day]
    FIX --> PREV[Class-of-bug prevention<br/>compute-don't-trust, pipeline-owned<br/>config, never-null counting]
    PREV --> MEM[Written lesson in<br/>institutional memory<br/>+ this public page]
    MEM -. next incident starts<br/>from accumulated wisdom .-> INC

Every incident ends in three artifacts: a fix, a class-of-bug prevention, and a written lesson in the project's institutional memory — so the same mistake needs new creativity to happen twice.

The Compression Rule That 500'd Everything — May 27, 2026

Impact: the technology site went down in both environments with HTTP 500s at IIS startup — total outage for the affected apps.

Root cause: a Brotli/Gzip URL-rewrite optimization added allowedServerVariables rules to web.config. On Azure App Service, IIS rejects server-variable declarations at the application level — the config that worked locally was fatal at the host. A performance improvement shipped as an availability incident.

Fix & lesson: rules removed, sites restored. The durable lesson went into the deployment pipeline itself — the pipeline now writes a known-good web.config inline, and the incident is cited in a comment at the exact spot a future optimizer would repeat it. Config that can kill the host belongs under the pipeline's control, not a developer's enthusiasm.

The $50 Trillion Company — August 16, 2026

Impact: Taiwan Semiconductor displayed a market cap of ~$50 trillion — about half of world GDP — in report output. No availability impact; a correctness incident of the worst kind: a plausible-looking page containing an absurd number.

Root cause: the data vendor's key-metrics endpoints return values in each company's reported currency (TWD for TSM) while prices are USD. A derived metric multiplied across the two without conversion. The bug was in trusting a vendor-derived field's implicit unit.

Fix & lesson: caught within hours because the founder's own portfolio runs through the platform and the number was personally unbelievable. The fix went beyond the symptom: valuation and quality metrics are now computed in-house from raw statements only — immune to vendor unit and schema surprises by construction. Dogfooding is a detection system; "compute, don't trust" is the prevention. (Full data-trust policy: Data Provenance.)

The Charges That Silently Never Happened — discovered August 17, 2026

Impact: report-generation charge records had silently failed to write for weeks — cost-tracking rows were missing, and a database constraint was also rejecting the free tier's legitimate $0 usage records. Nobody was overbilled; the business was underinformed.

Root cause: schema drift. Migration scripts in the repository were never auto-applied to production; a months-old migration adding a cost column had never run, so every charge insert threw — and the exception was swallowed on a background path. A separate check constraint (Amount <> 0) predated the free tier's zero-amount design.

Fix & lesson: schema reconciled, constraint relaxed to non-negative, cost values backfilled. Process change: every migration now ships with an explicit apply-to-prod step in the deployment sequence, verified against the live schema before dependent code deploys — and background-path exceptions that represent money are logged at error level, never swallowed. Silent failure is the most expensive kind; a system must be loud about the things it failed to record.

The Viewer Counter That Counted Nothing — August 22, 2026

Impact: the report-sharing feature's "unique viewers" analytics recorded zero, ever — including real, verified visits. Owners saw 0 where engagement existed; the data-license viewer accounting had no rows.

Root cause: the code resolved the viewer's IP from a request-header API that, in the deployed Functions hosting model, never surfaced the forwarded client address — and a missing IP was treated as "skip counting" rather than an error. Diagnosed with a live probe: a request that returned 200 while writing no row proved the failure server-side.

Fix & lesson: IP resolution now chains the hosting model's native context, forwarded headers, connection address, and finally a fixed "unknown viewer" bucket — a view always counts, at worst with reduced dedup precision. Counting failures now log at error level. Analytics code needs the same failure discipline as billing code: a metric that silently records zero is worse than no metric, because it answers confidently and wrong.

Why These Are Public

Every production system has incidents; the only variables are whether the operator learns from them and whether users get to see the learning. This site is GMI's site of record — the same transparency that publishes the full grade algorithm and the model's blind spots applies to its operational history. Each incident above produced a fix, a class-of-bug prevention, and a written lesson in the project's institutional memory. The bug count isn't zero. The same-bug-twice count is.