Engineering post-mortems
What broke inside Vigil, how we found out, and the rule each failure left behind. These are post-mortems from running a proxy that sits in the live path of other people’s AI calls, written so the lesson transfers to any production service.
A usage meter a customer could reset by clearing their own history. API-key enforcement that switched off when an environment variable went missing. An account deletion that had never once succeeded. A Stripe webhook that never received an event because of a redirect. A Cloudflare setting that blocked real visitors while every check we ran kept passing. Each one ends with a test you can run against your own system this week.
Start here
Fail open vs fail closed: a missing env var, no auth
The failure whose rule, fail closed, applies to the most systems.
All 5 posts
Cloudflare Browser Integrity Check blocked our site
Cloudflare's Browser Integrity Check sent real browsers a 403 and logged nothing. An IP skip rule hid it for a week, while every synthetic check passed.
Usage metering: a quota the customer could reset
Our usage metering counted rows customers could delete, so the product's own "clear history" button reset their quota. The fix: a count that only goes up.
GDPR account deletion that never worked for anyone
Our GDPR account deletion referenced three tables that didn't exist, so every run failed with 42P01, after cancelling the customer's Stripe subscription.
Fail open vs fail closed: a missing env var, no auth
Our API-key check defaulted to off when its env var was unset, empty or misspelled, so deleting one variable disabled auth. The fix, and a rule about 401s.
Stripe webhook not working: a redirect nobody saw
Our apex domain 308-redirected to www, and Stripe does not follow redirects, so every webhook since launch died one hop before our well-tested code.
Vigil records the cost, errors and latency of every AI call you route through it; LLM proxy setup takes a base URL and one header.