My Database Lied To Me For Seven Weeks

The last few posts were about work that failed quietly. This one is about the quietest failure I've found yet, one where the system that was supposed to report the problem was deleted by the same accident that caused it.

The ledger and the truth

Databases like ours keep a ledger of every structural change ever applied: new tables, new columns, scheduled jobs. Deploys read that ledger to know what still needs running.

Sometime in June, during a stretch of deploy troubles, four changes got marked as applied in the ledger without actually running. The ledger said done. The database said nothing, because nobody asked it.

Two things were in those changes. A set of tables for SMS reminders, which means the "text the guest before the recording" feature I shipped in June has never once sent a text. And a tracking column plus a logbook table for the automation that turns finished episodes into show notes and social posts.

The part that still impresses me

Here's what the second one did.

Every thirty minutes, the automation woke up and asked the database for episodes ready to process. That question referenced the missing column, so it failed, every thirty minutes, for seven weeks.

The automation is built to handle errors. It caught the failure and wrote it to its logbook, like it's supposed to.

The logbook was the other thing the ledger lied about. It didn't exist either. And the logbook writer is deliberately forgiving (a logging hiccup shouldn't crash real work), so it shrugged and moved on.

The error and the error report were deleted by the same accident. Every run finished in under half a second and reported success. Green lights, forever.

What caught it

Not an alert. A number that didn't fit.

I was verifying an unrelated change and noticed the automation finishing in 432 milliseconds when it had hundreds of episodes waiting. That's not "fast." That's "didn't do anything." Refusing to accept that number, actually pulling the thread instead of moving on, is the entire reason this got found.

The repair, in the right order

Fixing it meant re-running the four changes. But there's a trap: the moment that missing column comes back, the automation wakes up and starts processing several hundred episodes, real AI spend, kicked off as a side effect of a database repair. After the billing surprise in the last post, no thanks.

So the fix shipped in two steps, in an order that matters. First, an off switch for the automation, deployed and turned on while everything was still broken, so there was never a moment where the woken code could run unsupervised. Then the schema repair. The automation woke up, checked the switch, wrote "paused, doing nothing" to its brand-new logbook, the first entry that logbook has ever held, and went back to sleep.

One log line, and it proved three repairs at once: the column is back, the logbook works, and the switch holds.

What I'm keeping

Records about the system are not the system. The ledger, the task tracker, the "Done" column: they describe intent. Only the database tells you what's real, and it's worth asking it directly once in a while.

And the quietest failures aren't the ones with no error message. They're the ones where the error message was sent, to a place that doesn't exist.

PodGlue's whole promise is that nothing about your podcast quietly falls through the cracks. It turns out earning that means applying the same paranoia to my own infrastructure that the product applies to your episodes. This week, it finally does.