The Fix That Worked, And Quietly Charged Me Twice

Two posts ago I wrote about episodes that stopped without telling me. Last post, about moving the transcript work somewhere it had room to finish.

This one is about what happened after that, which is that it worked and I still had to redo it.

Working, technically

I moved the transcript job onto a bigger machine with an hour to finish. Then I did something subtle and wrong: I had it say "got it, working on that" and then start the work after it had already replied.

That sounds harmless. It isn't, on the kind of service I was using. Once it replies, the machine effectively goes to sleep. The work I'd started keeps going only when something else happens to wake it up.

So transcripts did come out. Just slowly, and unpredictably, depending on what else happened to be going on. One took forty-three minutes for what should have taken two.

Where I got it wrong twice

First, I decided it was completely broken. I watched for sixteen minutes, saw nothing finish, and called it dead.

It wasn't dead. It was slow. The only reason I caught my own mistake was that a number didn't add up later, more episodes had transcripts than my story allowed for. If I hadn't gone back and checked, I'd have written up a confident explanation of a thing that wasn't happening.

Second, and more expensive: I thought the problem was the stalling. It wasn't.

The part that cost money

The system has a safety rule: if a job doesn't finish within twenty minutes, assume something crashed and start it again. Sensible.

Except the jobs were taking forty minutes. So the safety rule kept starting a second copy of work that was still running. Sometimes a third.

Of thirteen episodes that got transcribed that morning, eight were transcribed two or three times. About twenty-one transcriptions to produce thirteen transcripts. Every duplicate is a full audio download and a full charge from the AI service.

The safety net had quietly become a bill. And nothing about it looked wrong from the outside, episodes were completing, transcripts were appearing. It was working. It was just working expensively, and the only way to see it was to look at how many attempts each success took, not whether it succeeded.

What I changed

Now nothing gets started and abandoned. A scheduler calls the machine, the machine does the work while the call is still open, and it only answers when it's done. Boring, and boring is the point.

A few things fell out of that which I like:

The batch runs several episodes at once. They're mostly sitting there waiting on the AI anyway, so four at a time costs about the same as one.

Episodes that already have a transcript get skipped instantly instead of being redone.

And I flipped which AI service gets tried first. I'd been trying the cheaper-looking one first and falling back, but a failed attempt still costs a download and a call, and then the second service downloads the same file again. Trying the one that handles everything first means I stop paying twice for one episode.

Did it work

Six transcripts in thirteen minutes. Eight jobs, eight different episodes, no duplicates, every fresh one finishing on the first attempt.

And a batch of four finished inside five minutes, which told me the forty-three-minute version was never the AI being slow. It was a two-minute job being dragged out across an hour of interruptions.

The lesson I'm keeping

"It's producing output" is not the same as "it's working."

The stalled version was obvious once I looked. The version that quietly ran everything twice looked completely fine, right up until I counted attempts instead of successes. If you've automated something that costs money per run, the number worth watching isn't how many finished. It's how many tries each one took.

That's the whole reason PodGlue tracks attempts on every job now, not just outcomes. An episode that took three goes to succeed isn't a success story. It's a bill I didn't know I was paying.