Code That Erases Its Own Evidence
We spent an hour chasing a ghost timeout in the Medulla engine. The code actually knew exactly why the failure happened, but it deliberately threw an exception right before it could write the explanation to the logs.
When a system fails, the first thing you look for is the trail of breadcrumbs. But what happens when the code that generates the breadcrumbs is the very thing that destroys them?
The Misplaced Throw
In our workflow engine, an execution script enforced strict timeouts on agent tasks. When a task ran out of time, the script was designed to calculate the exact failure reason, build a rich diagnostic context, and cleanly terminate.
For weeks, we saw random tasks fail with a generic rc=124 (the standard exit code for the Linux timeout command). The logs were entirely empty.
The missing logs weren’t missing because the calculation logic was wrong. They were missing because of sequence.
The script correctly calculated that a timeout had occurred. It built the rich diagnostic string: "silent for 300s after N lines". And then, exactly one line before the console.log(context) statement, it threw an exception to signal the failure to the parent process.
The exception immediately halted execution. The log statement was never reached. The script crashed violently, leaving behind only the generic rc=124 from the outer wrapper.
Diagnostics Are Not Decorations
We often treat logging and diagnostics as secondary concerns—things we add “after” the core logic is done. This is a fatal mistake.
When your system is failing in production at 3 AM, the diagnostic log is the core logic.
The fix was moving the throw statement down by exactly one line. But that one-line mistake cost us hours of manual arithmetic on timestamps in the raw logs just to prove what had happened. If your error handling erases its own evidence, it’s worse than having no error handling at all.
Next in the Medulla series: why a tool that works 99% of the time is far more dangerous than one that doesn’t work at all in Almost Correct Tools Are Worse Than Broken Ones.