We spent hours rewriting the retry logic in the NTK CLI. We added tests. We confirmed the logic was mathematically perfect. But in production, the retries never triggered, and the application just crashed.

We were fixing a bug that didn’t exist.

When a bug report comes in, the temptation is to immediately form a hypothesis, look at the code, and write a logical fix. But a logical fix applied to the wrong hypothesis isn’t a fix at all. It’s just a new bug wearing a trench coat.

The Environment, Not the Code

We had an error classifier designed to intercept and categorize network failures. When the Notion API timed out, the classifier was supposed to catch the specific network error, tag it as a transient issue, and trigger our newly perfected retry loop.

The code parsed the error object and matched the exact error codes emitted by Node.js, such as ECONNRESET and ETIMEDOUT.

The breakthrough came only when we stopped looking at the code and started looking at the execution environment. The classifier was built and tested under Node.js. But the CLI was actually being executed by Bun in the production environment.

Bun’s internal networking stack is entirely different from Node’s. When a timeout occurred, Bun didn’t emit ECONNRESET. It emitted its own set of internal system errors with completely different naming conventions. Our classifier was looking for signatures that the runtime was fundamentally incapable of producing.

Evidence over Hypothesis

We had formed a hypothesis (“the retry logic is broken”) and wrote a fix for it, without ever gathering the evidence (“what exactly is the runtime throwing?”).

Before you write a line of code to fix a bug, prove the bug exists exactly where you think it does. A perfectly logical fix is useless if it’s applied to an environment that doesn’t share your logic.

---

Next up, we shift gears to the Medulla engine to explore how parsing markdown from LLMs leads to silent data loss in Machine Readable Verdicts.

• #bun
• #node
• #ntk
• #engineering
• #debugging