In July, we merged a commit to our multi-agent engine with a confident title: feat: a body that stops talking is dead, not thinking.

It sounded perfectly reasonable. If an agent hasn’t emitted a token or a log line in a long time, it has likely crashed or hung. We needed a watchdog.

The Trap of the “Healthy” Baseline

To set the timeout threshold, we looked at our telemetry. On a healthy, typical run, the engine processed 298 events in 586 seconds. The absolute longest pause between any two events was 66 seconds.

Taking that 66 seconds, we applied a nearly 5x safety margin and set the silence timeout to 300 seconds. We considered this generous.

A month later, a panel of four different LLMs died simultaneously, exactly 1158 seconds into a run.

They were in the middle of a massive generation. One model had already written 190 kilobytes of text. They hadn’t crashed. They hadn’t hung. They were just thinking and generating a very large payload.

By calibrating our threshold on a “healthy” run, we had accidentally calibrated it on an easy task. A heavy task is silent longer precisely because it is working hard, not because it is dead. The watchdog had assassinated a perfectly healthy panel.

Code That Erases Its Own Evidence

The worst part wasn’t that the watchdog killed the agents. It was that it hid the murder weapon.

When the silence watchdog triggered, the code meticulously calculated the exact reason: "silent for 300s after N lines". It then took this beautiful, descriptive string—and immediately threw it away as an unhandled exception.

Because the exception bubbled up and crashed the process ungracefully, the master orchestrator log only recorded: body died: rc=124.

rc=124 is the standard POSIX exit code for a generic timeout. It looked exactly like a network failure or an infinite loop. It took an hour of manual arithmetic on timestamps in the raw logs to prove that the silence watchdog was the culprit.

The Takeaway

Two expensive lessons were paid for in a single outage:

1. You cannot calibrate timeouts on easy tasks. A system working at its limit behaves differently than a system cruising at baseline.
2. Diagnostics are not decorations. If your failure-handling code destroys the specific reason for the failure, it is worse than having no error handling at all. The absence of a diagnostic string isn’t just an inconvenience; it is a tax measured in hours of debugging.

• #architecture
• #failure
• #diagnostics
• #ai-agents