A watchdog process assassinated a panel of four LLMs in the middle of a massive generation. The models hadn’t crashed, and they hadn’t hung. They were just working hard on a heavy task, but because they were silent for 300 seconds, the watchdog assumed they were dead and killed them.

If an agent hasn’t emitted a token or a log line in a long time, it’s reasonable to assume it has crashed. But setting the threshold for that assumption is dangerous.

The Trap of the “Healthy” Baseline

To set the silence timeout threshold, we looked at our telemetry. On a standard, healthy run, the engine processed 298 events in 586 seconds. The absolute longest pause between any two events was 66 seconds.

Taking that 66 seconds, we applied a generous 5x safety margin and set the silence timeout to 300 seconds.

A month later, the engine tackled a massive workload. One model was generating 190 kilobytes of text in a single block. During this heavy generation, the model was entirely silent, fully consuming CPU without emitting intermittent logs. At the 300-second mark, the watchdog triggered and slaughtered the entire panel.

Calibrate on the Limit

By calibrating our threshold on a “healthy” run, we had accidentally calibrated it on an easy task.

A heavy task is silent longer precisely because it is working hard, not because it is dead. A system working at its extreme limit behaves fundamentally differently than a system cruising at its baseline.

If you calibrate your fail-safes on average workloads, your system will actively destroy itself the moment it is asked to do heavy lifting.

---

Next in the Medulla series: what happened after the watchdog killed the agents, and how the error-handling code destroyed the only evidence of the murder in Code That Erases Its Own Evidence.

• #medulla
• #ai-agents
• #engineering
• #failure