Machine Readable Verdicts
Our 5-agent review panel confidently approved a pull request while explicitly writing that they had found a critical security flaw. The code was merged automatically.
The models didn’t fail to identify the issue. The orchestrator failed to understand them because it relied on parsing Markdown prose.
The Markdown Illusion
When we built the Medulla engine’s review panel, we let the LLMs output beautifully formatted Markdown reports. To make routing decisions, the orchestrator used regex to search for keywords like “Approved” or “Blocked” in the text.
If a model encountered an edge case, it might write: “The code is largely approved, but I am blocking the merge because of a security flaw.”
A naive regex looking for Approved fires true. The pipeline proceeds. The blocking security flaw is silently ignored.
Parsing prose from an LLM is silent data loss. You are taking high-fidelity reasoning and reducing it to a brittle string match. When the match fails, the system doesn’t crash; it quietly executes the wrong action.
The verdict.json Gate
We threw away prose parsing entirely. The agents are still allowed to think and write their analysis in Markdown, but their final output must be a strict verdict.json file matching a rigid schema.
The schema enforces:
state: Enum ofGO,NO-GO, orABSTAIN.confidence: Float between 0 and 1.blocking_findings: Array of specific, actionable strings.
If the JSON is malformed, the agent’s vote is discarded. If the state is missing, the run fails.
The orchestrator no longer reads minds. It calculates a mathematical quorum over the state enums and aggregates the blocking_findings arrays.
LLMs are brilliant at reasoning, but terrible at consistent formatting. Let them reason in prose, but force them to communicate decisions in strict data.
Next in the Medulla series: how calibrating a timeout threshold on the wrong tasks led to a mass execution of healthy agents in The Silence Watchdog That Killed the Living.