Why the verification layer must exist: 8 failure classes in multi-agent systems

AgentGate is an open-source Python framework for making multi-agent systems reliable.

It solves a structural problem: in most systems, nobody verifies what agents produce. The orchestrators in LangChain, CrewAI and LangGraph all assume agent output is trustworthy and hand it straight downstream. That assumption fell apart for me after 100+ runs of a 12-step pipeline — success rate without verification was about 60%, and most failures were silent: the agent reported success, and the disk was empty.

AgentGate’s verification exists precisely for this. Three gates: produced files must actually exist and be non-empty; every Bash verification must carry an EXIT_CODE fingerprint — an agent can fake test results, but it cannot fake an exit code; client-facing deliverables pass a desensitization check. On top of that, a downstream agent randomly spot-checks the interface coverage an upstream agent claims. If any gate fails, the output is not released; the pipeline rolls back to the right agent by error type, and repeated failures escalate to a human gate.

This article is about why the verification layer must exist — and what class of failure each of the five layers catches.

Failures are not random: there are 8 classes

After being fooled by agents often enough, I started logging every crash. At some point no new ones appeared — multi-agent failures come in exactly 8 classes.

① Execution hallucination. The agent says “ran pytest, all 28 tests passed.” It didn’t run. The sentence was invented.

② Artifact hallucination. The agent says “main.py is done.” The file doesn’t exist. Downstream keeps working off the sentence, and several steps are wasted.

③ Capability hallucination. The agent says “all acceptance criteria covered.” Spot-check two; one isn’t done.

④ Process hallucination. The rule was: read the requirements before writing code. It skipped the reading; what came out doesn’t match the requirements.

⑤ Context hallucination. Upstream renames an interface from /api/users to /api/v2/users without telling anyone. Downstream calls the old one, fails, and burns several rounds of tokens before noticing.

⑥ Prompt hallucination. The same prompt works under Claude; under Qwen it gets understood completely differently. Not a dumber model — an unadapted prompt.

⑦ State loss. A 12-step pipeline dies at step 8. The first seven steps’ output lives in the context window; when it blows up, everything is gone. Restart — and this time the code comes out different.

⑧ Silent freeze. No error, no timeout, no crash. It just stops. You come back three hours later; it’s stuck at step 7.

All eight share one property: not one is a model problem. I switched from Claude to Qwen to small local models, and the crash rate barely moved. The hole isn’t in the model. It’s in the system structure — the segment between agents, where nobody is in charge.

Eight mainstream frameworks, none guarding this segment

I went through the docs of eight mainstream frameworks looking for output verification.

Framework	GitHub stars	Orchestration	Output verification
Dify	158K	Visual workflows	None
AutoGen	61K	Agent conversations	None
CrewAI	59K	Role collaboration	None
LangGraph	43K	State machine	None
PydanticAI	20K	Type-safe agents	None
LlamaStack	8K	Model serving	None
AG2	5K	Agent conversations	None
BeeAI	3K	Workflows	None

(stars as of Oct 2026)

None of them does verification interception. It’s not one framework’s oversight — it’s the industry’s default assumption: agent output is trustworthy. RAG solves “the cited fact doesn’t exist”; it solves none of the eight classes above. RAG won’t tell you the file is missing, and it won’t tell you the tests never ran.

AgentGate goes the other way: assume no agent output is trustworthy until verified.

The five layers: what each failure class needs

Gates are the core, but gates alone aren’t enough. Eight failure classes need five layers of defense.

① Memory layer → state loss. Project memory goes to disk, session logs record progress, checkpoints restore state. Agents stop vanishing when a call ends.

② Context layer → context hallucination. Upstream artifacts freeze into immutable snapshots; downstream reads a short summary first and decides whether to dive into the source. Total context has a budget, so even an 8K-window model doesn’t drown in a 15KB document.

③ Anti-hallucination layer → execution / artifact / capability / process hallucination. This is the three gates from the intro, plus cross-checking. One detail worth calling out: the EXIT_CODE fingerprint. An agent can say “all tests passed”; it cannot fake Bash’s exit code. Ask it to paste EXIT:0 — either it’s there, or the claim is invented.

These gates don’t block hypothetical problems. One project had UI style changes “not take” three times; two days of digging revealed two CSS files with identical selectors, the later one silently overriding the earlier. Another time a database “lost” all its tables — the agent connected via a relative path ./data/db.sqlite; the working directory changed, and it connected to a brand-new empty database. The gates catch exactly this kind of thing: does the file exist, is the exit code right, is the coverage claim true.

④ Prompt layer → prompt hallucination. Prompts are generated from a five-module structure and adapted per scenario. Every run scores its prompt — the scorer is a regex rule engine, not an LLM (whoever scores must not hallucinate). Below 60: auto-rewrite, and the retry uses the new prompt; 60–80: logged as a warning. Often the model isn’t bad — the prompt is unadapted.

⑤ Process layer → silent freeze. A stuck pipeline gets diagnosed to root cause automatically; repeated failures trigger a human gate with a structured question — not a blank input box; failures roll back to the right agent by type — code bugs to the developer agent, architecture to the design brain — instead of blind full restarts.

The five layers aren’t independent. The eight classes chain: an upstream artifact hallucination becomes downstream context hallucination. Miss one layer and a class leaks through — which is why they ship together.

Three non-obvious design decisions

The design brain is an LLM; the execution brain is not.

AgentGate has two brains. The design brain analyzes requirements, decomposes agents, defines acceptance criteria — creativity needed, so an LLM, used once, and it clocks out after producing a config file. The execution brain schedules, runs gates, decides rollbacks — not a single LLM call, a pure Python rules engine. Every PASS/FAIL, every rollback target, every retry count is hard-coded, traceable, reproducible.

Why? Because if the execution brain were an LLM, it would hallucinate too. CrewAI’s Manager Agent is exactly an LLM doing scheduling — it can make wrong decisions, and nobody verifies the Manager’s decisions. AgentGate takes distrust all the way: even the main brain is subject to the gates.

Constrain the hands, not the mind.

Gates only block verifiable things: file existence, exit codes, interface coverage. They don’t police how an agent thinks, what reasoning path it takes, what code style it writes. The gate is a floor, not a ceiling — only “not lower than,” never “how high.” Thought can’t be verified; constraining it just cages the model. Hands can be verified — so hands get constrained.

Don’t do model intelligence; do inter-call reliability.

AgentGate doesn’t train models or optimize inference — that’s the model vendors’ job. It does reliability between calls. A single call doesn’t matter; the more steps, the more it’s worth. Estimating from per-step independent error rates: a 12-step pipeline sees ~60% unverified success; with a verification layer, 95%+. That’s an estimate, not a benchmark — measured numbers to follow. Everyone is building engines; the brake lane is empty.

Some honest limitations

Orchestration does subtraction: only serial and parallel, no conditionals. Not impossible to build — deliberately not built. Config sprawls into code the moment it gets complex, betraying the declarative idea. In real retrospectives, the usual conditional needs — error recovery, content routing, empty-data adaptation — are covered by targeted rollback plus acceptance criteria.

No agent runtime. ReAct loops, tool calling, streaming are the underlying frameworks’ business; an AgentGate agent can be a LangChain wrapper or a bare LLM call.

127 tests pass; CI runs on every commit. MIT licensed, still iterating, issues welcome.

github.com/zw-onemaker-ai/agent_gate