AI writes code better every month — so why does delivery still feel like luck?

Over the past year, one contrast kept bothering me: model capability is visibly doubling — benchmarks, demos, one flex after another. But the felt experience of “letting AI deliver something end to end” has improved far more slowly.

Not a little slower. The moment you upgrade the task from “write a function” to “deliver a complete product,” what separates dozens of smooth steps from one crash is not model strength — it’s process.

Four crash scenes

Four real ones, from my own run logs.

① The ghost database. Everything green locally; after deploy, data simply wouldn’t write. Two hours of digging: the code used a relative path, ./data/db.sqlite. Change the working directory and it connects to a brand-new empty database — the schema was never created. The nastiest part: zero errors anywhere. SQLite happily lets you build a new database. Worse, code review can’t catch this: the logic is right; the running environment is wrong — and AI will never volunteer “I assumed this working directory.”

② The CSS that didn’t take for a whole evening. Asked AI to tweak the homepage visuals. Refresh — no change. Again — no change. Three rounds in: the project had two style files, and a same-named selector in the later one silently overrode the earlier one. The file being edited was never applied by the browser.

③ Restart, database deleted. A small tool that had been running for two weeks; after a server restart, all data was gone. In the startup script, tucked in a corner: rm -f *.db — a line written in the earliest days for a “clean start,” never removed. Two weeks of user data. No backup.

④ The context cliff. At step 8, upstream artifacts passed 15KB and everything got dumped into the next model call. No error — but output quality fell off a cliff: missing fields, invented interface names, “this should be fine” creeping in. You can’t point at what’s wrong; it just went fuzzy. Same task, 3KB of context versus 15KB — two different qualities of work.

See the pattern? Not one failure is “the model can’t write code.” They’re all classic software engineering topics: path management, build artifacts, lifecycle, context budget.

Stronger models won’t fix this

People say: wait for stronger models, bigger context windows, and these problems disappear.

My conclusion: no. The reason is simple — none of the four scenes above is an intelligence problem:

• The ghost database is an engineering state problem (paths), not a knowledge problem;
• The CSS override is a build artifact problem, not an aesthetics problem;
• The deleted database is a lifecycle problem, not a code problem;
• The context cliff is an information management problem, not a comprehension problem.

Stronger models raise the baseline of average output quality. But delivery succeeds or fails on the long tail and its silence — one bad step in a hundred, and that step raises no error. Process isn’t about the average; it’s about the boundary: making sure the worst case gets stopped at the door.

A context window growing from 128K to 1M doesn’t solve it either — stuffing more in just delays the fuzziness. Institutional problems are solved by institutions.

Why do teams deliver? Not because they’re smarter

A mature engineering team receiving a requirement runs: requirements review → design → development → testing → code review → release checks → incident retrospective. Every handoff has deliverables and release criteria.

What makes a team’s delivery stable was never “human engineers are smarter” — human engineers commit those same four mistakes daily. Stability comes from process: errors get intercepted before they leave the workstation; “it broke somehow” becomes “here is the gate that stopped it.”

And most people using AI strip all of it out: one omniscient brain, from requirement straight to production, not one gate in between.

The model is the brain. What you’re missing is the skeleton.

Compile 14 roles and a team’s process into one pipeline

This is what I’ve been building (and recently open-sourced as OneMaker Pipeline): 14 roles — requirements analysis, product design, visual creative, backend, frontend, testing + security, ops, AI integration, CTO review, integration verification, plus on-demand operations, global audit, and copywriting — strung into a stateful delivery pipeline: requirements enter at the first role; an MVP exits at the last.

The point isn’t “many roles.” It’s the four mechanisms at every handoff:

① No pass, no route. Every artifact clears three gates: claimed files actually exist and are non-empty; Bash verification must carry an EXIT:0 fingerprint (an agent can fake words; it can’t fake exit codes); client-facing output passes a desensitization check. Any failure, and the artifact does not go downstream. The fingerprint protocol has a side effect: it forces every role to leave evidence — you can’t say “I verified it” anymore, because verification must leave a trace.

② Targeted rollback, not full reruns. When review finds a problem, it rolls back by type: code bugs to development, architecture to product design, security to the security role. Not the whole pipeline again — you don’t know whether a full rerun will break the parts that already passed.

③ Three-tier memory. A pipeline runs for hours, across days, across sessions. Memory has three layers: a lightweight index (4KB read at startup), full project memory (loaded only when selected), and archive streams. Every layer has a hard cap; over the cap it auto-prunes — snapshotting before pruning. Memory is an asset, not a log.

④ Context budget. Upstream artifacts past 12KB switch downstream to “summary-first, deep-read on demand”; past 20KB, summary only. Built exactly to stop scene ④. (The thresholds aren’t vibes: 12KB is where quality measurably starts drifting; 20KB is where only summaries work at all.)

Four design decisions, and why

Decision one: a role saying “I’m done” is worth nothing.

Every self-assessment is treated as untrusted input. The only standard: has it been externally verified? Early on I let models self-score completion — 95%. Opened the directory: empty files. Self-assessment is where hallucinations hide best.

Decision two: rollbacks must be targeted, because failure costs are asymmetric.

A full rerun costs more than time — it re-gambles the seven stages that already passed. Targeted rollback: only the failing stage redoes work; the blast radius of the fix stays bounded.

Decision three: put ceilings on AI, don’t cheer it on.

Memory has 4KB/8KB/10KB hard caps; the pipeline has a 5-round iteration cap, after which it must enter attribution instead of retrying forever. Five consecutive failures at the same spot means “try again” was never the answer. Undamped auto-retry is just another way of giving up.

Decision four: plan first, and gate the memory.

Every project starts with a plan (goal, acceptance criteria, scenario type) — “thinking it through” is itself a quality gate. Before writing memory, four questions: will the next session need it? Can it be derived from what’s already there? Has it been written already? Will it exceed the cap? What gets written is “what the next session needs to know,” not “what happened.”

It has actually shipped things

Not slides. The pipeline has completed multiple real deliveries; one public output is AgentGate — reliability gates for multi-agent pipelines (127 automated tests, CI green).

One dataset stuck with me: running the full pipeline with a weaker model, the “code generation” stage failed to meet the bar five times in a row — every attempt was stopped by a gate; not one slipped through. After the fifth, diagnostics fired automatically and nailed it as model variance. After fixing the flow, the pipeline ran 8 stages end-to-end (with 3 targeted rollbacks).

Without the gates, at least 2 of those 5 failures would have disguised themselves as success and buried a bomb downstream.

The pipeline itself is at v4.x — behind every version number is a mechanism added after a crash: 4.2 added the fingerprint protocol, 4.3 added context budget and global audit, 4.4 added targeted rollback and memory anti-bloat. The list of mechanisms is the list of accidents.

You might ask

Q: What scenarios is this for? A: Solo engineering, small teams — anywhere delivery quality must be stable: products, client work, open source. One-off scripts don’t need it; don’t use a cleaver on a chicken.

Q: Why not a framework or SDK? A: Because it’s fundamentally a set of protocols and discipline (role definitions + gates + memory rules), not a runtime. It sits on top of Claude Code or any model — I use it as my working system every day. Frameworks age; discipline doesn’t.

Q: Why open source? A: Because this discipline is worth more used than hoarded. As I said — the mechanism list is the accident list. Open-sourcing it hands over my potholes so the next person steps in fewer.

Honest limitations

• Designed for solo and small-team scenarios — within one person’s compute and attention budget — not distributed multi-machine orchestration;
• Memory is file-based — simple, auditable, git-diffable — but it’s not a database, and not for high concurrency;
• The pipeline deliberately slows down: interception, audit, rollback all cost time. If your bar is “it runs, ship it,” it will feel annoying;
• It can’t replace your judgment. It guarantees “what reaches you has been verified” — not that the direction was right.

If you want the discipline too

Don’t start with 14 roles. A suggested path:

1. Add three gates first: file existence, EXIT_CODE fingerprint, deliverable desensitization. Just those three stop most silent failures;
2. Write your release criteria: what counts as done? Make it a machine-checkable sentence. “Feature complete” doesn’t count; curl /health returning 200 counts;
3. Give it memory and ceilings: progress must survive sessions; retries must have a ceiling.

One honest word to end: this isn’t fundamentally technology. It’s discipline. The tech — file checks, exit codes, snapshots — takes half a day to learn. The hard part is making every single artifact pass the gates, including when you’re in a hurry. The pipeline’s value is exactly this: it walled off the shortcuts you’d want to take when rushed.

MIT licensed. Three zero-dependency demos in the repo, five minutes:

git clone https://github.com/zw-onemaker-ai/onemaker-pipeline
cd onemaker-pipeline && python3 examples/01_quality_gate/demo.py

Architecture diagrams, the full rulebook, and all 14 role definitions are in the repo. Issues welcome.