Building Production AI Agents: Lessons from 50 Deployments

In the last 18 months I've helped ship or salvage roughly 50 AI agents — internal tools, customer bots, workflow automations, coding assistants. Maybe six shipped clean the first time. The other 44 all failed the same way: demoed beautifully on the founder's laptop, then broke inside two weeks of real traffic. Not because the model wasn't smart enough. Because nobody built the boring scaffolding around it. The AI agent discourse in 2026 is still 90% about which model is best. The actual bottleneck to shipping is 90% about everything except the model. Eval harnesses. Retry budgets. Tool schemas that don't rot. State machines that constrain what an LLM is allowed to do. Human review queues that don't melt throughput. The parts of the stack that look like ordinary distributed systems work — because that's what they are — decide whether your agent lasts a month. Here's the cheat sheet I wish someone had handed me the first time I put an agent behind a login. The eval harness is the product Every failed agent I've inspected had the same root cause: no eval. The founder tested it "by vibes" — running a dozen queries themselves, feeling good, shipping. Then a customer sends the 200th query, it does something insane, and the founder has no way to know if fixing prompt X will regress behavior Y. Every change is a coin flip. You need a written eval before you write the second prompt. It doesn't have to be fancy. A CSV of 40 real inputs, expected outputs (or behavior descriptions if outputs are open ended), and a scoring function that runs on every commit. Braintrust and Langfuse both do this well; PromptFoo if you want an OSS only stack. The tool doesn't matter — having 40 test cases matters. The…