The first agent we put in front of real users did exactly what we asked, most of the time. That was the problem. "Most of the time" is a fine property for a demo and a terrible one for a system that files reports, sends messages and touches records other people depend on.
For a few weeks we did what everyone does. We tuned the prompt. We added examples. We wrote longer and longer system messages full of capital letters. Each fix helped one case and quietly broke another, and we had no way of knowing which until a user told us.
The turning point was a boring realisation: an agent is a distributed system where one of the services is non-deterministic. We already know how to build distributed systems. We define contracts, we bound side effects, we observe everything and we test before we ship. So we started doing that.
Every tool is an API with an owner
In our first version, tools were thin wrappers around functions with a docstring for the model to read. Now every tool has a typed contract, a human owner and an explicit list of side effects.
export const createReferral = defineTool({
name: "create_referral",
owner: "clinical-platform",
input: z.object({
studentId: z.string().uuid(),
reason: z.enum(["fever", "injury", "mental_health", "other"]),
notes: z.string().max(600),
}),
sideEffects: ["writes:referrals", "notifies:clinician"],
requiresConfirmation: true,
budget: { maxCallsPerRun: 1 },
});
Three things happened once tools looked like this. Validation errors became messages we could feed back to the model instead of crashes. Reviews got easier, because a pull request that adds a side effect is visible in the diff. And the model got better at choosing tools, because narrow, well-named inputs are easier to reason about than a free-text parameter called query.
Separate planning from doing
We split the loop into a planner that proposes steps and an executor that runs them. The planner can be creative. The executor is strict: it checks each step against the tool contract, the run budget and the user's permissions before anything happens.
This sounds like ceremony, and for a single tool call it is. But most real tasks are three to eight steps, and the failure that hurts is rarely the first step. It is step five, after the agent has built up a confident but wrong picture of the world. Having a checkpoint between deciding and acting gives us somewhere to stop, ask the user, or roll back.
The most useful line of code in our agent runtime is the one that refuses to run a step.
Budgets are a feature
Every run has a budget for tokens, tool calls and wall-clock time. When an agent runs out, it has to summarise what it has done and hand control back. Before budgets, a confused agent would loop politely until it hit a provider limit. After budgets, a confused agent stops early and says so, which users trust far more.
Budgets also made costs predictable. Inference is not free, and for a startup serving schools and farmers, an unbounded loop is a pricing problem as well as a reliability one.
Evaluation is the real product work
None of this matters without evaluation. We keep a set of real, anonymised tasks with expected outcomes, and every change to a prompt, a model or a tool runs against it before it ships. Some checks are exact, such as "the referral was created with the right reason". Others use a grader model with a rubric, and we review a sample of those grades by hand every week, because graders drift too.
The rule we settled on is simple: no eval, no merge. It slowed us down for about two weeks and has saved us many times since.
What I would tell my past self
- Start with the contracts and the eval set, not the prompt.
- Make side effects explicit, reviewable and confirmable.
- Separate planning from execution so you have a place to say no.
- Give every run a budget and treat running out as a normal outcome.
- Log every step with enough context to replay it.
Agents are genuinely useful. They are also software, and software gets reliable the same way it always has: clear interfaces, small blast radius and tests that run before your users do.