Building an AI agent that works in a demo takes an afternoon. Building one that works reliably on real inputs, inside real systems, takes engineering. This guide walks through the steps we follow, in order, and why each one matters.
1. Choose the task
Start with one task, not a general assistant. Good first tasks are frequent, follow a recognisable pattern, vary enough that fixed rules fail, and have outcomes someone can check. Write down what the agent must achieve, what it must never do, and how you will know it succeeded.
2. Build the evaluation set first
Collect real past examples of the task — tickets, requests, documents — with the correct outcome for each, agreed with the people who own the work. Include the awkward cases that caused problems before. This set is how you will judge every decision that follows: which model, which prompt, which tools. Without it, improvements are guesses.
3. Design the tools
Tools are the actions the agent can take. Resist exposing every API endpoint. Design a small set of tools around the task, each with a clear name and description, typed inputs and results the model can understand — including errors it can recover from. Give each tool the narrowest permission that works and keep read and write tools separate.
If the tools wrap existing systems, the Model Context Protocol (MCP) is now the standard way to expose them, with authorization per user and per tool.
4. Choose the model
Try several models on the evaluation set. Larger models reason better; smaller ones are faster and cheaper. Many agents use a capable model for planning and a smaller one for simple steps. Keep the provider behind an interface so you can change it later.
5. Manage state and failures
Real runs fail halfway: an API times out, a rate limit hits, a process restarts. Store the agent's progress outside the model so a run can resume, and make every action idempotent so a retry does not send the same email or create the same refund twice. A workflow engine handles much of this for you.
6. Put people in the loop
Decide which actions need human approval — anything that moves money, changes access or contacts customers is a sensible default. Show the reviewer what the agent intends to do and why. Start with more approval than you think you need and reduce it as accuracy is proven.
7. Add observability and limits
Record a trace of every run: each model call, each tool call, inputs, outputs, timing and tokens. Set a maximum number of steps and a cost ceiling per run and per day. When something goes wrong in production, the trace is how you find out why — and each surprise becomes a new test case.
8. Roll out in stages
Run the agent in shadow mode first, where it proposes actions but a person does the work. Then let it act on a small share of cases with review, then widen. Gate each step on the evaluation score and on what the production traces show.
What it costs
Most of the build effort goes into integration, evaluation and permissions rather than prompts. A single well-scoped workflow inside an existing product is typically weeks of work to a first production release, and running cost can be measured precisely during evaluation before you launch.
Questions people ask
Do we need a framework to build an agent?
- Not necessarily. A simple loop with good tools works for many tasks. Frameworks and workflow engines help with state, retries and multi-step orchestration as complexity grows.
How do we stop an agent going wrong?
- Narrow permissions, human approval for consequential actions, step and cost limits, and an evaluation set that catches regressions before release.
How long until an agent is production-ready?
- For one well-defined workflow with existing APIs, typically a few weeks to a first staged release; longer if the systems it needs to use must be built or cleaned up first.