Almost every software agency now lists AI among its services, and most can produce an impressive demo. The difference between a demo and a system you can rely on is engineering discipline, and a few questions reveal quickly whether a team has it.
How will you know it works?
The best answer involves an evaluation set: real examples from your business with agreed correct outcomes, scored automatically on every change. Be wary of answers that rely on "we tested it and it looked good" or on public benchmark scores, which say nothing about your use case.
How do you stop it doing something harmful?
For any AI that takes actions, ask how permissions work. Good answers mention narrow tool scopes, separating read and write access, human approval for high-impact actions, and audit logs. If the answer is mainly about the system prompt, the team has not dealt with prompt injection.
What will it cost to run?
AI features have running costs that scale with use. A capable team will measure cost per request during development, tell you the expected monthly cost at your volume, and put limits in place so a bug cannot run up an unexpected bill.
Where does our data go?
- Which AI providers will receive your data, and under what terms?
- Is your data excluded from model training?
- Where is it processed and stored, and for how long?
- Can it stay within your own cloud account if needed?
Who owns the result?
You should own the code, prompts, evaluation sets and infrastructure, and be able to switch AI providers without rebuilding. Ask whether models are behind an interface and whether anything depends on the agency's own proprietary platform.
Can they show production work?
Ask about AI systems the team has put in front of real users — what went wrong after launch and how they found out. Teams that have operated AI in production talk about monitoring, regressions and cost surprises; teams that have only built demos talk about model capabilities.
A short checklist
- An evaluation set from your data, run on every change
- Narrow tool permissions and human approval for consequential actions
- Measured running costs and spending limits
- Clear answers on data handling and training
- You own the code, prompts and evaluations
- Experience operating AI in production, not just demos
Questions people ask
Should we hire an AI agency or build in-house?
- An agency can move faster on a first project and transfer practices to your team; in-house makes sense once AI is core and ongoing. Many companies do both — an agency builds the first system with their engineers alongside.
How much does an AI project cost?
- It depends mostly on integration and evaluation work rather than the model. A focused first use case is typically weeks of engineering; we publish starting prices on our pricing page.
What should be in an AI project contract?
- Ownership of code and evaluation data, data-handling terms, the success measures for the project, and running-cost estimates.