Add AI to a feature only if you can answer four questions with specifics. What user job is failing today? How costly is a wrong answer? Would a simpler fix work? Can you pay to run and maintain the feature after launch? If any answer is vague, don't build yet. The vague answer is where the project will stall.

Most teams get pushed into AI from the top: a board question, a competitor's launch, a roadmap slot labelled 'AI'. The audit below turns that pressure into a decision you can defend. It takes days, not months, and it usually ends in one of three places: build it, build something simpler, or drop it.

Question 1: What job is failing, and how will you measure the fix?

Start from a user task that is slow, error-prone or abandoned, not from the technology. If you can't point to the task and the number that shows it is broken, you have no way to know whether AI improved anything.

Good answers are concrete. Support agents spend most of each ticket hunting for the right policy document. Users drop out of onboarding at the step where they have to categorise their own data. Analysts rewrite the same weekly summary report every week. Each of these has a baseline you can measure now, such as handling time, drop-off rate or hours per report, and a target you can check after launch.

Weak answers sound like 'make the product smarter' or 'add a chat assistant'. They describe a feature, not a problem. Write the answer down as one sentence: 'For [user], [task] currently takes or fails at [measured baseline], and we will call this a success if it reaches [target].' If the team can't agree on that sentence, stop here.

Question 2: How wrong can the output be before it causes harm?

Every AI feature will sometimes produce wrong, incomplete or made-up output, so you need to know the cost of a mistake before you design anything else. Error tolerance decides whether the feature can run on its own, needs a human to approve it, or shouldn't exist.

A practical way to sort tasks:

  • Low cost of error: drafting text the user will edit, suggesting tags, ranking search results. A wrong answer wastes a few seconds and the user fixes it. These suit AI well.
  • Medium cost of error: summarising customer conversations, routing tickets, extracting fields from documents. Mistakes cause rework or delays. These need confidence thresholds, easy correction and spot checks.
  • High cost of error: anything touching money movement, medical or legal advice, access control, or irreversible actions. A wrong answer causes real harm. These need a human to approve every output, a narrow scope, or a decision not to use AI at all.

Also ask whether users can tell when the output is wrong. A wrong suggestion is cheap when it's obvious. When it looks plausible and nobody checks it, it gets expensive. If users can't verify the output easily, plan for citations, links to source records or a review step, and count the cost of that step in the business case.

Question 3: Would something simpler solve it?

Before committing to a model, check whether rules, better search, a cleaner interface or a fixed workflow would get most of the result. Simple fixes are cheaper to build, predictable and easier to debug. When they fall short, they still give you a baseline to compare the AI version against.

Common cases where AI turns out to be unnecessary:

  • The 'unstructured' input is actually a handful of known formats that a parser or a few rules can handle.
  • Users can't find information because of poor search indexing or navigation, not because they need a conversational interface.
  • The task is confusing because the form or flow is badly designed, and a redesign removes the need for assistance.
  • The 'personalisation' could be done with a few user settings and sensible defaults.

AI earns its place when the input is varied, written in natural language or too large for people to read, and when a rough answer is useful. Classifying free-text feedback, drafting replies from a knowledge base and pulling fields out of inconsistent documents are typical good fits. If a simple version gets you most of the way, ship it first and add AI only for the cases it can't handle.

Question 4: Can you afford to run and maintain it?

Building the feature is only the start of its cost. Plan for inference spend that grows with usage, the work of evaluating output quality, monitoring after launch and regular changes as models and providers change.

The main cost drivers to estimate before you commit:

  • Inference: cost per request multiplied by expected volume. Long prompts, large retrieved context and multi-step agent flows multiply this quickly.
  • Data: whether the data the feature needs is clean, accessible and permitted for this use. Cleaning and connecting data is often the longest part of the project.
  • Evaluation: a test set of real examples with known good answers, and someone responsible for running it whenever prompts or models change.
  • Latency: whether users will wait for a response, or whether you need streaming, caching or a smaller model.
  • Privacy and compliance: where data is sent, whether providers keep it, and what your contracts and regulators allow.
  • Upkeep: providers retire and update models, which can change behaviour without warning. Someone has to own regression testing.

If the projected running cost is more than the value of the improvement you defined in Question 1, the feature doesn't pay for itself, however good the demo looks.

How to run the audit and what it costs

Run the audit as a short, time-boxed exercise with product, engineering and someone who owns the data. For a single feature, a few working sessions over one or two weeks is usually enough. The cost is mostly staff time, plus a little model spend if you test prompts.

A workable sequence:

  • Write the one-sentence problem statement and collect the baseline metric.
  • Gather a few dozen real examples of the input, including messy and edge cases.
  • Build the simplest possible version without AI and measure it against those examples.
  • Prototype the AI version against the same examples, and record accuracy, failure types, latency and cost per request.
  • Decide: ship the simple version, ship AI with the right level of human review, or stop.

The prototype step matters most. A throwaway prompt against real data reveals in days what a planning document would miss: whether the model handles your edge cases, how often it makes things up and what each call really costs. If you bring in an outside engineering team, ask them to deliver this evaluation before quoting a full build.

What usually goes wrong

Most failed AI features fail for predictable reasons, and the audit is built to catch them. Teams demo on hand-picked examples, never set a success metric and find out about running costs after launch.

Other common problems: no one owns the evaluation set, so quality drifts unnoticed. The feature goes out without a fallback, so a provider outage breaks a core flow. The interface gives users no way to correct or report bad output, so the team never finds out what is failing. Each one is cheap to prevent before the build and expensive to fix after it.

A 'no' or 'not yet' from this audit is a good result. It frees roadmap time for work that pays off and gives you a documented reason when someone asks why the product doesn't have an AI feature yet. If the answer is 'yes', you start the build with a metric, an error budget, a baseline and a cost model, which is what you need to scope it properly.

Questions people ask

How do I know if my product needs AI?

Your product needs AI when a specific user task involves varied or natural-language input, a rough answer is still useful and simpler fixes have been tried and fall short. If you can't name the task and a metric that shows it's broken, you aren't ready to decide.

What does it cost to add an AI feature to an existing product?

Cost depends mainly on usage volume, prompt and context size, data preparation, how much human review is needed and the ongoing evaluation and maintenance work. Running costs keep going after launch, so estimate cost per request at the volume you expect before you commit.

How long does it take to validate an AI feature idea?

For one feature, a time-boxed audit and prototype against real data usually takes one to two weeks. It is enough to measure accuracy, failure types, latency and cost per request before you start a full build.
← All insights