01
New projects
A paragraph about the problem is enough to start. We reply within two working days, usually with questions rather than a quote.
AI · Service LLM evaluation and observability
Evaluation suites built from your real inputs, a trace for every run, drift alerts and cost ceilings, so an AI feature is measured rather than assumed.
In short
LLM evaluation is the practice of testing an AI feature against a fixed set of real examples and scoring the results, so you know whether a change made it better or worse. Observability is seeing what each production run actually did. We build both: a regression suite that runs in CI, and traces, cost tracking and alerts for the live system.
- How to engage
- Product build, Modernization audit
- Practice
- AI
Signs this is the right service
- Nobody can say whether last week's prompt change improved things or broke something.
- The feature is live and the first sign of a problem is a customer complaint.
- Model costs are rising and it is unclear which feature or customer is responsible.
- You want to change model or provider and need to know what you would lose.
What you get
Evaluation set
Cases drawn from your real historical inputs, with the expected behaviour written by the people who own the outcome.
Scoring that fits the task
Exact checks where there is a right answer, rubric-based grading where there is not, and human review for a sample.
CI integration
The suite runs on every pull request and blocks a release that scores below the agreed threshold.
Production traces
Each run recorded end to end — prompts, tool calls, latencies, tokens — and searchable when something goes wrong.
Cost and drift alerts
Spend per feature and per customer, with alerts when cost per run or answer quality moves outside its normal range.
The process
- 01
Define good
We agree what a correct, acceptable and unacceptable result looks like, in writing, with examples.
- 02
Collect the cases
Real inputs, including the awkward ones that caused problems before.
- 03
Automate the scoring
Graders are themselves checked against human judgement before they are trusted.
- 04
Wire it into release and production
The same definitions drive the CI gate and the production alerts.
Eval harnessOpenTelemetryTracesCIPostgresTypeScriptPython
LLM evaluation & observability: common questions
Are public benchmarks enough?
- No. A benchmark ranks models in general. It does not tell you whether your prompt change broke the twelve request types your team cares about. That takes a test set made from your own inputs.
How many test cases do we need?
- Start with the cases that matter most — often fifty to a few hundred — and add one every time production surprises you. A small set that runs on every change beats a large one that runs once.
Can a model grade another model's output?
- Yes, for rubric-style judgements, provided the grader is checked against human scores first and re-checked when it changes. For anything with a definite right answer we use exact checks instead.
Does this work with our existing monitoring?
- Yes. Traces are emitted in OpenTelemetry format, so they can go to the observability tools you already run.
06 / Let’s build
Bring us the difficult problem.
02
Careers
Engineers, designers and researchers who like systems that are hard to get right. Send something you built and what you would change about it now.
03
Studio
Based in India and working remotely, with overlap hours agreed up front so your team always has a time to reach us.
04
Read
Guides on AI agents, RAG, MCP, tokenization and chain abstraction, written by the engineers who build them.