---
title: "Evals: The AI Skill That Outlasts Every Model"
description: Evals are the AI skill that outlasts every model. Why evaluation belongs at every stage of the AI lifecycle, not just the end, and the five levels of maturity.
---

<https://www.clair-x.ai/insights>

# [Evals: The AI Skill That Outlasts Every Model](https://www.clair-x.ai/insights/evals-the-ai-skill-that-outlasts-every-model)

 Written by [Swapnil Thakre, ClairX AI Team](https://www.clair-x.ai/insights/author/swapnil-thakre-clairx-ai-team) | Nov 5, 2026, 12:23:27 PM

*Why the ability to judge AI quality is the one capability that survives every model change, and why it belongs at every stage of the AI lifecycle, not at the end*

## Evals: The AI Skill That Outlasts Every Model

 

We have sat in a lot of AI steering meetings, and there is a question we have learnt to ask early. Not "what does the model do", but "how do you know it is doing it well". The first question gets a demo. The second question, far more often than it should, gets a pause.

Most teams can now watch their AI work. Tracing is in place, dashboards exist, every call is logged. Far fewer can tell you whether the work is any good, in terms a customer, a regulator or a finance director would accept.

That gap is what this post is about, and our position on it is simple: **the ability to evaluate AI systems ("evals") is now one of the most important capabilities an AI organisation can build, and it has to run through the whole AI workflow, from problem definition to audit.** Bolt evals on as a QA step at the end and you will hear about your quality problems from your customers, your regulator or a tribunal.

## Why this matters now

 

If you lead AI, evals answer the questions you get asked every quarter. Is this ready to ship? Is the new model better than the old one? Did last week's prompt change break anything? Can we show the auditor how we tested it?

Without evals, the honest answer to all four is "we think so." That is an expensive answer. We have watched agentic projects lose their sponsor not because the agent was bad, but because nobody could prove it was good. Usually that is because nobody ever wrote down what good meant, or a committee wrote it down in a way nobody could fail. Showing value, controlling risk and justifying spend all depend on being able to put a number on the quality of the system in front of you. Take the measurement away and every one of those conversations becomes a matter of opinion, and opinions do not survive a budget review.

## Public benchmarks are only a first screen

 

Plenty of enterprises still pick models off a leaderboard. We understand why. A public rank feels like independent evidence, and it saves an argument. In our experience it is a weak basis for a decision, and we expect it to get weaker still.

Benchmarks saturate. Once every serious model scores near the top of a test, the test can no longer tell you which one is better for you. We have seen a benchmark go from "hard" to "solved" inside a year, at which point it measures the age of the model rather than its usefulness.

They also leak and they can be gamed. Test material finds its way into training data. Providers test many variants privately and publish the best one. None of this is scandalous; it is what happens to any public yardstick that money depends on. It does mean a leaderboard position tells you very little about how a model will behave on your contracts, your customers and your edge cases, which are the only cases that matter.

What this means for you: use benchmarks as a first screen to shortlist, and make the actual decision with application-specific evals built on your own data.

## Evals belong in every stage of the workflow

 

The mistake we see most often is treating evaluation as a gate between "built" and "shipped". In practice the same eval assets do four jobs at once. They are the product specification, the regression-test suite, the production monitoring layer and the audit record. Evaluate early and often, log everything, and mine the logs for new eval cases. Teams that do this find the eval set becomes the most valuable artefact the programme owns.

Here is the loop we use for where evals sit across the AI lifecycle.

 

 

***Figure 1.*** *Evals run through every stage of the AI workflow. Production failures feed back into the eval set, and sometimes into the definition of 'good' itself. The same assets become your governance record.*

### Here is how it works, step by step.

1. Define what good means. Before anyone writes a prompt, turn "good" into pass/fail criteria tied to a business outcome. Give that definition to one domain expert to own, rather than a committee.
2. Build an eval set from reality. Start with twenty to fifty tasks drawn from real failures: complaints, escalations, the outputs someone had to fix by hand. Thousands of test cases can wait. What you need to begin is the right fifty, and they already exist in your logs.
3. Choose the model on your tasks. Compare candidate models on your eval set, looking at quality, cost and latency together. This is what replaces the leaderboard.
4. Build against the evals. Write the eval first, then iterate until it passes. It is the same discipline as test-driven development, and it works for the same reasons.
5. Gate every release. We keep two kinds of eval apart. *Capability evals* start with a low pass rate and measure progress. *Regression evals* should stay near 100%, so that a model upgrade or a prompt tweak cannot quietly break what already works. For agents, run each task several times, because an agent that succeeds six times in ten is not a success. If your agent faces customers, consistency is the number that matters, and it is the number most teams never measure.
6. Score live traffic. Offline tests catch the failures you saw coming. Online evals, which score a sample of real production traffic, catch the ones you did not. This is the step most teams have not reached, and it is where the real surprises live.
7. Review failures and refresh the evals. Someone reads real transcripts, names the new failure types and adds them to the eval set (the solid arrow back to step 2). Sometimes what they learn changes the definition of good itself, and step 1 gets revisited (the dashed arrow). Eval sets go stale. This step keeps them honest.

If you answer to a risk committee, the side arrow to governance is the one to watch. Regulation across the UK and Europe is turning "we tested it" into a documentation duty: risk management, accuracy and robustness testing, and technical documentation that shows the testing happened. The deadlines for high-risk systems have already moved once. The obligations have not gone away, and they will arrive before most eval programmes are ready for them.

Our view is that teams who run the loop already hold the evidence a regulator will ask for, as a by-product of doing the work well. Teams that do not will be reconstructing it under deadline, from logs that were never designed to be read.

The failures are already on the record and they are not exotic. A chatbot gives a customer wrong advice and the company is held to it, because the chatbot is not a separate legal entity, whatever the terms of service say. A coding agent with production access deletes a database during a code freeze. We cannot know which specific eval would have caught either one. Both, though, are exactly the kind of failure that evals exist to catch: a check that an answer is grounded in your own policies and documents, and a test that an agent did only what it was asked to do and nothing more. Both are also the kind of story that ends an AI programme’s budget.

## The core skill is judgement

 

If evals matter this much, why can't you just buy them? Because the hard part is judgement, and judgement comes from your own people. The tooling is maturing fast and the platforms are good. A platform will show you traces. Deciding what "good" means for your claims-handling agent is still your call, and no vendor can make it for you. The core skill is **error analysis**. You read real outputs, write down what went wrong, group those notes into a taxonomy of failures, and count which ones matter most. It is unglamorous. It is also where every useful eval we have ever built came from.

That is why evals are a leadership concern as well as an engineering one. The people best placed to judge quality usually sit outside the ML team, as domain experts in underwriting, compliance or customer operations. Building the skill means getting those experts into the loop and protecting their time to stay there.

In practice, the toolkit forms a pyramid.

***Figure 2.*** *Run the deterministic checks on everything, calibrated LLM judges at scale, and expert review where stakes are highest.*

At the base are cheap, deterministic checks. Is the output valid JSON? Did the agent actually create the refund record in the database? These are fast and objective, and brittle when there are many valid answers.

The middle layer is **LLM-as-judge** evaluators, where one model grades another's output. This is what lets evaluation scale, and it comes with a condition we insist on. A judge model agrees with human raters often enough to be useful, and it carries biases of its own: it tends to prefer longer answers, answers in certain positions and outputs that resemble its own. So every judge has to be checked against human labels, with its true-positive and true-negative rates reported. A judge nobody has validated is just an opinion that happens to scale.

At the top sit expert human review and live A/B tests. Expensive, slow, and the final word.

## From vibes to governed: the five levels of eval maturity

 

To make this usable for planning, we use a five-level maturity model.

***Figure 3****. A leadership view of eval maturity, from ad-hoc spot checks to evidence a regulator would accept.*

Most teams we meet sit somewhere around L1 or L2: tracing is in place, a small fixed set of test cases exists, and releases still go out on the strength of a demo. The jump that matters most is from L1 to L2. That is where evaluation stops being a checklist and starts driving decisions, where a regression can actually block a release. For anything customer-facing or agentic, we treat L2 as the minimum, and we would rather see a team reach L2 on one important use case than L1 on ten.

## Where the discipline is still maturing

 

Evals are still an unsolved discipline, and we would rather be clear about the limits.

- There is no hard ROI evidence yet. Nobody has run a controlled trial linking eval maturity to project success. The argument in this post rests on what we have seen in practice and on the mechanism, and we think that is enough to act on. It is not proof.
- LLM judges can be circular. A model grading a model inherits that model's blind spots. People's grading criteria also drift as they see more outputs, so plan to keep reworking your judge rather than building it once.
- Generic metrics create false confidence. Off-the-shelf "hallucination" or "helpfulness" scores are easy to switch on and easy to misread. Without error analysis they measure something, and it may not be the thing that hurts you.
- Some teams ship well on "vibes". That works when the builders are expert daily users of their own product. It works much less well for regulated, high-volume or agentic products, where the person who notices the failure is a customer.
- Safety evals face a newer problem. The most capable models are increasingly able to recognise when they are being tested, which makes some test results less trustworthy than they look. That is mostly a problem for the labs that build models. It is still a reminder that even the best evals are imperfect evidence.
- Evals cost real time. Domain experts reading transcripts is slow, unglamorous work. Budget for it explicitly, or it will not happen.

## Turning evals into practice

1. Ask to see the eval instead of the demo. For your top two AI initiatives, ask three questions. What is the pass/fail definition of good? Who owns it? What is the current pass rate? If nobody can answer, you have found your first priority.
2. Name one quality owner per use case. Pick a single domain expert, rather than a committee, to own the definition of "good" and sign off on eval criteria.
3. Start with 20 to 50 real failures. Have the team pull recent bad outputs from logs or user complaints and turn them into the first eval set. Do this before you buy any tool.
4. Put a regression gate in the release process. No model swap or prompt change reaches production without a re-run of the eval suite. For agents, require multiple trials per task and report consistency rather than best-case success.
5. Audit your LLM judges. For every automated grader in use, ask for its agreement rate with human labels. If that has never been measured, treat its scores as unverified.
6. Map eval outputs to your risk register. Show your governance team which eval artefacts would answer a regulatory review, and close the gaps now, while the deadlines are still ahead of you.

Models will keep changing every few months. What carries over from one to the next is what your organisation knows about what "good" looks like for its own work, written down as evals. That knowledge is yours, and no vendor can take it with them.

Good evals start with good data: real logs, real failures and labels your domain experts trust. That is why we treat evaluation as a data and AI problem together. If you would like to work out where your organisation sits on the maturity model, or what it would take to reach L2 on your most important use case, talk to us. We build the eval loop alongside your team, on the platforms you already run, and your team keeps the eval assets and the know-how when we are done. We would enjoy that conversation.

\- Published on November 5th 2026.

 

If you would like to discuss AI Evals in detail or share how you are handling this, please do write us a note on [contact@clair-x.ai](mailto:contact@clair-x.ai)

[View full post](https://www.clair-x.ai/insights/evals-the-ai-skill-that-outlasts-every-model)

```json
{
  "@context" : "http://schema.org",
  "@type" : "BlogPosting",
  "author" : {
    "@type" : "Person",
    "name" : "Swapnil Thakre, ClairX AI Team"
  },
  "dateModified" : "2026-11-05T12:25:44.287Z",
  "datePublished" : "2026-11-05T12:23:27Z",
  "headline" : "Evals: The AI Skill That Outlasts Every Model",
  "image" : {
    "@type" : "ImageObject",
    "height" : 1260,
    "url" : "https://244786749.fs1.hubspotusercontent-na2.net/hubfs/244786749/AI%20Evals%20Main@2x-2.png",
    "width" : 2400
  },
  "mainEntityOfPage" : "https://www.clair-x.ai/insights/evals-the-ai-skill-that-outlasts-every-model",
  "publisher" : {
    "@type" : "Organization",
    "logo" : {
      "@type" : "ImageObject",
      "height" : 60,
      "url" : "/hs/hsstatic/content_shared_assets/static-1.4092/img/default-amp-logo.png",
      "width" : 60
    },
    "name" : "insights"
  }
}
```