LLM Evals and LLMOps: A Practical Guide

Most teams ship AI features on vibes. Someone tries fifteen prompts by hand, the outputs look good, and the feature goes live. Then quality drifts, a model gets upgraded, a prompt gets tweaked, and nobody can say whether things got better or worse, because there was never a measurement in the first place.

Hamel Husain, who advises teams building LLM products, puts the root cause bluntly: the failed AI products he sees almost all share the absence of a robust evaluation system [1]. The pattern is consistent enough that “do you have evals?” is the single most diagnostic question you can ask about an AI feature.

The encouraging part is that none of this is new. Reading outputs and categorizing failures is exploratory data analysis. Validating an LLM judge against human labels is model evaluation. Building representative test sets from production data is experimental design. Monitoring whether a system works in production is production ML [2]. The discipline exists; most teams building LLM features have just not applied it yet.

This guide covers the evaluation and operations practice end to end: error analysis, test sets, graders, judges, CI, tracing, and production monitoring. It is the hub for our series on evals and LLMOps, and it builds on our series on adding AI features to an existing product.

Why generic metrics fail

The most common first move is also the least useful: install an eval framework, switch on its built-in metrics, and put helpfulness, coherence, and hallucination scores on a dashboard. Those numbers sound reasonable and diagnose nothing, because they are generic by construction and your failures are specific [2].

The metrics that change decisions are application-specific and name a concrete failure, things like “failed to escalate to a human” or “scheduled the meeting in the wrong timezone” [2]. You cannot get those from a library. You only get them from looking at what your system actually does.

Two more habits to drop early. Off-the-shelf similarity metrics rarely fit LLM outputs, because a correct answer phrased differently scores badly and a fluent wrong answer scores well [2]. And Likert scales hide the decision you need to make. A 1-to-5 helpfulness score gives you a number that moves without telling you what to fix; binary pass or fail on a scoped, specific criterion forces the ambiguity to be resolved up front [2].

Step 1: Error analysis, the highest-return activity

Before designing any metric, read your system’s actual outputs. Take a sample of real traces, read them end to end, write a short note on each failure, then group those notes into categories and count them. That is it. The process is unglamorous and routinely skipped, and it is the highest-return work available to a team building an AI feature [2].

A workable first pass: sample roughly 100 traces spanning your common input types and known edge cases, read them with the input, retrieved context, and output all visible together, and tag each failure with a short description in your own words. Then cluster the descriptions into failure categories and count how often each occurs. The categories that appear most often are your first metrics. The rest can wait.

A few practical notes. Build or buy a trace viewer that shows the whole interaction in one place; if annotators have to read raw JSON, they will annotate less and worse [2]. Get domain experts and product managers in front of raw outputs rather than summary scores [2]. And expect the criteria to change as you look, which is not sloppiness but a documented phenomenon: researchers studying LLM evaluation named it criteria drift, the observation that people need criteria to grade outputs but need to grade outputs before they can define their criteria [3]. Plan for a few passes.

We go deeper in error analysis for LLM failures.

Step 2: Build test sets from real data

Once you know what breaks, build the dataset that measures it.

Start from production, not from a prompt. Asking a model for fifty test queries produces generic, unrepresentative data. The better sequence is to look at real traffic first, form hypotheses about which dimensions matter (input length, document type, language, user role, edge conditions), and only then generate synthetic examples that vary along those dimensions, grounded in real logs [2].

Size it for the decision, not for comfort. Fifty to two hundred cases per task is usually enough to start, weighted toward the failure modes error analysis surfaced rather than sampled uniformly from easy traffic. These are practitioner numbers, not a statistical rule; what matters is that the set is representative and that it grows every time production surfaces a new failure.

Keep splits. If you are going to tune prompts or judges against the data, hold out a test set you do not touch, or you will optimize your way into numbers that do not survive contact with production [2].

Version it. The dataset is a product artifact. It belongs in version control alongside the prompts it grades.

Step 3: Choose the cheapest grader that works

Three kinds of grader, in ascending order of cost and descending order of reliability.

The ordering matters. Anything a code assertion can check should be checked that way: it is free, deterministic, and runs in milliseconds. Reach for a judge only where the criterion is genuinely fuzzy, and treat human review as both the fallback and the calibration source for everything else.

One caveat on that last row. Human judgment is the reference, not an infallible one. Human evaluators have documented biases of their own, including a tendency to rate confident, assertive answers above accurate but hedged ones [4]. Write rubrics that tell annotators what to ignore as well as what to look for.

Step 4: Validate your judge like a classifier

Using an LLM to grade outputs is standard practice and frequently done without any check on whether the grader is right. Ask most teams how they know their judge is trustworthy and the answer is silence [2].

The fix is to treat the judge as what it is: a classifier producing predictions you need to trust. Collect human labels on a sample, split into train, dev, and test, hill-climb the judge prompt against dev, and confirm on the held-out test set that you have not overfit [2]. Source few-shot examples for the judge from your labeled data rather than inventing them.

Report precision and recall, not accuracy. If a failure occurs in 5% of cases, a judge that never detects it still scores 95% accurate. Accuracy hides exactly the thing you are trying to measure [2].

The research literature supports both the promise and the caution. Judges can reach agreement with human raters comparable to agreement between humans themselves on some tasks [5], and individual studies report high agreement, along with systematic skews such as a judge being more stringent than human annotators on underspecified answers [6]. But the known biases are real and specific: position bias, where the order in which candidates appear changes the verdict, shown across 15 judges and roughly 150,000 evaluation instances to vary significantly by judge and task and to be strongly affected by the quality gap between candidates [7]; verbosity bias toward longer answers; and self-preference, where a judge favors text generated by its own model family [8]. Temperature matters too: same-verdict rates on repeat runs are high at temperature zero and degrade substantially as temperature rises [9].

Practical mitigations: run pairwise comparisons in both orders and discard inconsistent verdicts, hold temperature at zero, keep judge rubrics narrow and binary, prefer a judge from a different model family than the system under test, and re-validate the judge whenever you change it or the underlying feature.

More detail in validating an LLM judge.

Judges examined for position bias

15

The effect varied significantly by judge and by task.

Evaluation instances in that study

~150k

Bias was strongest where the two candidates were close in quality.

Accuracy of a judge that detects a 5%-rate failure never

95%

Which is why judges are reported on precision and recall instead.

Step 5: Evaluate components separately

For any feature with more than one step, evaluate the parts before the whole, or a failing end-to-end score tells you nothing about where to look.

For retrieval-augmented features, that means measuring retrieval and generation independently: whether the right documents were fetched, and separately whether the answer was faithful to whatever was fetched. A generation failure caused by bad retrieval and one caused by a bad prompt need completely different fixes. See RAG evaluation, our enterprise RAG guide, and reducing hallucinations in enterprise RAG.

For agents, add step-level checks: was the right tool chosen, were the arguments correct, did the loop terminate, did the final state match the goal. A task-level pass rate on a fifteen-step agent is nearly undebuggable on its own. See AI agents in the enterprise.

Step 6: Instrument before you evaluate

You cannot evaluate what you cannot see. Tracing comes first, and it should be in place before the first prototype leaves anyone’s laptop.

Capture, for every model call: the input, the retrieved context, the full output, model and prompt version, token counts by category, latency, cost, and the outcome. For multi-step features, capture the whole tree, so a single trace shows the orchestration span, each model call, and each tool invocation beneath it.

Use a standard schema rather than inventing attribute names. OpenTelemetry’s GenAI semantic conventions define a shared gen_ai.* vocabulary covering the model, operation, token usage, finish reasons, and, when content recording is enabled, the messages themselves [10]. Major observability vendors have adopted it, so instrumenting once keeps your telemetry portable instead of locked to one backend [11]. Note that parts of the convention are still evolving, and instrumentations gate the newest version behind an opt-in environment variable, so pin the version you emit [12].

One boundary to understand: OpenTelemetry records what happened, not whether it was good. A span can tell you a call returned 1,200 tokens in 850 milliseconds; it cannot tell you those tokens contradicted the retrieved sources [13]. Telemetry is the data plane. Evaluation sits on top of it.

See LLM observability and tracing.

Step 7: Run evals in CI

Once the harness exists, wire it into the pipeline so that every change that could affect quality gets measured: prompt edits, model changes, retrieval settings, chunking strategy, tool definitions, and context assembly.

Practical shape:

• Fast tier on every pull request: code assertions and a small judge subset, seconds to a couple of minutes.
• Full tier nightly or before release: the complete dataset including judges.
• Report per-category pass rates, not one aggregate number, so a regression points at a failure mode.
• Set thresholds per criterion, and treat the threshold as a product decision rather than a testing convention. Unlike unit tests, the right bar is rarely 100% [1].
• Check cost and latency in the same run, so a prompt change that doubles tokens fails before it merges. See LLM API costs in production.

This is what makes a model migration routine. When a provider announces a deprecation or a cheaper model appears, you run the suite against the candidate and compare, rather than guessing. See choosing an LLM for your product.

More in regression testing AI features in CI.

Step 8: Keep evaluating in production

Pre-launch evals tell you the feature worked on the cases you thought of. Production tells you the rest.

Sample and review real traces on a regular cadence, weekly to start. Watch the behavioral signals your interface already produces, since accept, edit, discard, and regenerate are richer than any thumbs-up widget and map directly onto new eval cases. See AI copilot UX patterns. Run a subset of judges on live traffic to get a continuous quality signal alongside your operational metrics. Track schema failures, guardrail rejections, and refusals as first-class metrics. And watch for drift in inputs as much as outputs: new customer segments, new document formats, and new languages change the distribution your evals were built on.

Every genuine production failure should end up as a test case. That loop is what keeps the eval set representative instead of slowly becoming a museum of last year’s bugs.

See monitoring LLM quality in production.

What not to automate

LLMs can write your harness, generate boilerplate, and help cluster failure notes. They cannot look at the data for you, because the entire point of looking is to discover what you want, and you do not know that until you have seen the outputs [2][3].

The related trap is measuring everything. There are infinitely many things you could score. Form a hypothesis about what is actually broken, measure that, fix it, and repeat [2]. A dashboard with twenty metrics nobody acts on is not an evaluation system.

Common failure modes

• Generic metrics from a library, with no error analysis behind them.
• A judge nobody validated against human labels.
• Accuracy reported for rare failure modes, hiding them.
• Synthetic test data generated from a prompt rather than grounded in production traffic.
• Likert scales standing in for decisions nobody wanted to make.
• End-to-end scores only, with no component-level evaluation.
• Evals run once before launch and never again.
• Annotators reading raw JSON, so nobody annotates carefully.
• No tracing, so production failures cannot be reconstructed.
• Scores reported with no sense of variance or confidence.

The maturity ladder

Most teams sit at 0 or 1 and think they are at 3. Getting to 2 is the step that changes outcomes, and it costs a few days of reading traces.

Keep reading

This guide is the hub for our series on evaluating and operating AI features:

• Error analysis for LLM failures
• Validating an LLM judge
• RAG evaluation: retrieval and generation
• LLM observability and tracing
• Regression testing AI features in CI
• Monitoring LLM quality in production

Related: how to add AI features to an existing software product, making LLM features reliable, and build in-house, hire, or partner.

CMW AI Labs builds AI features with evaluation harnesses from the first prototype: error analysis on real data, validated judges, component-level evals in CI, and production monitoring that feeds failures back into the dataset. If you have an AI feature in production and no way to tell whether it is working, talk to our team.

Sources

1. Hamel Husain, “Your AI Product Needs Evals.” https://hamel.dev/blog/posts/evals/
2. Hamel Husain, “The Revenge of the Data Scientist,” March 26, 2026. https://hamel.dev/blog/posts/revenge/
3. Shankar et al., “Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences,” UIST 2024, arXiv:2404.12272. https://arxiv.org/abs/2404.12272
4. Galileo, “LLM-as-a-Judge vs Human Evaluation,” February 2026. https://galileo.ai/blog/llm-as-a-judge-vs-human-evaluation
5. Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” arXiv:2306.05685. https://arxiv.org/abs/2306.05685
6. “Beyond Case Law: Evaluating Structure-Aware Retrieval and Safety in Statute-Centric Legal QA,” arXiv:2604.06173, 2026. https://arxiv.org/abs/2604.06173
7. Shi et al., “Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge,” arXiv:2406.07791. https://arxiv.org/abs/2406.07791
8. “Testing the Testers: Human-Driven Quality Assessment of Voice AI Testing Platforms,” arXiv:2511.04133, 2025. https://arxiv.org/abs/2511.04133
9. “Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models,” arXiv:2606.19544, 2026. https://arxiv.org/abs/2606.19544
10. OpenTelemetry, “Inside the LLM Call: GenAI Observability with OpenTelemetry,” May 2026. https://opentelemetry.io/blog/2026/genai-observability/
11. MLflow, “OpenTelemetry GenAI Semantic Conventions.” https://mlflow.org/docs/latest/genai/tracing/opentelemetry/genai-semconv/
12. OpenTelemetry, “Semantic Conventions for GenAI agent and framework spans.” https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-agent-spans/
13. Fiddler AI, “OpenTelemetry for AI Observability: What It Covers and Where It Stops,” July 2026. https://www.fiddler.ai/blog/opentelemetry-ai-observability-guide