RAG Evaluation: Measuring Retrieval and Generation Separately

A retrieval-augmented feature gives a wrong answer. Was the right document never retrieved? Was it retrieved and ignored? Was it retrieved, used correctly, and simply not an answer to the question the user asked?

Those are three different bugs with three different fixes, and a single end-to-end quality score cannot tell them apart. This is the central discipline of RAG evaluation: measure the components separately, so that a failing number points at a cause instead of a mood.

This guide covers how to do that: information retrieval metrics for the retrieval half, a structural framework for the generation half, how to build test sets without hand-annotating a corpus, and a diagnosis table mapping symptoms to fixes. It is part of our series on evals and LLMOps, and it assumes the architecture covered in our enterprise RAG guide.

Start with retrieval, because generation inherits its failures

If the retriever surfaces the wrong passages, the generator is beaten before it starts. Teams routinely invert this, spending weeks on prompt engineering for a generation step that was handed irrelevant context [1].

So debug retrieval first, using traditional information retrieval metrics. They are fast, deterministic, need no model calls, and give you a tight feedback loop for tuning chunking, embeddings, hybrid search, and reranking [1][2].

Recall@k is usually where to start. A generator given a longer, noisier list can often still find the right passage, so maximizing recall and letting the generation step filter is a reasonable strategy, but a passage that was never retrieved is unrecoverable.

Which k you measure at is a product decision tied to your context budget. Measuring recall@50 when you only pass 5 chunks to the model tells you about your embedding space, not your feature.

Build the retrieval test set backwards

The usual objection to IR metrics is that they need labeled query-document pairs, which sounds like weeks of annotation. It is not, because you can generate them in reverse.

Take documents from your own corpus, extract specific facts from each, and generate the questions those facts would answer. You now have query-document pairs where the correct document is known by construction, with no manual labeling [2]. Run those queries through your retriever and compute the metrics.

Three cautions. Generate from a representative spread of your corpus, not just the clean, well-structured documents. Include multi-hop questions that require two or more sources, since single-document pairs will not catch a retriever that cannot assemble evidence. And validate a sample by hand; generated questions are sometimes answerable from the document but phrased in a way no real user would use.

Supplement with real queries as soon as you have them. Production questions are messier, shorter, and more ambiguous than anything a model will generate for you.

Then evaluate generation, using a complete framework

For the generation half, there is a neat structural result worth knowing. A RAG interaction has exactly three components: the question (Q), the retrieved context (C), and the answer (A). Evaluating the quality of one given another yields exactly six relationships, and Jason Liu’s observation is that this set is exhaustive: when a RAG system fails, it fails along one of these six [1].

That completeness is useful mostly as a defense against metric sprawl. If a vendor offers twenty RAG metrics, the fair question is which of the six each one actually measures [1].

The three that matter most

Context relevance (C given Q). Do the retrieved passages actually address the question’s information need? This is the generation-side check on your retriever, and it catches a specific failure IR metrics can miss: topically related content that does not contain the needed information. A question about the health effects of a practice, answered with context describing the practice’s history and variants, is topically on point and useless [1].

Faithfulness or groundedness (A given C). Does the answer restrict itself to claims verifiable from the retrieved context? The dangerous failures here are not wild fabrications but plausible additions: a fluent sentence that may even be factually true in the world, but is not supported by anything retrieved [1]. Those are the hardest for users to catch, which is exactly why citations linked to source records matter so much in the interface. See reducing hallucinations in enterprise RAG and AI copilot UX patterns.

Answer relevance (A given Q). Does the answer address the specific need expressed? An answer can be perfectly grounded and still fail this: correct, on-topic, and never actually answering the question asked. This is your primary user-experience metric [1].

The three for deeper diagnosis

Context support coverage (C given A). Does the retrieved context contain everything needed to support every claim in the answer? This catches partial grounding, where most of an answer is supported and one clause is not [1].

Question answerability (Q given C). Given what was retrieved, is a satisfactory answer even possible? This is the metric that tells you whether a refusal was correct behavior rather than a failure [1].

Self-containment (Q given A). Could the original question be inferred from the answer alone? This matters where answers are read outside the context that produced them, such as in an email digest or a notification [1].

Emphasis should shift by domain. Systems where a wrong claim is dangerous need to weight faithfulness heavily; customer-facing support systems usually care most about answer relevance; technical documentation systems often hinge on answerability [1].

Evaluate abstention deliberately

A RAG system that never says “I don’t know” will hallucinate whenever retrieval comes up empty, and one that refuses too readily is useless. Both failures are invisible unless you test for them specifically.

Build an evaluation set with two halves: questions genuinely answerable from your corpus, and questions designed to tempt fabrication, including those with false premises, those about information deliberately absent from the corpus, and those well outside its scope. Roughly balanced is a fine starting point; diversity and difficulty matter more than the exact ratio [2].

Then score it as a single binary: the system passes only if it answers the answerable questions and declines the unanswerable ones. Fabricating an answer to an unanswerable question is the failure you are looking for [2]. In the research literature this capability is called abstention ability.

Your judges still need validating

Context relevance, faithfulness, and answer relevance all require judgment, so they are usually implemented as LLM judges. Every caution from judge validation applies: an off-the-shelf faithfulness prompt has never seen your documents, your domain, or your definition of a supported claim.

Do not skip the process. Run error analysis, iterate the judge prompt against expert labels, split into train, dev, and test, and report true positive and true negative rates rather than raw agreement. Once you know those rates, you can correct the judge’s estimates to recover the actual failure rate in your system [2]. Full method in validating an LLM judge.

Add the failure modes only your data reveals

The six relationships are complete as a structural framework, but they are generic by design. Your specific corpus will produce failures the framework names only abstractly.

A medical system that confuses adult and pediatric dosing, a legal system that mixes jurisdictions, an HR system that cites a policy superseded last quarter: each of these is technically a faithfulness or context relevance failure, and each deserves its own targeted evaluator because each has a distinct fix [2]. These surface only through systematic review of real failures, which is why error analysis comes before metric selection rather than after.

Diagnosing a failure

That last row is worth its own note. Aggregate metrics hide segment failures routinely. Always break results down by document type, query category, language, and tenant. A system at 90% overall can be at 40% on scanned PDFs, and the aggregate will never tell you.

Tune chunking as a hyperparameter

Chunk size is not a setting you get right once by reading a blog post. It interacts with your corpus, your queries, and your model, and it needs to be tuned against your retrieval metrics like any other hyperparameter [2].

The rough principle: larger chunks preserve context but increase the model’s cognitive load and risk detail in the middle being under-attended, while smaller chunks bound the problem and keep attention focused but fragment context across boundaries. You are trading global context for local focus [2]. Where you want to sit on that trade depends on whether your task has fixed-size output (extract a value, answer a specific question, which favors larger chunks) or output that grows with input (summarize each section, list every complaint, which favors smaller ones) [2].

Respect content boundaries where you can. Paragraphs, sections, and clauses make better chunk edges than a fixed token count.

What to run, and when

The cheap deterministic metrics belong in the fast loop, because they are the ones you will run hundreds of times while tuning retrieval. Judge-based metrics cost money and time, so they run on a curated set rather than everything. See regression testing AI features in CI and monitoring LLM quality in production.

Common mistakes

• Measuring only end-to-end quality, so failures cannot be attributed.
• Tuning generation prompts when the real problem is recall.
• Using off-the-shelf faithfulness judges without validating them on your documents.
• Reporting aggregate scores that hide per-format and per-tenant collapse.
• Never testing what happens when the corpus contains no answer.
• Measuring recall at a k you do not actually pass to the model.
• Treating chunk size as a fixed best practice rather than a tuned parameter.
• Adding metrics because a framework offers them, rather than because error analysis found a failure.

Checklist

• Retrieval and generation are measured separately.
• A retrieval test set exists, generated from your corpus and spot-checked by hand.
• Recall@k is measured at the k you actually use.
• Context relevance, faithfulness, and answer relevance are implemented as validated judges.
• An abstention set tests both answerable and unanswerable questions.
• Domain-specific failure modes from error analysis have their own evaluators.
• Every metric is segmented by document type, query category, and tenant.
• Chunking has been tuned against retrieval metrics, not chosen by default.
• Cheap metrics run in the fast loop; expensive ones run on a curated set.

Keep reading

• LLM evals and LLMOps: how to know your AI feature actually works (the pillar for this series)
• Error analysis for LLM failures
• How to validate an LLM judge
• Our enterprise RAG guide
• Reducing hallucinations in enterprise RAG

CMW AI Labs builds retrieval-grounded assistants with component-level evaluation from the start, so quality problems point at a cause instead of a guess. If your RAG system is wrong often enough to matter and you cannot say why, talk to our team.

Sources

1. Jason Liu, “There Are Only 6 RAG Evals,” May 2025. https://jxnl.co/writing/2025/05/19/there-are-only-6-rag-evals/
2. Hamel Husain and Shreya Shankar, “AI Evals: Everything You Need to Know,” May 2025, updated September 2026. https://hamel.dev/blog/posts/evals-faq/