Measuring Retrieval Quality: What 80 Questions Can Tell You

Introduction

Retrieval evaluation is how you find out whether the search in a retrieval system hands the model the right material. A set of test questions goes in, each with a known answer, and a score comes out. Every decision in the chunking post rested on one of those scores: the share of questions whose answering passage appeared in the top five results.

This post is the second in the series that builds one retrieval system over this blog. It takes the same 23 posts and the same 80 questions and asks how far the scores can be trusted. The answer on 80 questions is about 9 points either way. Two of last week's comparisons survive that test and three do not, including the gain from chunk overlap. Generated test questions turned out to hide broken questions inside a normal-looking average, and a judge model grading the final answers caught failures the passage score cannot see.

Score the searchGrade the answerAll routes

Scoring the retriever alone, against an answer key written before the run. The score comes with an interval, because 80 questions is a small sample.

Grading what the reader actually gets. The same results feed a model, and a second model grades the answer, because a right passage does not guarantee a right answer.

Every route at once: one dot leaves the start and splits wherever the routes go different ways.

1

Each of the 80 test questions has an answer key: the passage that answers the question.

The retriever ranks every chunk against the question and keeps the top 10.

The top 10 are compared to the answer key, giving recall, MRR and nDCG for that question.

Resampling the 80 questions gives a 95% interval of about ± 9 to 10 points on the top-5 hit rate.

The overlap gain of 7.5 points falls inside that interval, so the run cannot tell overlap from noise.

The same 80 test questions go in.

The retriever ranks the chunks exactly as before.

The top 10 results are the same list the search score used.

Only the top 5 chunks go to the model, the same cut a reader would get.

The model writes an answer from the five chunks. A model answer can mix two numbers even when the right passage is present.

A judge model reads the answer next to the answer key.

The judge returns correct, partial or wrong for each question.

All 5 steps

1. 1.Each of the 80 test questions has an answer key: the passage that answers the question.
2. 2.The retriever ranks every chunk against the question and keeps the top 10.
3. 3.The top 10 are compared to the answer key, giving recall, MRR and nDCG for that question.
4. 4.Resampling the 80 questions gives a 95% interval of about ± 9 to 10 points on the top-5 hit rate.
5. 5.The overlap gain of 7.5 points falls inside that interval, so the run cannot tell overlap from noise.

1. 1.The same 80 test questions go in.
2. 2.The retriever ranks the chunks exactly as before.
3. 3.The top 10 results are the same list the search score used.
4. 4.Only the top 5 chunks go to the model, the same cut a reader would get.
5. 5.The model writes an answer from the five chunks. A model answer can mix two numbers even when the right passage is present.
6. 6.A judge model reads the answer next to the answer key.
7. 7.The judge returns correct, partial or wrong for each question.

• Score the search · Score the search: rank, compare, resampleStep through
• Grade the answer · Grade the answer: a model answers, a judge gradesStep through

Use the zoom buttons, or focus the diagram and press plus or minus to zoom, arrow keys to pan, and 0 to reset.

Four Ways to Score a Ranked List

A search returns a ranked list. Each scoring method looks at a different part of it.

1/** Recall at k: did any relevant item make the top k? (1 or 0 per2 *  question; averaged over questions this is the "hit rate".) */3export function recallAtK(relevance: number[], k: number): number {4  return relevance.slice(0, k).some((r) => r > 0) ? 1 : 0;5}6 7/** Precision at k: what share of the top k is relevant. */8export function precisionAtK(relevance: number[], k: number): number {9  return relevance.slice(0, k).filter((r) => r > 0).length / k;10}

nDCG is the only one of the four that takes graded relevance. Here a returned chunk scores 2 if it holds the whole answering passage, 1 if it holds part of it, and 0 otherwise. Each grade is discounted by $\log_2$ of its position, so a perfect chunk at rank one counts more than the same chunk at rank five, and the total is divided by the best score the same chunks could have reached in the best order.

1/** nDCG at k: graded relevance, discounted by position (log2), divided by2 *  the best score the same items could get in the best possible order. */3export function ndcgAtK(relevance: number[], k: number, ideal: number[] = relevance): number {4  const dcg = (rels: number[]) =>5    rels.slice(0, k).reduce((sum, r, i) => sum + (2 ** r - 1) / Math.log2(i + 2), 0);6  const best = dcg([...ideal].sort((a, b) => b - a));7  return best === 0 ? 0 : dcg(relevance) / best;8}

Here is every measure for last week's strategies on the same 80 questions:

When Each Measure Misleads

Precision has a ceiling of 20%. Each question has one answering passage, so at most one of five results can be relevant, and precision at five cannot pass 0.20. Whole posts, parent-child sections and overlapping chunks all sit at 19.0% or close to it. Overlap scores 18.5% against plain fixed chunks' 14.7% partly because the same passage appears in two overlapping chunks, which counts twice. Precision is a useful measure when a question has many relevant documents, as in product search. For "find the one passage that answers this", it mostly rewards duplication.

Recall at 10 rewards a small haystack. Whole posts reach 100% at ten results because there are only 23 posts to choose from: ten results is 43% of the collection. Recall at a fixed $k$ is only comparable between systems that return results of similar size, which is why the chunking post reported the words sent alongside every hit rate.

MRR and recall disagree about the same system. Parent-child with 400-word sections and with 200-word sections both found 95.0% in the top five, but the larger sections ranked the answer first more often, so MRR separates them (0.843 against 0.790). Recall says the two are equal; MRR says one puts the answer where a reader looks first.

nDCG disagrees with recall about overlap. With partial credit, overlap barely beats plain fixed chunks (0.717 against 0.713), even though its recall at five is 7.5 points higher. When a fixed chunk cuts an answer in two, nDCG still gives credit for the halves, and the halves were usually there. The chunking post found the same effect with its coverage measure; nDCG shows it through a standard metric.

How Much of a Score Is Noise

Eighty questions is a sample. A different 80 questions from the same posts would give different scores, and the gap between two strategies has to be larger than that wobble before it means anything.

The bootstrap estimates the wobble without collecting more questions. It resamples the 80 questions with replacement thousands of times, scores each resample, and reads the spread of the results.

1/** Seeded random numbers (mulberry32), so every interval is reproducible. */2export function rng(seed: number): () => number {3  let a = seed >>> 0;4  return () => {5    a = (a + 0x6d2b79f5) >>> 0;6    let t = a;7    t = Math.imul(t ^ (t >>> 15), t | 1);8    t ^= t + Math.imul(t ^ (t >>> 7), t | 61);9    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;10  };11}12 13/** 95% bootstrap interval for the mean of per-question scores: resample14 *  the questions with replacement many times and take the middle 95% of15 *  the resampled means. */16export function bootstrapInterval(scores: number[], rounds = 10_000, seed = 1): [number, number] {17  const next = rng(seed);18  const means: number[] = [];19  for (let b = 0; b < rounds; b++) {20    let sum = 0;21    for (let i = 0; i < scores.length; i++) sum += scores[Math.floor(next() * scores.length)];22    means.push(sum / scores.length);23  }24  means.sort((x, y) => x - y);25  return [means[Math.floor(rounds * 0.025)], means[Math.floor(rounds * 0.975)]];26}27 28/** Paired bootstrap for "is A better than B on the same questions?":29 *  resample questions, keep each question's pair together, and report the30 *  95% interval of the difference in means. An interval that crosses 031 *  means the questions cannot tell the two apart. */32export function pairedDifference(a: number[], b: number[], rounds = 10_000, seed = 1): [number, number] {33  if (a.length !== b.length) throw new RangeError("score lists must be paired");34  return bootstrapInterval(a.map((x, i) => x - b[i]), rounds, seed);35}

On recall at five, the 95% interval runs about 9 points either side for most strategies: recursive chunking's 85.0% could be anything from 77.5% to 92.5%, and fixed chunks' 73.8% anything from 63.7% to 82.5%. Comparing two strategies is sharper than that, because both run on the same questions, so the paired version resamples questions and keeps each question's two outcomes together.

Two of last week's conclusions hold: respecting sentence boundaries beats cutting blindly, and parent-child retrieval beats recursive chunking alone. The ranking among overlap, sentence and recursive chunking does not. Those three are within the noise of each other on this question set, and the chunking post's ordering of them should be read as a tie.

How many questions would settle it? Resampling the 80 up to larger sizes gives the interval each size would produce:

The interval shrinks with the square root of the number of questions, so halving it costs four times the questions. Telling apart two strategies a few points apart takes several hundred questions; 80 is enough to catch differences of ten points or more.

Where Test Questions Come From

The 80 questions in this series were written in a reader's words, each paired with the exact passage that answers it. The common shortcut is to have a model write them: hand it each passage and ask for a question the passage answers. To see what that does, llama3.2 wrote one question for each of the same 80 passages.

The model's questions reused far more of the passage's wording, 40.9% of their words against 27.1% for the reader-style questions. The usual worry about generated questions is exactly that copying, which should make search look easier. It did not: recursive chunking found 76.3% of the model's questions against 85.0% of the reader-style ones, a gap whose paired interval crosses zero.

Reading all 80 explains why the average looks ordinary:

Thirteen questions misstated their passage. One asked whether an admissible heuristic is an upper bound on the true cost; the passage says lower. Another asked what share of objects were requested more than once, when the passage says exactly once. Eleven depended on context the question did not carry, such as "what is the time complexity of this algorithm?". Eleven more turned the passage's own sentence into a yes/no question. The restating questions inflate the score and the wrong ones deflate it, so the average landed close to the reader-style score while 35 of 80 questions were unfit to test anything. The score alone gives no sign of that. Reading the questions does.

Real user queries are the best source when a system has users: they are worded the way people ask, including the vague and the misspelled. A collection like this blog's has none yet, which is why the series uses written questions with exact answers and states their limits.

Grading the Answer as Well as the Search

Every score so far asks whether search found the passage. What a user sees is the answer. To measure that directly, a small model (llama3.2) answered each question from its top five chunks, and a larger model (gpt-oss:20b, both running on a homelab GPU) graded each answer against the reference passage as correct, partial or wrong.

The two scores drift apart in both directions. Parent-child retrieval put the answering passage in front of the model 95.0% of the time, but only 81.3% of answers came out right. In 13 cases the passage was there and the answer still failed. The model replied "not in the context" with the answer in the context, and in one case read "a flat 4.8 more bits per item" and multiplied 4.8 by 9. Going the other way, fixed chunks produced 12 correct answers without ever returning the whole passage, because the halves of a cut answer were enough. For fixed chunks the two errors cancel exactly, which makes the strict score look right for the wrong reason.

A judge is a model too, so it needs checking. Two judges graded the same 240 answers and agreed on correct-or-not 85.4% of the time, with llama3.2 marking 31 more answers correct than gpt-oss:20b. Grading a random 20 of those answers by hand against the reference passage, gpt-oss:20b matched the hand grade on all 20 and llama3.2 on 17. All three of llama3.2's misses were leniency: it passed an answer that turned 4.8 into 48, one that left out the measured number entirely, and one that added a false claim.

When You Do Not Need It

• A handful of questions you already know the answers to. Ten questions checked by hand catch the obvious failures. The statistics matter when choosing between options that score close together.
• The difference is large. A change that moves recall from 50% to 90% does not need an interval to be believed; 80 questions separate differences of ten points or more.
• You only care about one configuration. Intervals matter most for comparisons. A single score is still worth reporting with its interval, so nobody reads 85.0% as exact.

Conclusion

A retrieval score is an estimate, and on 80 questions it carries about 9 points of noise either way. Paired bootstrap intervals separated two of last week's comparisons, sentence over fixed chunks and parent-child over recursive, and left three as ties, including the overlap gain. Precision capped out at 20%, recall at ten rewarded a small collection, and MRR and nDCG each disagreed with recall about something worth knowing. Generated questions copied the passages' wording but averaged out close to the reader-style score while 35 of them were broken in ways only reading could find. Finding the passage also fell short of a right answer. Parent-child retrieval found it 95.0% of the time, and a judge model, checked against hand grades, marked 81.3% of the answers correct.

Key Takeaways

• On 80 questions, recall at five carried a 95% interval of about 9 points either way; reaching about 4 points takes around 320 questions.
• Overlap's 7.5-point gain from last week has a paired interval of −2.5 to 17.5 points, so it is not separated from noise; sentence over fixed and parent-child over recursive (10.0 points each) are.
• Precision at five cannot pass 20% when each question has one answering passage, so it mostly rewards duplicated text.
• MRR separated two parent-child setups that recall scored equal (0.843 against 0.790), and nDCG barely separated overlap from plain fixed chunks (0.717 against 0.713).
• Model-written questions reused 40.9% of the passage's words against 27.1% for reader-style questions, and 35 of 80 were wrong, vague or restated the passage, yet the average stayed close.
• Parent-child retrieval found the passage for 95.0% of questions, but only 81.3% of answers were judged correct; gpt-oss:20b matched a hand grade on 20 of 20 answers, llama3.2 on 17.

Sources

• Chroma, Evaluating Chunking Strategies for Retrieval
• Efron and Tibshirani, An Introduction to the Bootstrap (Chapman and Hall, 1993)
• Järvelin and Kekäläinen, Cumulated Gain-Based Evaluation of IR Techniques (ACM TOIS, 2002), the origin of nDCG