Known-Item Search Success
Also known as: Known-Item Retrieval Evaluation, Target Document Search Evaluation, Reciprocal Rank Evaluation, Known-Item Finding Success
Known-item search is the case where the user is looking for one specific document they already know exists — a particular paper, book, web page, or record — rather than exploring a topic. Evaluation is correspondingly specialized: with exactly one correct answer per query, the question is simply how high the system ranks that single target. The natural measures are reciprocal rank (and its mean, MRR), success-at-k, and Cooper's expected search length, which counts how many wrong documents the user must wade through before reaching the right one. These metrics, averaged over many known-item topics, give a clean, interpretable picture of how well a system supports re-finding a specific document.
Key highlights
- Matches the task structure exactly: with one correct answer, the metrics are simple, interpretable, and need no graded relevance.
- Mean reciprocal rank cleanly rewards placing the target at or near the top, mirroring how users re-find documents.
- Expected search length expresses effectiveness as concrete user effort — the number of wrong documents passed — rather than an abstract score.
- Success-at-k gives an operational, threshold-based answer to whether the target reaches the page the user actually inspects.
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
Use known-item evaluation whenever the task has a single correct answer per query — re-finding a specific document, navigational web search to a known page, entity or fact lookup, question answering with one gold passage, or home-page finding. It is appropriate when you can build topics each tied to exactly one target and you care about how quickly the user reaches that target. Mean reciprocal rank, success-at-k, and expected search length are the right tools here. The approach is not suited to topical or exploratory search, where many documents are relevant and partial credit matters; there you need set- and ranking-based measures like precision, recall, MAP, or nDCG, which the single-target metrics deliberately simplify away.
Strengths & limitations
- Matches the task structure exactly: with one correct answer, the metrics are simple, interpretable, and need no graded relevance.
- Mean reciprocal rank cleanly rewards placing the target at or near the top, mirroring how users re-find documents.
- Expected search length expresses effectiveness as concrete user effort — the number of wrong documents passed — rather than an abstract score.
- Success-at-k gives an operational, threshold-based answer to whether the target reaches the page the user actually inspects.
- Applies only when exactly one document is the correct answer; it is meaningless for topical search with many relevant items.
- Mean reciprocal rank is dominated by the top rank and conveys little about behavior deeper in the list.
- A single target makes the measure sensitive to noise: one badly ranked query can swing a small evaluation set.
- Constructing realistic known-item topics is hard, since artificial title-based queries may not resemble how users actually remember items.
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
How is known-item evaluation different from ordinary precision and recall?
Precision and recall assume many documents can be relevant and measure how completely and purely a system retrieves that set. Known-item search has exactly one correct answer, so those set-based measures collapse to a trivial form and the meaningful question becomes positional: how high is the single target ranked? That is why known-item evaluation uses reciprocal rank, success-at-k, and expected search length instead. Applying full precision-recall machinery here adds no information and can even mislead, since there is no set of relevant documents to be complete about.
Why use mean reciprocal rank rather than just average rank?
Because users care disproportionately about the top positions. Average rank treats the difference between positions 1 and 2 the same as between 51 and 52, but for re-finding a document, moving the target from rank 50 to 49 barely helps while moving it from 2 to 1 matters a lot. Reciprocal rank captures this by weighting top positions heavily and flattening out deep in the list, and its mean across topics, MRR, has become the standard known-item and question-answering measure for exactly this reason.
What does expected search length add over MRR and success-at-k?
It expresses effectiveness as concrete user effort. Cooper's expected search length counts how many non-relevant documents the user must examine before reaching what they want — for a single target, simply the rank minus one. Where MRR is an abstract score and success-at-k a yes/no threshold, ESL says directly how much wasted scanning the user endures, which is interpretable to practitioners and generalizes cleanly to cases where the user needs several items rather than one. The three measures are complementary views of the same single-target outcome.
Sources
- 1.Cooper, W. S. (1968). Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems. American Documentation, 19(1), 30-41.
- 2.Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press.ISBN 9780521865715
- 3.Voorhees, E. M., & Harman, D. K. (Eds.). (2005). TREC: Experiment and Evaluation in Information Retrieval. MIT Press.ISBN 9780262220736
You have read it. What now?
Cite this page
ScholarGate. (2026, June 23). Known-Item Search Success. ScholarGate. https://scholargate.app/library-information-science/known-item-search-success