TREC Pooling and Relevance Judgments
Also known as: Pooling Method, Depth Pooling, TREC Pooling, Pooled Relevance Assessment
Pooling is the technique that lets the Cranfield evaluation paradigm scale to collections of millions of documents, where judging every document for every topic is impossible. Developed and institutionalized at the US National Institute of Standards and Technology for the Text REtrieval Conference (TREC), pooling gathers the top-ranked documents returned by many participating systems for each topic, merges them into a single pool, has human assessors judge only that pool, and treats every unjudged document as non-relevant. The result is a reusable test collection — documents, topics, and pooled relevance judgments (qrels) — on which new systems can later be scored without further assessment. Pooling is what made large-scale, reproducible retrieval evaluation feasible.
Key highlights
- Makes relevance assessment feasible on very large collections by bounding judging to a finite, system-targeted pool.
- Produces reusable test collections that can score future systems without new human judgments.
- Leverages diversity across many systems so the pool covers most relevant documents despite judging only a fraction.
- Underpins decades of reproducible, comparable shared-task evaluation in TREC, CLEF, and NTCIR.
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
Use pooling whenever you must build a reusable test collection over a corpus too large to judge exhaustively, especially when you can gather ranked runs from several diverse systems — the classic setting of a shared evaluation campaign. It is the right approach when the goal is a permanent, reusable benchmark and you can afford a bounded but substantial assessment budget. Pooling is less safe when only one or a few similar systems contribute (coverage of relevant documents will be poor), when relevant documents are extremely sparse or extremely abundant, or when you intend to score systems radically unlike those that built the pool — there the unjudged-as-non-relevant assumption can bias results, and you should use bias-aware measures or supplementary judging.
Strengths & limitations
- Makes relevance assessment feasible on very large collections by bounding judging to a finite, system-targeted pool.
- Produces reusable test collections that can score future systems without new human judgments.
- Leverages diversity across many systems so the pool covers most relevant documents despite judging only a fraction.
- Underpins decades of reproducible, comparable shared-task evaluation in TREC, CLEF, and NTCIR.
- Unjudged documents are assumed non-relevant, which biases scores against systems that retrieve relevant-but-unpooled documents.
- Reusability is not guaranteed: collections built from few or similar systems can be unfair to novel later systems.
- Finite pool depth misses some relevant documents, so recall-based measures can be systematically understated.
- Assessment, though bounded, is still costly and subject to assessor inconsistency that propagates into every score.
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
Why is it acceptable to assume unjudged documents are non-relevant?
Because a broad, deep pool drawn from many diverse systems surfaces nearly all the relevant documents, so the documents no system ranked highly are overwhelmingly non-relevant. TREC research showed that under these conditions the assumption barely perturbs the relative ranking of systems, which is what evaluation cares about. The assumption is least safe when pools are shallow, drawn from few or similar systems, or when relevant documents are very sparse — situations where genuinely relevant documents may be missing from the pool and wrongly scored as non-relevant.
Will a pooled collection fairly evaluate a brand-new system?
Often yes, but it must be checked. A system that did not contribute to the pool may retrieve relevant documents that were never judged, which are counted against it and deflate its score. Voorhees and Harman's TREC studies found that with sufficiently broad and deep pools the bias is usually small and the system ordering is stable, but reusability is an empirical property of each collection. Leave-one-group-out experiments and bias-aware measures (such as bpref or condensed measures) are used to confirm a collection is safe for out-of-pool systems.
How does pool depth affect the evaluation?
Pool depth d sets how many top documents from each run are judged, trading assessment cost against coverage of relevant documents. A deeper pool captures more relevant material, raising the reliability of recall-oriented measures, but costs proportionally more human judging. TREC's classic depth of 100 was chosen as a workable compromise for its collections. The right depth depends on how densely relevant documents are spread through the rankings: sparser relevance can tolerate shallower pools, while topics with many relevant documents need deeper ones to avoid understating recall.
Sources
- 1.Voorhees, E. M., & Harman, D. K. (Eds.). (2005). TREC: Experiment and Evaluation in Information Retrieval. MIT Press.ISBN 9780262220736
- 2.Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press.ISBN 9780521865715
- 3.Cleverdon, C. W. (1967). The Cranfield tests on index language devices. Aslib Proceedings, 19(6), 173-194.
You have read it. What now?
Cite this page
ScholarGate. (2026, June 23). TREC Pooling and Relevance Judgments. ScholarGate. https://scholargate.app/library-information-science/trec-pooling-relevance-judgments