Compare methods

	Inter-Indexer Consistency	TREC Pooling and Relevance Judgments
Field	Library Information Science	Library Information Science
Family	Process / pipeline	Process / pipeline
Year of origin	1969	2005
Originator	Pranas Zunde & Margaret Dexter; L. Rolling (Hooper coefficient tradition)	Ellen M. Voorhees & Donna K. Harman (NIST TREC)
Type	Agreement measurement for index-term assignment	Pooled relevance-assessment pipeline for large test collections
Seminal source	Rolling, L. (1981). Indexing consistency, quality and efficiency. Information Processing & Management, 17(2), 69-76.	Voorhees, E. M., & Harman, D. K. (Eds.). (2005). TREC: Experiment and Evaluation in Information Retrieval. MIT Press. ISBN: 9780262220736
Aliases	Indexer Agreement, Inter-Indexer Agreement, Hooper Consistency, Indexing Reliability	Pooling Method, Depth Pooling, TREC Pooling, Pooled Relevance Assessment
Related	3	3
Summary	Inter-indexer consistency measures how far two or more people agree when they independently assign subject terms to the same documents. Because subject indexing is a judgment task — choosing which descriptors best represent a document's content — different indexers routinely pick overlapping but not identical term sets, and the degree of that overlap is a fundamental indicator of the reliability of an indexing system. The standard quantity is the Hooper-style consistency coefficient, the size of the shared term set divided by the size of the combined term set, averaged across documents; Zunde and Dexter and later Rolling refined it and connected it to indexing quality. Low consistency signals that retrieval will be unpredictable, since whether a document is found can depend on which indexer happened to process it.	Pooling is the technique that lets the Cranfield evaluation paradigm scale to collections of millions of documents, where judging every document for every topic is impossible. Developed and institutionalized at the US National Institute of Standards and Technology for the Text REtrieval Conference (TREC), pooling gathers the top-ranked documents returned by many participating systems for each topic, merges them into a single pool, has human assessors judge only that pool, and treats every unjudged document as non-relevant. The result is a reusable test collection — documents, topics, and pooled relevance judgments (qrels) — on which new systems can later be scored without further assessment. Pooling is what made large-scale, reproducible retrieval evaluation feasible.