Cranfield Evaluation Paradigm

It does so deliberately, to gain control and reproducibility. By fixing the documents, topics, and relevance judgments, the design makes effectiveness a quantity that can be measured identically for every system and re-checked by anyone, which is impossible when results depend on which user happened to search. The cost is that the paradigm abstracts away interaction, satisfaction, and learning. The standard response is complementary: use Cranfield-style evaluation to compare core retrieval effectiveness, and use user and session studies to capture the interactive dimension it omits.

Recall is the fraction of all relevant documents that a system retrieved, measuring completeness; precision is the fraction of retrieved documents that are relevant, measuring purity. Cleverdon's analysis showed these usually trade off: widening retrieval to capture more relevant items typically also drags in more irrelevant ones. Reporting both — often as a recall-precision curve — characterizes a system far better than either alone, which is why the paradigm treats them as a pair rather than collapsing them into a single score by default.

TREC is the Cranfield paradigm scaled up. The Text REtrieval Conference kept Cranfield's essential design — a fixed collection, a topic set, reusable relevance judgments, and comparison on averaged effectiveness — but extended it to very large document collections by introducing pooling to make judgment feasible and by organizing many research groups around shared tasks. Voorhees and Harman's account of TREC makes the lineage explicit: the laboratory logic is Cleverdon's; the scale, infrastructure, and pooled-judgment machinery are TREC's contributions on top of it.