השוואת שיטות

	Indexing Consistency Analysis	Cranfield Evaluation Paradigm
תחום	Library Information Science	Library Information Science
משפחה	Process / pipeline	Process / pipeline
שנת המקור	1981	1967
הוגה השיטה	L. Rolling; Pranas Zunde & Margaret Dexter (indexing-evaluation tradition)	Cyril W. Cleverdon
סוג	Diagnostic analysis of indexing variability and its retrieval consequences	Test-collection evaluation pipeline for retrieval effectiveness
מקור מכונן	Rolling, L. (1981). Indexing consistency, quality and efficiency. Information Processing & Management, 17(2), 69-76.	Cleverdon, C. W. (1967). The Cranfield tests on index language devices. Aslib Proceedings, 19(6), 173-194.
כינויים	Indexing Variability Analysis, Indexing Reliability Analysis, Indexing Consistency Study, Within- and Between-Indexer Consistency Analysis	Cranfield Methodology, Test Collection Evaluation, Cranfield Tests, Laboratory IR Evaluation
קשורות	3	3
תקציר	Indexing consistency analysis goes beyond reporting a single agreement number to diagnose why indexing varies and what that variability costs. It distinguishes between-indexer consistency (do different people agree?) from within-indexer consistency (does the same person agree with themselves on re-indexing?), models how factors such as indexing exhaustivity, vocabulary specificity, document subject, and indexer experience drive the variability, and — following Rolling's question of whether consistency stands in for quality — traces how inconsistency degrades retrieval. The aim is actionable: identify the terms, subjects, and conditions where indexers diverge most, and feed that back into guidelines, vocabulary design, and training.	The Cranfield evaluation paradigm is the foundational experimental design for measuring how well an information retrieval system finds relevant documents. Devised by Cyril Cleverdon at the College of Aeronautics in Cranfield during the 1960s, it fixes three ingredients — a document collection, a set of search requests, and human relevance judgments linking requests to documents — and then holds them constant so that competing indexing methods or retrieval algorithms can be compared on recall and precision under controlled, repeatable conditions. By abstracting evaluation away from any single live user and turning it into a reusable laboratory experiment, Cranfield made retrieval effectiveness a measurable quantity and supplied the template that every later large-scale campaign, including TREC, has built upon.