手法を比較

	Indexing Consistency Analysis	Cranfield Evaluation Paradigm
分野	Library Information Science	Library Information Science
系統	Process / pipeline	Process / pipeline
提唱年	1981	1967
提唱者	L. Rolling; Pranas Zunde & Margaret Dexter (indexing-evaluation tradition)	Cyril W. Cleverdon
種類	Diagnostic analysis of indexing variability and its retrieval consequences	Test-collection evaluation pipeline for retrieval effectiveness
原典	Rolling, L. (1981). Indexing consistency, quality and efficiency. Information Processing & Management, 17(2), 69-76.	Cleverdon, C. W. (1967). The Cranfield tests on index language devices. Aslib Proceedings, 19(6), 173-194.
別名	Indexing Variability Analysis, Indexing Reliability Analysis, Indexing Consistency Study, Within- and Between-Indexer Consistency Analysis	Cranfield Methodology, Test Collection Evaluation, Cranfield Tests, Laboratory IR Evaluation
関連	3	3
概要	Indexing consistency analysis goes beyond reporting a single agreement number to diagnose why indexing varies and what that variability costs. It distinguishes between-indexer consistency (do different people agree?) from within-indexer consistency (does the same person agree with themselves on re-indexing?), models how factors such as indexing exhaustivity, vocabulary specificity, document subject, and indexer experience drive the variability, and — following Rolling's question of whether consistency stands in for quality — traces how inconsistency degrades retrieval. The aim is actionable: identify the terms, subjects, and conditions where indexers diverge most, and feed that back into guidelines, vocabulary design, and training.	The Cranfield evaluation paradigm is the foundational experimental design for measuring how well an information retrieval system finds relevant documents. Devised by Cyril Cleverdon at the College of Aeronautics in Cranfield during the 1960s, it fixes three ingredients — a document collection, a set of search requests, and human relevance judgments linking requests to documents — and then holds them constant so that competing indexing methods or retrieval algorithms can be compared on recall and precision under controlled, repeatable conditions. By abstracting evaluation away from any single live user and turning it into a reusable laboratory experiment, Cranfield made retrieval effectiveness a measurable quantity and supplied the template that every later large-scale campaign, including TREC, has built upon.