Indexing Consistency Analysis

Inter-indexer consistency reports a single agreement figure; indexing consistency analysis explains and acts on it. The analysis decomposes variability into between- and within-indexer components, models the factors (exhaustivity, vocabulary, subject, experience) that drive disagreement, links inconsistency to retrieval outcomes, and prescribes interventions. In short, the consistency coefficient is the input; the analysis is the diagnostic workflow built around it. You would compute inter-indexer consistency to get a number and run the full analysis when you need to understand why that number is what it is and how to improve it.

Because the two diagnose different problems. Between-indexer consistency tells you whether different people agree; within-indexer consistency — comparing an indexer to themselves on re-indexing the same documents after a delay — tells you whether the task itself is stable. If even a single indexer cannot reproduce their own decisions, the indexing rules or vocabulary are ambiguous and no amount of harmonizing across people will help. High within- but low between-indexer agreement instead points to stable personal styles that training or guidelines can align. The contrast directs the fix to the right level.

It can, but not uniformly, which is why the analysis estimates the link rather than assuming it. Inconsistency on terms that searchers actually use, or on a document's central subject, scatters similar documents under different headings and makes them harder to find; inconsistency on rarely searched peripheral terms may cost almost nothing. Rolling raised exactly this question of whether consistency tracks effectiveness. By tracing retrieval measures across consistency levels and identifying which disagreements have retrieval impact, the analysis separates the inconsistency worth fixing from the inconsistency that is merely cosmetic.