Inter-Indexer Consistency

Because subject indexing is a judgment task with no single correct answer. Indexers differ in how exhaustively they index, how specific a term they choose, and how they interpret a document's aboutness, so they end up with overlapping but non-identical term sets. Decades of studies have found agreement to be persistently modest, especially as indexing depth increases. This is not necessarily a failure of the indexers; it reflects the genuine indeterminacy of subject representation, which is precisely why measuring consistency — and trying to improve it through vocabularies and guidelines — is a standard part of indexing quality control.

Not by itself. Consistency measures agreement, not correctness, so two indexers could reliably assign the same inappropriate terms and score high. Rolling examined this very gap, asking whether consistency can stand in for quality, and the answer is qualified: consistency is necessary for predictable retrieval but not sufficient for good retrieval. A complete evaluation pairs consistency with a quality or effectiveness assessment — for instance, checking whether the agreed terms actually support finding the documents — rather than relying on agreement alone.

Strongly, and usually downward. When indexers assign more terms per document (higher exhaustivity) or use a more specific vocabulary, there are more opportunities to diverge, so raw set-overlap coefficients tend to fall even if the indexers agree on the central subjects. This is why importance-weighted measures, which credit agreement on the key descriptors over peripheral ones, give a fairer picture, and why consistency figures must be interpreted relative to the indexing depth and vocabulary specificity of the operation rather than compared blindly across systems.