Relevance Feedback Evaluation
Also known as: Rocchio Feedback Evaluation, Feedback Effectiveness Measurement, Residual Collection Evaluation, Relevance Feedback Assessment
Relevance feedback evaluation measures how much a retrieval system improves when it reformulates a query using user judgments on the first results. The technique that defined the field is Rocchio's vector-space feedback, in which documents the user marks relevant pull the query vector toward themselves and documents marked non-relevant push it away; Salton and Buckley's 1990 study systematized its evaluation and showed substantial effectiveness gains. The central methodological challenge is fairness: because the user has already seen and judged some documents, naively re-scoring the whole collection rewards the system for re-finding documents it was just told about. Residual-collection and frozen-rank evaluation solve this by measuring improvement only on documents the user has not yet seen.
Key highlights
- Isolates the genuine benefit of feedback by removing already-seen documents, preventing the inflated scores that naive re-ranking produces.
- Builds directly on the well-understood Rocchio vector-space model, giving an interpretable, tunable account of how feedback reshapes a query.
- Integrates cleanly with the Cranfield paradigm and test collections, so feedback gains are reproducible and comparable across systems.
- Generalizes naturally to pseudo-relevance feedback and to interactive and learning-based reformulation, providing a common evaluation template.
Intuition
This section is available to Pro members. Upgrade to Pro
How it works
This section is available to Pro members. Upgrade to Pro
When to use it
Use relevance feedback evaluation whenever you are assessing a system that revises a query from user judgments — classic Rocchio feedback, pseudo-relevance feedback, or interactive learning-to-rank — and you need the measured improvement to be credible rather than inflated. It is appropriate when you have a test collection with relevance judgments and can define a clear feedback protocol (how many documents are shown, how relevance is marked). The residual-collection and frozen-rank designs are essential whenever the user has seen documents before the second retrieval. The approach is less relevant for purely one-shot retrieval with no reformulation, and for pseudo-relevance feedback (where no human marks documents) the fairness concern is weaker, though consistent residual or held-out evaluation is still good practice.
Strengths & limitations
- Isolates the genuine benefit of feedback by removing already-seen documents, preventing the inflated scores that naive re-ranking produces.
- Builds directly on the well-understood Rocchio vector-space model, giving an interpretable, tunable account of how feedback reshapes a query.
- Integrates cleanly with the Cranfield paradigm and test collections, so feedback gains are reproducible and comparable across systems.
- Generalizes naturally to pseudo-relevance feedback and to interactive and learning-based reformulation, providing a common evaluation template.
- Automatic experiments simulate user judgments from qrels, which may not reflect how real users actually mark relevance.
- Residual-collection scores are not directly comparable to ordinary full-collection scores, complicating cross-study comparison.
- Measured gains depend heavily on the feedback protocol — number of documents shown, weighting parameters — so results are protocol-specific.
- Negative feedback is noisy and often contributes little, so the model's symmetric form can overstate the role of non-relevant documents.
Common pitfalls
This section is available to Pro members. Upgrade to Pro
Applications
This section is available to Pro members. Upgrade to Pro
Frequently asked
Why must already-seen documents be removed before evaluating feedback?
Because relevance feedback is built from documents the user has already inspected and judged. If those documents stay in the collection during the second evaluation, the reformulated query will naturally rank the known relevant ones near the top, producing a large apparent improvement that is worthless to the user — they have already seen those documents. Residual-collection evaluation removes the seen documents (and frozen-rank evaluation freezes them in place) so the measured gain reflects only newly surfaced relevant documents. This is the methodological contribution Salton and Buckley emphasized.
How does Rocchio's formula actually change the query?
It moves the query vector. The modified query is a weighted sum of the original query, the centroid (average vector) of the documents marked relevant, and the negated centroid of those marked non-relevant, with weights alpha, beta, and gamma. Adding the relevant centroid pulls the query toward terms common in good documents; subtracting the non-relevant centroid pushes it away from terms common in bad ones. Salton and Buckley found that weighting positive feedback heavily and negative feedback lightly works best, since non-relevant documents are heterogeneous and their centroid is a weaker signal.
Is pseudo-relevance feedback evaluated the same way?
Largely, with one difference. Pseudo-relevance (blind) feedback assumes the top initial documents are relevant without any human judgment, then reformulates as if they were marked relevant. Since no user actually inspected documents, the fairness motivation for removing seen documents is weaker, but consistent held-out or residual evaluation against the real qrels is still used so that the gain measured is genuine and comparable. The Rocchio-style reformulation and the recall-precision or MAP comparison are identical; only the source of the 'relevant' set changes.
Sources
- 1.Salton, G., & Buckley, C. (1990). Improving retrieval performance by relevance feedback. Journal of the American Society for Information Science, 41(4), 288-297.
- 2.Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press.ISBN 9780521865715
- 3.Voorhees, E. M., & Harman, D. K. (Eds.). (2005). TREC: Experiment and Evaluation in Information Retrieval. MIT Press.ISBN 9780262220736
You have read it. What now?
Cite this page
ScholarGate. (2026, June 23). Relevance Feedback Evaluation. ScholarGate. https://scholargate.app/library-information-science/relevance-feedback-evaluation