Recall and precision are the two measures every student of information retrieval meets first, and for good reason. They tell us whether a system finds the relevant documents and whether it avoids the irrelevant ones. But these two numbers describe only part of what an indexing system actually does. A system can have respectable recall and precision and still frustrate users by burying good results in clutter, by quietly ignoring material it should have surfaced, or by returning the same documents the user has already read. To capture these subtler behaviours, evaluators rely on a family of additional metrics. This post walks through the most useful of them: the noise and snobbery ratios, the fallout and specificity measures, and the novelty ratio.
Table of Contents
Why recall and precision need company
Every evaluation of an indexing system rests on a simple four-way split of the document collection. For a given query, each document is either relevant or not, and either retrieved or not. This produces four groups, usually arranged in a contingency table. The relevant documents that are retrieved are the hits. The non-relevant documents that are retrieved are the noise. The relevant documents the system fails to retrieve are the misses, and the non-relevant documents it correctly leaves out are the rejected set. The widely used Indian course material from INFLIBNET on information storage and retrieval lays out this exact matrix and labels the four cells as hits, noise, misses, and rejected.
Recall and precision each use only two of these four cells. Recall compares hits to all relevant documents, while precision compares hits to everything retrieved. The other measures discussed below simply look at the same table from different corners. They were not invented to replace recall and precision but to diagnose specific kinds of failure that the headline numbers can hide. The tradition of measuring system performance this way goes back to the Cranfield experiments led by Cyril Cleverdon in the late 1950s and 1960s, which established the methodology that the field still uses.
Snobbery and noise ratios
These two measures are best understood as the “shadow” partners of precision and recall. Where precision and recall count what the system gets right, these ratios count what it gets wrong, which often makes a problem easier to spot.
The noise ratio
The noise ratio measures how much irrelevant material is mixed into the results. It is the number of non-relevant documents retrieved divided by the total number of documents retrieved. If a search returns 50 documents and only 20 are relevant, then 30 are noise, giving a noise ratio of 0.6, or 60 per cent. The noise ratio is the direct complement of precision: noise ratio = 1 − precision. Framing the problem as “noise” rather than as a precision shortfall is useful because it points the evaluator straight at the unwanted documents and prompts a practical question, namely why the indexing or matching process is letting them through. In the contingency table, noise is simply the proportion of the retrieved set that falls into the non-relevant column.
The snobbery ratio
The snobbery ratio describes the opposite weakness. It captures the tendency of a system to be too selective, so that it overlooks relevant documents that should have been retrieved. It is expressed as the proportion of relevant documents that the system fails to retrieve, which is the complement of recall. A system that retrieves only 40 of the 100 relevant documents on a topic has a snobbery ratio of 0.6, meaning it is “snobbishly” turning away 60 per cent of the useful material. The term itself dates from the Cranfield work, where it appears alongside recall and precision in the list of performance measures recorded in the Cranfield test index of retrieval measures. A high snobbery ratio is a warning sign about the indexing side of the system. It often points to weak synonym control, an incomplete thesaurus, or index terms that are too narrow to catch documents phrased differently from the query.
Fallout and specificity
Recall and precision both ignore one large group of documents: the non-relevant items that the system correctly leaves alone. Fallout and specificity bring this group into the picture, which matters because a system can look good on precision while still doing a poor job of rejecting unwanted material across a large collection.
Fallout
The fallout ratio is the proportion of non-relevant documents that are retrieved, out of all the non-relevant documents in the collection. The INFLIBNET material expresses it with the formula b/(b+d), where b is the noise and d is the correctly rejected set. Fallout answers a question that precision cannot: of all the documents that should have been ignored, how many leaked into the results? It is a measure of the errors the system makes against the irrelevant portion of the collection. One quirk is worth remembering. As the overview of information retrieval evaluation measures notes, fallout can be driven to zero simply by retrieving nothing at all, so it is only meaningful when read together with recall. The classic treatment of effectiveness measures by van Rijsbergen points out that the trade-off analysis usually applied to precision and recall can equally be carried out for the fallout and recall pair, which together describe how well a system separates the relevant from the non-relevant.
Specificity
In indexing studies, the word specificity carries two related meanings, and it helps to keep them apart. As a performance measure borrowed from classification, specificity is the proportion of non-relevant documents that the system correctly rejects, written as d/(b+d). It is the mirror image of fallout, so that fallout = 1 − specificity. A system with high specificity is good at keeping irrelevant documents out of the results. The second sense is older and belongs to indexing language itself. Here, specificity refers to how narrowly an index term describes a document. The INFLIBNET material observes that precision tends to rise with the specificity of the index terms, because a narrow, exact term matches fewer off-topic documents. Both senses point in the same practical direction. Choosing more specific index terms tightens the match between query and document, which raises specificity as a measure and lifts precision at the same time. The standard Stanford reference on evaluation in information retrieval situates these measures within the wider toolkit, including precision-recall curves and the related ROC analysis, that researchers use to compare systems.
Novelty ratio
The measures above all assume that every relevant document is equally valuable to the user. In real life this is rarely true. A user who has already read three of the relevant documents on a topic gains little from seeing them again. The novelty ratio addresses this directly. It is the proportion of the relevant retrieved documents that were previously unknown to the user, that is, brought to their attention for the first time by the search. A classification of retrieval effectiveness metrics by Demartini and Mizzaro defines novelty as exactly this percentage of relevant retrieved documents that the user did not already know, and pairs it with its companion, the coverage ratio, which is the percentage of relevant and already-known documents that the system manages to retrieve.
A high novelty ratio means the system is genuinely expanding what the user knows rather than echoing material they have already seen. This makes the measure especially valuable for current-awareness services. As an early NIST-archived discussion of performance measures including the novelty ratio explains, the metric is particularly appropriate for selective dissemination of information, or SDI, because a good current-awareness service is precisely one that surfaces documents before users learn of them through other channels. For students preparing literature reviews or running alerts on a research topic, novelty is often the difference between a useful service and a tedious one.
Choosing the right measure for the task
No single number captures the quality of an indexing system, which is exactly why these supplementary measures exist. Each one isolates a particular type of behaviour. The noise ratio flags clutter in the results. The snobbery ratio flags excessive selectivity and weak vocabulary control. Fallout and specificity describe how cleanly the system rejects irrelevant material. The novelty ratio judges whether the system is telling the user something new. A practical evaluation reads them together. A high noise ratio and adequate specificity might point to a few troublesome terms rather than a systemic fault, while a high snobbery ratio alongside a strong novelty ratio might mean a system is selective but still genuinely useful. Tracked over time, these metrics let designers see whether changes to a controlled vocabulary or a matching algorithm are fixing the specific problem they were meant to fix, rather than just shifting it elsewhere. That diagnostic precision is the real reason to look beyond recall and precision.
What do you think? If you were evaluating the catalogue of your own college library, which of these measures would tell you the most about its weaknesses, and why? And for a current-awareness alert that you personally rely on, would you trade a little precision for a higher novelty ratio?
References
- https://ebooks.inflibnet.ac.in/lisp7/chapter/evaluation-and-measurement-of-information-retrieval-system/
- https://wiki.ubc.ca/Course:LIBR557/2020WT2/cranfield_experiments
- https://www-nlpir.nist.gov/projects/irlib/pubs/cranv2/cranv2_text/subject_index_299.txt
- https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)
- https://www.dcs.gla.ac.uk/Keith/Chapter.7/Ch.7.html
- https://nlp.stanford.edu/IR-book/pdf/08eval.pdf
- https://www.ccs.neu.edu/home/vip/teach/IRcourse/5_eval_userstudy/other_notes/mizzaro_IRmetrics.pdf
- https://www-nlpir.nist.gov/projects/irlib/pubs/ire/ire_text/pg_114.txt

Leave a Reply