Building an information storage and retrieval (ISAR) system is only half the job. The harder question comes after it goes live: does it actually work? An ISAR system that holds millions of records is useless if it buries relevant results under irrelevant ones, takes too long to respond, or frustrates the people using it. Evaluation is how we answer that question with evidence rather than guesswork. It tells us whether a system meets its objectives, how efficiently it does so, and whether it deserves the money and effort spent on it. This post walks through the main criteria and methods used to judge how well an ISAR system performs.
Table of Contents
- Why evaluation matters
- The foundation: the Cranfield experiments
- System coverage and the criteria of effectiveness
- What coverage means
- The trickier criteria
- Recall and precision: the core measures
- The recall and precision trade-off
- Different users want different balances
- The F-measure
- User experience and response time
- Why user satisfaction is its own criterion
- The problem of relevance
- Bringing the methods together
Why evaluation matters
Evaluation is the systematic process of determining a system’s worth against a set of standards. For an ISAR system, the goal is straightforward to state but difficult to achieve: retrieve all the relevant documents for a query, and only the relevant ones. F. W. Lancaster framed the justification for evaluation around three questions: how well the system satisfies its objectives, how efficiently it does so, and whether it justifies its existence.
There are two broad viewpoints from which evaluation is conducted. A managerial-oriented evaluation looks at the system from the administrator’s perspective, focusing on cost, efficiency, and whether the system earns its keep. A user-oriented evaluation judges the system by how well it serves the people actually searching it. Both matter, and a complete evaluation usually combines them. The two big factors that emerge are performance and cost, and most of the difficulty lies in measuring performance reliably.
The foundation: the Cranfield experiments
Modern ISAR evaluation traces back to the Cranfield experiments, a series of studies run by Cyril W. Cleverdon in the 1960s at the College of Aeronautics in the United Kingdom. These experiments set out to measure the relative effectiveness of existing indexing systems and to develop a repeatable method for testing new ones. It is widely accepted that the formal, empirical evaluation of retrieval systems began here.
Cleverdon’s work established three components that together form a test collection, still the backbone of evaluation today: a collection of documents, a set of user queries, and a set of relevance judgments specifying which documents are correct for each query. With these three pieces in place, a system’s output can be checked against a known answer key, and its success measured. The first Cranfield collection used 1,400 technical documents in aeronautical engineering, a deliberately focused domain that made testing manageable.
The lasting contribution of Cranfield was the pair of measures it produced: recall and precision. Almost every discussion of ISAR evaluation since then has revolved around these two ideas.
System coverage and the criteria of effectiveness
Before measuring precision and recall, we need a fuller picture of what “good” means for an ISAR system. Several researchers proposed criteria, and they overlap considerably. B. C. Vickery in 1970 identified six criteria grouped into two sets. The first set relates to the availability of information and covers coverage, recall, and response. The second set relates to the selectivity of output and covers precision, the effort required by the user, and the form of presentation.
A year later, Lancaster proposed five evaluation criteria: the coverage of the system, its ability to retrieve wanted items (recall), its ability to avoid unwanted items (precision), its response time, and the amount of effort the user must put in.
What coverage means
Coverage refers to the extent to which the system includes the documents that exist on a given subject. A database on Indian agriculture that omits half the relevant research reports will fail its users no matter how clever its search algorithm is, simply because the right documents are not there to be found. Coverage is a policy decision made by system managers in advance, and it can usually be estimated by comparing the collection against published indexes and reference works in the field.
The trickier criteria
According to the SIGIR retrieval evaluation literature, most of these criteria are relatively easy to measure. Response time is directly measurable, the form of presentation is easy to describe, and user effort can be captured in part through the time spent formulating queries and examining results. The two that cause real difficulty, both in theory and in practice, are recall and precision, mainly because they depend on the slippery concept of relevance.
Recall and precision: the core measures
These two ratios sit at the heart of ISAR evaluation. They both depend on classifying documents as relevant or non-relevant to a query, and then counting.
Recall measures the system’s ability to find all the relevant documents in the collection. It is calculated as the number of relevant items retrieved divided by the total number of relevant items in the collection, expressed as a percentage. If a collection holds 50 documents relevant to your query and the system returns 40 of them, recall is 80 percent.
Precision measures the system’s ability to avoid retrieving non-relevant documents. It is calculated as the number of relevant items retrieved divided by the total number of items retrieved. If the system returns 100 documents in total and only 40 of those are relevant, precision is 40 percent.
A useful way to picture this is the four-way split that Lancaster described. For any search, the documents fall into four groups: relevant documents that were retrieved (the hits), non-relevant documents that were retrieved (the noise), relevant documents that were missed, and non-relevant documents that were correctly rejected. Recall focuses on minimising the misses; precision focuses on minimising the noise.
The recall and precision trade-off
An ideal system would achieve 100 percent recall and 100 percent precision at the same time, returning every relevant document and nothing else. In practice this is not achievable, because an increase in recall usually causes a decrease in precision. The two measures tend to move in opposite directions.
The reason is intuitive once you see it. To raise recall, you broaden the search and cast a wider net, which pulls in more relevant documents but also drags in more irrelevant ones, lowering precision. To raise precision, you tighten the search and demand stronger matches, which keeps out the noise but also risks excluding some genuinely relevant documents, lowering recall. Cleverdon’s own work established this inverse relationship as a fundamental property of retrieval systems.
This trade-off is often visualised with a recall-precision graph, which plots precision values at a series of recall points, typically from 0.1 up to 1.0. The curve shows how precision falls as the system is pushed to retrieve more of the relevant set.
Different users want different balances
There is no single correct point on the curve, because the right balance depends on what the user is trying to do. A researcher preparing a state-of-the-art review wants every relevant item on the topic and will tolerate sifting through some noise, so they favour high recall. A student who just needs a few good references on a topic prefers a short, clean list and favours high precision. A well-designed ISAR system lets the search be tuned toward one end or the other depending on the need.
The F-measure
Because recall and precision pull against each other, evaluators often want a single number that combines both. The F-measure, or F1 score, is defined as the harmonic mean of precision and recall. The harmonic mean is used rather than a simple average because it stays low unless both values are reasonably high, so a system cannot score well on the F-measure by maximising one measure while neglecting the other. It rewards genuine balance.
User experience and response time
Precision and recall describe the quality of the results, but they say nothing about how it feels to use the system. Two further criteria address this directly.
Response time is the speed at which the system returns results after a query is submitted. It is one of the easiest criteria to measure objectively, and it has an outsized effect on satisfaction. A search that takes too long encourages users to abandon it or to settle for a poorer query. User effort covers everything the user must do to get a useful result: formulating the query, interacting with the interface, refining searches, and examining the output. The lower this effort, the better the experience.
The form of presentation also belongs here. Even accurate results lose value if they are displayed in a confusing or cluttered way. How results are ranked, summarised, and laid out shapes whether a user can quickly judge what is worth opening.
Why user satisfaction is its own criterion
The ultimate aim of an ISAR system is to satisfy the information need of its users, so user satisfaction has come to be treated as a criterion of system success in its own right. Interestingly, research has found that satisfaction does not always track search success precisely. In some studies of database searching, users reported being pleased with a system regardless of how many useful citations they actually retrieved, with factors like comfort with the technology influencing their judgment. This is a caution for evaluators: a system can produce excellent precision and recall scores and still leave users unhappy, or score modestly yet feel satisfying. A complete evaluation has to look at both the numbers and the human response.
The problem of relevance
Every measure built on relevant versus non-relevant documents rests on someone deciding what counts as relevant, and this judgment is deeply subjective. Both recall and precision depend largely on the relevance judgments of the user, and different people, or the same person at different times, may judge the same document differently. The original Cranfield design simplified this by treating relevance as a binary, yes-or-no property and by relying on expert judgment. Real users experience relevance as a matter of degree. This is why modern systems increasingly use graded relevance and metrics such as Mean Average Precision and Normalised Discounted Cumulative Gain, which account for the ranked order in which results are presented rather than treating the output as an unordered set.
Bringing the methods together
A thorough evaluation of an ISAR system follows a structured process: define the purpose and scope of the study, assemble a test collection of documents, queries, and relevance judgments, run the queries, and then measure the results against the criteria discussed above. The quantitative side gives precision, recall, F-measure, and response time. The qualitative side captures user satisfaction, effort, and the suitability of the presentation. Neither side alone is enough. A system that wins on precision and recall but feels slow and clumsy will fail its users, while one that feels pleasant but returns poor results fails just as surely. Good evaluation holds both in view and uses each to check the other.
What do you think? If you were evaluating the search system in your college library or a favourite database, would you lean toward high recall or high precision, and why? And when a system scores well on precision and recall but users still feel dissatisfied, where do you think the real problem lies?
References
- https://www.slideshare.net/slideshow/ppt-evaluation-of-information-retrieval-system/27088624
- https://en.wikipedia.org/wiki/Cranfield_experiments
- https://wiki.ubc.ca/Course:LIBR557/2020WT2/cranfield_experiments
- https://en.wikipedia.org/wiki/Cyril_Cleverdon
- https://oercommons.org/courseware/lesson/123391/overview
- https://oercommons.org/courseware/lesson/138450/overview
- https://sigir.org/files/museum/introduction_to_modern_information_retrieval/chapter_5.pdf
- https://upload.wikimedia.org/wikipedia/commons/1/17/Evaluation_of_information_retrieval_system_purpose_and_retrieval.pdf
- https://dl.acm.org/doi/10.1145/122860.122861
- https://nlp.stanford.edu/IR-book/pdf/08eval.pdf
- https://www.researchgate.net/publication/220434706_A_Review_of_Factors_Influencing_User_Satisfaction_in_Information_Retrieval
- https://ebooks.inflibnet.ac.in/lisp7/chapter/evaluation-and-measurement-of-information-retrieval-system/

Leave a Reply