Building an information storage and retrieval (ISAR) system is only half the job. The harder question comes after it goes live: does it actually work? An ISAR system that holds millions of records is useless if it buries relevant results under irrelevant ones, takes too long to respond, or frustrates the people using it. Evaluation is how we answer that question with evidence rather than guesswork. It tells us whether a system meets its objectives, how efficiently it does so, and whether it deserves the money and effort spent on it. This post walks through the main criteria and methods used to judge how well an ISAR system performs.

Table of Contents

Why evaluation matters

Evaluation is the systematic process of determining a system’s worth against a set of standards. For an ISAR system, the goal is straightforward to state but difficult to achieve: retrieve all the relevant documents for a query, and only the relevant ones. F. W. Lancaster framed the justification for evaluation around three questions: how well the system satisfies its objectives, how efficiently it does so, and whether it justifies its existence.

There are two broad viewpoints from which evaluation is conducted. A managerial-oriented evaluation looks at the system from the administrator’s perspective, focusing on cost, efficiency, and whether the system earns its keep. A user-oriented evaluation judges the system by how well it serves the people actually searching it. Both matter, and a complete evaluation usually combines them. The two big factors that emerge are performance and cost, and most of the difficulty lies in measuring performance reliably.

The foundation: the Cranfield experiments

Modern ISAR evaluation traces back to the Cranfield experiments, a series of studies run by Cyril W. Cleverdon in the 1960s at the College of Aeronautics in the United Kingdom. These experiments set out to measure the relative effectiveness of existing indexing systems and to develop a repeatable method for testing new ones. It is widely accepted that the formal, empirical evaluation of retrieval systems began here.

Cleverdon’s work established three components that together form a test collection, still the backbone of evaluation today: a collection of documents, a set of user queries, and a set of relevance judgments specifying which documents are correct for each query. With these three pieces in place, a system’s output can be checked against a known answer key, and its success measured. The first Cranfield collection used 1,400 technical documents in aeronautical engineering, a deliberately focused domain that made testing manageable.

The lasting contribution of Cranfield was the pair of measures it produced: recall and precision. Almost every discussion of ISAR evaluation since then has revolved around these two ideas.

System coverage and the criteria of effectiveness

Before measuring precision and recall, we need a fuller picture of what “good” means for an ISAR system. Several researchers proposed criteria, and they overlap considerably. B. C. Vickery in 1970 identified six criteria grouped into two sets. The first set relates to the availability of information and covers coverage, recall, and response. The second set relates to the selectivity of output and covers precision, the effort required by the user, and the form of presentation.

A year later, Lancaster proposed five evaluation criteria: the coverage of the system, its ability to retrieve wanted items (recall), its ability to avoid unwanted items (precision), its response time, and the amount of effort the user must put in.

What coverage means

Coverage refers to the extent to which the system includes the documents that exist on a given subject. A database on Indian agriculture that omits half the relevant research reports will fail its users no matter how clever its search algorithm is, simply because the right documents are not there to be found. Coverage is a policy decision made by system managers in advance, and it can usually be estimated by comparing the collection against published indexes and reference works in the field.

The trickier criteria

According to the SIGIR retrieval evaluation literature, most of these criteria are relatively easy to measure. Response time is directly measurable, the form of presentation is easy to describe, and user effort can be captured in part through the time spent formulating queries and examining results. The two that cause real difficulty, both in theory and in practice, are recall and precision, mainly because they depend on the slippery concept of relevance.

Recall and precision: the core measures

These two ratios sit at the heart of ISAR evaluation. They both depend on classifying documents as relevant or non-relevant to a query, and then counting.

Recall measures the system’s ability to find all the relevant documents in the collection. It is calculated as the number of relevant items retrieved divided by the total number of relevant items in the collection, expressed as a percentage. If a collection holds 50 documents relevant to your query and the system returns 40 of them, recall is 80 percent.

Precision measures the system’s ability to avoid retrieving non-relevant documents. It is calculated as the number of relevant items retrieved divided by the total number of items retrieved. If the system returns 100 documents in total and only 40 of those are relevant, precision is 40 percent.

A useful way to picture this is the four-way split that Lancaster described. For any search, the documents fall into four groups: relevant documents that were retrieved (the hits), non-relevant documents that were retrieved (the noise), relevant documents that were missed, and non-relevant documents that were correctly rejected. Recall focuses on minimising the misses; precision focuses on minimising the noise.

The recall and precision trade-off

An ideal system would achieve 100 percent recall and 100 percent precision at the same time, returning every relevant document and nothing else. In practice this is not achievable, because an increase in recall usually causes a decrease in precision. The two measures tend to move in opposite directions.

The reason is intuitive once you see it. To raise recall, you broaden the search and cast a wider net, which pulls in more relevant documents but also drags in more irrelevant ones, lowering precision. To raise precision, you tighten the search and demand stronger matches, which keeps out the noise but also risks excluding some genuinely relevant documents, lowering recall. Cleverdon’s own work established this inverse relationship as a fundamental property of retrieval systems.

This trade-off is often visualised with a recall-precision graph, which plots precision values at a series of recall points, typically from 0.1 up to 1.0. The curve shows how precision falls as the system is pushed to retrieve more of the relevant set.

Different users want different balances

There is no single correct point on the curve, because the right balance depends on what the user is trying to do. A researcher preparing a state-of-the-art review wants every relevant item on the topic and will tolerate sifting through some noise, so they favour high recall. A student who just needs a few good references on a topic prefers a short, clean list and favours high precision. A well-designed ISAR system lets the search be tuned toward one end or the other depending on the need.

The F-measure

Because recall and precision pull against each other, evaluators often want a single number that combines both. The F-measure, or F1 score, is defined as the harmonic mean of precision and recall. The harmonic mean is used rather than a simple average because it stays low unless both values are reasonably high, so a system cannot score well on the F-measure by maximising one measure while neglecting the other. It rewards genuine balance.

User experience and response time

Precision and recall describe the quality of the results, but they say nothing about how it feels to use the system. Two further criteria address this directly.

Response time is the speed at which the system returns results after a query is submitted. It is one of the easiest criteria to measure objectively, and it has an outsized effect on satisfaction. A search that takes too long encourages users to abandon it or to settle for a poorer query. User effort covers everything the user must do to get a useful result: formulating the query, interacting with the interface, refining searches, and examining the output. The lower this effort, the better the experience.

The form of presentation also belongs here. Even accurate results lose value if they are displayed in a confusing or cluttered way. How results are ranked, summarised, and laid out shapes whether a user can quickly judge what is worth opening.

Why user satisfaction is its own criterion

The ultimate aim of an ISAR system is to satisfy the information need of its users, so user satisfaction has come to be treated as a criterion of system success in its own right. Interestingly, research has found that satisfaction does not always track search success precisely. In some studies of database searching, users reported being pleased with a system regardless of how many useful citations they actually retrieved, with factors like comfort with the technology influencing their judgment. This is a caution for evaluators: a system can produce excellent precision and recall scores and still leave users unhappy, or score modestly yet feel satisfying. A complete evaluation has to look at both the numbers and the human response.

The problem of relevance

Every measure built on relevant versus non-relevant documents rests on someone deciding what counts as relevant, and this judgment is deeply subjective. Both recall and precision depend largely on the relevance judgments of the user, and different people, or the same person at different times, may judge the same document differently. The original Cranfield design simplified this by treating relevance as a binary, yes-or-no property and by relying on expert judgment. Real users experience relevance as a matter of degree. This is why modern systems increasingly use graded relevance and metrics such as Mean Average Precision and Normalised Discounted Cumulative Gain, which account for the ranked order in which results are presented rather than treating the output as an unordered set.

Bringing the methods together

A thorough evaluation of an ISAR system follows a structured process: define the purpose and scope of the study, assemble a test collection of documents, queries, and relevance judgments, run the queries, and then measure the results against the criteria discussed above. The quantitative side gives precision, recall, F-measure, and response time. The qualitative side captures user satisfaction, effort, and the suitability of the presentation. Neither side alone is enough. A system that wins on precision and recall but feels slow and clumsy will fail its users, while one that feels pleasant but returns poor results fails just as surely. Good evaluation holds both in view and uses each to check the other.

What do you think? If you were evaluating the search system in your college library or a favourite database, would you lean toward high recall or high precision, and why? And when a system scores well on precision and recall but users still feel dissatisfied, where do you think the real problem lies?

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

References
  1. https://www.slideshare.net/slideshow/ppt-evaluation-of-information-retrieval-system/27088624
  2. https://en.wikipedia.org/wiki/Cranfield_experiments
  3. https://wiki.ubc.ca/Course:LIBR557/2020WT2/cranfield_experiments
  4. https://en.wikipedia.org/wiki/Cyril_Cleverdon
  5. https://oercommons.org/courseware/lesson/123391/overview
  6. https://oercommons.org/courseware/lesson/138450/overview
  7. https://sigir.org/files/museum/introduction_to_modern_information_retrieval/chapter_5.pdf
  8. https://upload.wikimedia.org/wikipedia/commons/1/17/Evaluation_of_information_retrieval_system_purpose_and_retrieval.pdf
  9. https://dl.acm.org/doi/10.1145/122860.122861
  10. https://nlp.stanford.edu/IR-book/pdf/08eval.pdf
  11. https://www.researchgate.net/publication/220434706_A_Review_of_Factors_Influencing_User_Satisfaction_in_Information_Retrieval
  12. https://ebooks.inflibnet.ac.in/lisp7/chapter/evaluation-and-measurement-of-information-retrieval-system/

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

Information Processing & Retrieval

1 Intellectual Organisation of Information

  1. Intellectual Organisation of Information
  2. Meaning of Intellectual Organisation of Information
  3. Why IOI is Necessary?
  4. IOI in Indexing Systems
  5. IOI and Indexing Languages
  6. IOI in User Services
  7. IOI and Content Analysis
  8. Information Retrieval Systems – Changing Environment
  9. Future Trends

2 Indexing Languages–Part I – Concepts and Types, Subject Headings Lists and Thesauri

  1. Indexing and its Types
  2. Indexing Language
  3. Vocabulary Control
  4. Classification Schemes
  5. Subject Headings Lists
  6. Thesaurus
  7. Thesaurofacet
  8. Classaurus
  9. Sears List of Subject Headings
  10. Library of Congress List of Subject Headings

3 Indexing Languages–Part II- Classification Schemes

  1. Dewey Decimal Classification (DDC) Scheme
  2. Universal Decimal Classification (UDC) Scheme
  3. Library of Congress Classification (LCC) Scheme
  4. Colon Classification (CC) Scheme
  5. Bibliographic Classification (BC) Scheme
  6. Library Bibliographical Classification (BBK) Scheme
  7. Broad System of Ordering (BSO) Scheme
  8. Special Classification Systems

4 Indexing Systems and Techniques

  1. Indexing Principles and Process
  2. Pre-Coordinate Indexing Systems
  3. Post-Coordinate Indexing Systems
  4. Automatic Indexing
  5. Non-Conventional Indexing: Citation Indexing
  6. Web Indexing

5 Evaluation of Indexing Systems

  1. Purpose of Evaluation
  2. Levels of Evaluation
  3. Evaluation Criteria
  4. Recall and Precision
  5. Other Performance Measures
  6. Relevance
  7. Evaluation Methodology
  8. Evaluation Experiments

6 Principles and Evolution of Bibliographic Description

  1. Bibliographic Description: An Overview
  2. Scope and Objectives of Bibliographic Description
  3. Evolution of Bibliographic Description
  4. Ranganathan’s Principles
  5. ISBDs
  6. Bibliographic Formats
  7. Electronic Resource Description
  8. Models of Bibliographic Description
  9. Bibliographic Description: Entities, Attributes and Relationships

7 Rules for Bibliographic Description

  1. Bibliographic Description: Its Origin
  2. Development of Anglo-American Code
  3. The International Standard Bibliographic Description (ISBD)
  4. Impact of ISBD on Catalogue Codes
  5. Bibliographic Description for Non-Print Materials
  6. Guidelines for Bibliographic Description of Electronic Resources
  7. Guidelines for Bibliographic Description of Internet Resources
  8. Rules for Description of Electronic Resources in AACR2 Revision 2002

8 Standards for Bibliographic Record Format

  1. International Standard Bibliographic Description (ISBD)
  2. MARC Format
  3. UNIMARC
  4. Common Communication Format (CCF)
  5. Indian Standard

9 Metadata- MARC21-856 Field, Dublin Core, TEI

  1. MARC21 – 856 Field
  2. Dublin Core Metadata Initiative (DCMI)
  3. Text Encoding Initiative (TEI)
  4. Procedure of Electronic Resource Description

10 Norms and Guidelines for Content Development

  1. Introduction
  2. Needs and Guidelines
  3. Standards Related to Electronic Content
  4. W3C Recommendations
  5. Electronic Text Encoding and Interchange
  6. Dynamic Content

11 Introduction to HTML and XML

  1. World Wide Web and Markup Languages
  2. Standard Generalized Markup Language (SGML)
  3. HyperText Markup Language (HTML)
  4. Basic HTML Tags
  5. Linking
  6. URLs
  7. HTML and the Browser
  8. eXtensible Markup Language (XML)
  9. XML Syntax and Semantic Tags
  10. Document Type Definition (DTD)
  11. Implications of XML in Library and Information Activities

12 Web-based Content Development

  1. What can be done with World Wide Web?
  2. Hypertext, Hyperlink, and Hypermedia
  3. Hypertext Markup Language (HTML)
  4. Introduction to Dynamic HTML
  5. Web Interface to Database Linking
  6. Introduction to XML
  7. XML Document Design
  8. Multimedia Web Resources
  9. Web Servers
  10. Website Hosting
  11. Tools for Web Page Designing

13 Multilingual Content Development (Using Unicode)

  1. Character Representation in Computer
  2. American Standard Code for Information Interchange (ASCII)
  3. Indian Scenario and Indian Standard Code for Information Interchange (ISCII)
  4. UNICODE
  5. Web Content Development Through UNICODE
  6. Applications of UNICODE
  7. Applying UNICODE to the Libraries
  8. Problems Associated with UNICODE

14 ISAR Systems- Objectives, Types, Operations and Design

  1. Users and Their Information Needs
  2. Objectives of ISAR Systems
  3. Types of ISAR Systems
  4. Design of ISAR Systems
  5. Evaluation of ISAR Systems

15 Compatibility of ISAR Systems

  1. Need for Compatibility Among ISAR Systems
  2. Scope of Compatibility in ISAR Systems
  3. Areas of Compatibilities in ISAR Systems
  4. Principal Issues of Compatibility in ISAR Systems
  5. Compatibility of Online IR Systems
  6. Approaches Towards Compatibility in ISAR
  7. Quality Control and Compatibility

16 Intelligent Information Retrieval Systems

  1. Introduction
  2. Expert Systems
  3. Expert Systems for Information Processing and Retrieval
  4. Components of Expert Systems
  5. Knowledge Representation
  6. Knowledge Engineering
  7. Artificial Intelligence Based Decision Support Systems (DSS)
  8. Pattern Recognition

17 Information Retrieval Processes and Techniques

  1. Information Retrieval Systems
  2. Databases
  3. Information Retrieval Systems: Purpose, Components, and Functions
  4. Indexing and Information Representation
  5. Vocabulary Control
  6. Searching
  7. Information Seeking and User Interfaces
  8. Web Information Retrieval Systems
  9. Intelligent Information Retrieval

18 Information Retrieval Models and Their Applications

  1. Information Retrieval
  2. Information Retrieval Techniques
  3. Models Based on Input/Output
  4. Models Based on Theories and Tools

19 Search Strategies, Processes and Techinques

  1. Search File – An Essential Component
  2. Search Strategies and Pre-requisites
  3. Search Techniques
  4. The Information Search Process
  5. Online Searching
  6. How the Search Engines Work
  7. Common Search and Retrieval Features of Web Search Engines