Building an indexing system is only half the job. The harder question is whether it actually works. When a student types a query into a library catalogue or a database, does the system return the right documents, ignore the wrong ones, and do it quickly without exhausting the searcher? To answer this, information scientists rely on a set of well-defined evaluation criteria. These criteria turn a vague sense of “good” or “bad” performance into something measurable. This post walks through the core yardsticks used to judge indexing systems: recall and precision, the effort demanded from users, and the time taken to respond.

Table of Contents

Why indexing systems need formal evaluation

An indexing system organises documents so they can be found later. But organisation alone is meaningless if retrieval fails. Evaluation tells us how well a system meets its purpose, how efficiently it does so, and whether it justifies its cost. The foundations of this field were laid in England during the late 1950s and 1960s, when Cyril Cleverdon, librarian at the College of Aeronautics in Cranfield, ran a famous series of experiments now known as the Cranfield tests.

The Cranfield experiments established a repeatable method for testing retrieval performance using a fixed collection of documents, a set of test queries, and pre-judged relevant items. The results were surprising for their time. They showed that system performance does not depend on the subject expertise of the indexer, and that simpler alphabetical and uniterm indexes could outperform elaborate faceted classification schemes. Most importantly, the tests identified recall and precision as the two central measures of retrieval effectiveness.

Cleverdon went further and proposed six criteria for evaluating retrieval systems: recall, precision, time lag, user effort, form of output presentation, and coverage of the collection. Later, F.W. Lancaster consolidated these into five widely cited criteria for performance measurement: coverage, recall, precision, response time, and user effort. These criteria remain the backbone of how indexing and retrieval systems are assessed today, from library catalogues to web search engines.

Recall and precision: the two core measures

Recall and precision sit at the heart of retrieval evaluation. Both depend on a single idea: dividing every document in a collection into two groups, those that are relevant to a query and those that are not. Once a search runs, the documents also split into retrieved and not retrieved. Combining these gives four outcomes, which Lancaster arranged in a simple two-by-two table.

In this table, a represents relevant documents that were retrieved (often called hits), b represents non-relevant documents that were retrieved (the noise), c represents relevant documents that were missed, and d represents non-relevant documents correctly left out. These four values let us calculate the two key ratios.

Understanding recall

Recall measures how good the system is at finding everything that matters. It is the proportion of relevant documents that the system actually retrieved, out of all the relevant documents in the collection. In the matrix above, recall equals a divided by (a + c), usually expressed as a percentage.

For example, if a database holds 100 documents relevant to a topic but a search retrieves only 60 of them, recall is 60 percent. A high recall means the system rarely misses useful material. This matters most when completeness is critical. A researcher preparing a literature review or a state-of-the-art report needs to find as many relevant items as possible, so high recall is the priority even if it means wading through extra results.

Understanding precision

Precision measures how good the system is at avoiding the wrong documents. It is the proportion of retrieved documents that are actually relevant, calculated as a divided by (a + b). If a search returns 50 documents and only 40 are relevant, precision is 80 percent.

High precision matters when users want focused, accurate results without sifting through clutter. Someone needing a quick factual answer, or a decision-maker under time pressure, values a small set of clearly relevant hits over a large, diluted list. The Stanford IR textbook notes that precision and recall are set-based measures, computed over the documents returned, and that modern ranked-retrieval systems extend them using tools like precision-recall curves.

The recall-precision trade-off

Here is the central tension every indexer must manage: recall and precision tend to move in opposite directions. Studies tracing back to Cranfield confirmed this inverse relationship. Pushing one up usually drags the other down.

The reason is intuitive. To raise recall, you widen the net, using broader terms and looser matching so that more relevant documents are caught. But a wider net also scoops up irrelevant material, which lowers precision. To raise precision, you tighten the search with narrow, specific terms. This filters out the noise, but it also risks excluding relevant documents that were phrased differently, which lowers recall.

This is why there is no single “best” setting for an indexing system. The right balance depends on who is using it and why. A patent searcher who cannot afford to miss a single prior reference will accept low precision to protect recall. A user looking up a quick fact will happily trade recall for a clean, precise result set. To capture both numbers in a single figure, evaluators sometimes use the F-measure, the harmonic mean of precision and recall.

Recall and precision rarely travel alone. A few companion measures add detail. Fallout is the proportion of non-relevant documents that were retrieved, calculated as b divided by (b + d), and it shows how well a system rejects unwanted material. Generality reflects the density of relevant documents in the collection relative to a query. As collections grow larger, the share of non-relevant items for any given query tends to rise, which makes high precision harder to sustain.

Lancaster’s fifth criterion, coverage, asks a different question: how much of the relevant literature on a subject does the collection contain in the first place? A system can have perfect recall over its own holdings and still fail the user if those holdings are incomplete. Coverage reminds us that retrieval quality depends partly on what was indexed before any search begins.

User effort: the human cost of searching

A system can score well on recall and precision and still frustrate the people using it. This is why user effort is treated as a distinct evaluation criterion. It captures the intellectual and physical work a searcher must put in to get useful answers, from formulating the query to running the search and screening the output.

According to work building on Lancaster’s framework, user effort covers the labour of constructing a query, conducting the search, and reviewing what comes back. Several factors shape it. The accessibility of the system, the availability of guidance from staff, the volume of retrieved items, and the quality of the interface all influence how much work the user must do.

Why effort affects the whole experience

Consider what happens when recall is pushed very high. The system returns hundreds of documents, many only loosely relevant. Recall looks excellent on paper, but the user now has to read and filter all of them. The effort cost is enormous. This is a direct link between the abstract metrics and the lived experience of searching: a result set that is technically comprehensive but practically unusable has shifted the burden onto the person.

Effort also includes the time spent learning the system in the first place. A powerful database with a steep learning curve demands more from a casual user than a simple catalogue with an intuitive interface. Lancaster, when discussing the cost of retrieval systems, explicitly listed learning time and the effort of actual use among the factors that determine a system’s true cost. Good indexing system design therefore tries to deliver strong recall and precision while keeping the user’s workload as low as possible.

Form of output presentation

Closely tied to user effort is the form in which results are presented. The same set of retrieved documents can be far more or less useful depending on how it is displayed. Clear citations, informative snippets, sensible ordering, and the ability to refine a search all reduce the effort needed to act on results. A list of bare document numbers forces far more work than a ranked list with titles, abstracts, and relevance cues. Presentation does not change which documents were found, but it shapes whether the user can actually use them.

Time lag and response: measuring efficiency

The final piece of the evaluation puzzle is speed. Time lag, also called response time, is the interval between the moment a user submits a query and the moment the system presents its response. Cleverdon included it among his six original criteria, and it has only grown more important as users have come to expect near-instant results.

Response time is influenced by several system-level factors. Salton and McGill identified the organisation of documents, the type of query, the location of the information centre, the frequency of incoming queries, and the size of the collection as key determinants. A large collection with complex queries naturally takes longer to search than a small one with simple lookups.

Speed as part of effectiveness, not separate from it

It is tempting to treat response time as a purely technical concern, separate from the quality of results. But the two are connected. A system that takes too long to respond effectively increases user effort, because the searcher must wait, and waiting is itself a cost. In interactive searching, where a user refines a query several times before finding what they need, slow response times multiply with each iteration and quickly degrade the overall experience.

There is also a balance to strike. A system could, in theory, run an exhaustive search to maximise recall, but if that search takes minutes rather than seconds, its practical value drops. Designers therefore weigh thoroughness against speed, just as they weigh recall against precision. Efficient document organisation, good indexing structures, and appropriate hardware all help keep response time low without sacrificing result quality.

Bringing the criteria together

No single measure captures whether an indexing system is good. Recall and precision tell us about the relevance of results but pull in opposite directions. User effort tells us about the human cost of getting those results. Response time tells us about efficiency. Coverage tells us whether the relevant material was even there to be found. A complete evaluation looks at all of them together.

The right priorities depend on the setting. An academic research service may favour high recall and tolerate greater user effort to support exhaustive literature reviews. A public information service may emphasise precision, low effort, and fast response to serve a broad and impatient audience. A specialised professional system may need to balance every criterion at once. Understanding how these measures interact, rather than chasing any one of them in isolation, is what separates a well-designed indexing system from a merely impressive one.

What do you think? If you were designing an indexing system for your own college library, would you optimise for recall, precision, or low user effort, and why? And how would you decide whether a one-second improvement in response time is worth a drop in the relevance of results?

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

References
  1. https://en.wikipedia.org/wiki/Cyril_Cleverdon
  2. https://ebooks.inflibnet.ac.in/lisp7/chapter/evaluation-and-measurement-of-information-retrieval-system/
  3. https://oercommons.org/courseware/lesson/123391/overview
  4. https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-ranked-retrieval-results-1.html
  5. https://www.iro.umontreal.ca/~nie/IFT6255/IR-Evaluation.pdf
  6. https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)
  7. https://www.cs.utexas.edu/~mooney/ir-course/slides_pdfs/Evaluation.pdf
  8. https://irp.cdn-website.com/69a5b730/files/uploaded/22986820.pdf

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

Information Processing & Retrieval

1 Intellectual Organisation of Information

  1. Intellectual Organisation of Information
  2. Meaning of Intellectual Organisation of Information
  3. Why IOI is Necessary?
  4. IOI in Indexing Systems
  5. IOI and Indexing Languages
  6. IOI in User Services
  7. IOI and Content Analysis
  8. Information Retrieval Systems – Changing Environment
  9. Future Trends

2 Indexing Languages–Part I – Concepts and Types, Subject Headings Lists and Thesauri

  1. Indexing and its Types
  2. Indexing Language
  3. Vocabulary Control
  4. Classification Schemes
  5. Subject Headings Lists
  6. Thesaurus
  7. Thesaurofacet
  8. Classaurus
  9. Sears List of Subject Headings
  10. Library of Congress List of Subject Headings

3 Indexing Languages–Part II- Classification Schemes

  1. Dewey Decimal Classification (DDC) Scheme
  2. Universal Decimal Classification (UDC) Scheme
  3. Library of Congress Classification (LCC) Scheme
  4. Colon Classification (CC) Scheme
  5. Bibliographic Classification (BC) Scheme
  6. Library Bibliographical Classification (BBK) Scheme
  7. Broad System of Ordering (BSO) Scheme
  8. Special Classification Systems

4 Indexing Systems and Techniques

  1. Indexing Principles and Process
  2. Pre-Coordinate Indexing Systems
  3. Post-Coordinate Indexing Systems
  4. Automatic Indexing
  5. Non-Conventional Indexing: Citation Indexing
  6. Web Indexing

5 Evaluation of Indexing Systems

  1. Purpose of Evaluation
  2. Levels of Evaluation
  3. Evaluation Criteria
  4. Recall and Precision
  5. Other Performance Measures
  6. Relevance
  7. Evaluation Methodology
  8. Evaluation Experiments

6 Principles and Evolution of Bibliographic Description

  1. Bibliographic Description: An Overview
  2. Scope and Objectives of Bibliographic Description
  3. Evolution of Bibliographic Description
  4. Ranganathan’s Principles
  5. ISBDs
  6. Bibliographic Formats
  7. Electronic Resource Description
  8. Models of Bibliographic Description
  9. Bibliographic Description: Entities, Attributes and Relationships

7 Rules for Bibliographic Description

  1. Bibliographic Description: Its Origin
  2. Development of Anglo-American Code
  3. The International Standard Bibliographic Description (ISBD)
  4. Impact of ISBD on Catalogue Codes
  5. Bibliographic Description for Non-Print Materials
  6. Guidelines for Bibliographic Description of Electronic Resources
  7. Guidelines for Bibliographic Description of Internet Resources
  8. Rules for Description of Electronic Resources in AACR2 Revision 2002

8 Standards for Bibliographic Record Format

  1. International Standard Bibliographic Description (ISBD)
  2. MARC Format
  3. UNIMARC
  4. Common Communication Format (CCF)
  5. Indian Standard

9 Metadata- MARC21-856 Field, Dublin Core, TEI

  1. MARC21 – 856 Field
  2. Dublin Core Metadata Initiative (DCMI)
  3. Text Encoding Initiative (TEI)
  4. Procedure of Electronic Resource Description

10 Norms and Guidelines for Content Development

  1. Introduction
  2. Needs and Guidelines
  3. Standards Related to Electronic Content
  4. W3C Recommendations
  5. Electronic Text Encoding and Interchange
  6. Dynamic Content

11 Introduction to HTML and XML

  1. World Wide Web and Markup Languages
  2. Standard Generalized Markup Language (SGML)
  3. HyperText Markup Language (HTML)
  4. Basic HTML Tags
  5. Linking
  6. URLs
  7. HTML and the Browser
  8. eXtensible Markup Language (XML)
  9. XML Syntax and Semantic Tags
  10. Document Type Definition (DTD)
  11. Implications of XML in Library and Information Activities

12 Web-based Content Development

  1. What can be done with World Wide Web?
  2. Hypertext, Hyperlink, and Hypermedia
  3. Hypertext Markup Language (HTML)
  4. Introduction to Dynamic HTML
  5. Web Interface to Database Linking
  6. Introduction to XML
  7. XML Document Design
  8. Multimedia Web Resources
  9. Web Servers
  10. Website Hosting
  11. Tools for Web Page Designing

13 Multilingual Content Development (Using Unicode)

  1. Character Representation in Computer
  2. American Standard Code for Information Interchange (ASCII)
  3. Indian Scenario and Indian Standard Code for Information Interchange (ISCII)
  4. UNICODE
  5. Web Content Development Through UNICODE
  6. Applications of UNICODE
  7. Applying UNICODE to the Libraries
  8. Problems Associated with UNICODE

14 ISAR Systems- Objectives, Types, Operations and Design

  1. Users and Their Information Needs
  2. Objectives of ISAR Systems
  3. Types of ISAR Systems
  4. Design of ISAR Systems
  5. Evaluation of ISAR Systems

15 Compatibility of ISAR Systems

  1. Need for Compatibility Among ISAR Systems
  2. Scope of Compatibility in ISAR Systems
  3. Areas of Compatibilities in ISAR Systems
  4. Principal Issues of Compatibility in ISAR Systems
  5. Compatibility of Online IR Systems
  6. Approaches Towards Compatibility in ISAR
  7. Quality Control and Compatibility

16 Intelligent Information Retrieval Systems

  1. Introduction
  2. Expert Systems
  3. Expert Systems for Information Processing and Retrieval
  4. Components of Expert Systems
  5. Knowledge Representation
  6. Knowledge Engineering
  7. Artificial Intelligence Based Decision Support Systems (DSS)
  8. Pattern Recognition

17 Information Retrieval Processes and Techniques

  1. Information Retrieval Systems
  2. Databases
  3. Information Retrieval Systems: Purpose, Components, and Functions
  4. Indexing and Information Representation
  5. Vocabulary Control
  6. Searching
  7. Information Seeking and User Interfaces
  8. Web Information Retrieval Systems
  9. Intelligent Information Retrieval

18 Information Retrieval Models and Their Applications

  1. Information Retrieval
  2. Information Retrieval Techniques
  3. Models Based on Input/Output
  4. Models Based on Theories and Tools

19 Search Strategies, Processes and Techinques

  1. Search File – An Essential Component
  2. Search Strategies and Pre-requisites
  3. Search Techniques
  4. The Information Search Process
  5. Online Searching
  6. How the Search Engines Work
  7. Common Search and Retrieval Features of Web Search Engines