Every indexing system promises to connect a searcher with the right document at the right time. But how do we know whether it actually delivers on that promise? Asking “is this system good?” is too vague to be useful. A library committee, a database manager, and an end user will each define “good” differently. This is why evaluation in information science is structured into distinct levels, each answering a sharper question than the last. At the base, we ask whether a system works. Then we ask whether it works at a reasonable price. Finally, we ask whether it is worth having at all. These three levels-system effectiveness, cost effectiveness, and cost-benefit evaluation-form a logical ladder. Understanding where each level sits and what it measures helps you judge any indexing system, from a small institutional catalogue to a national bibliographic database.

Table of Contents

Why levels of evaluation exist

The framework of layered evaluation owes much to the work of F.W. Lancaster, whose writing in the 1970s shaped how libraries assess retrieval systems. Lancaster argued that an information retrieval system can be judged by considering three connected issues: how well it satisfies its objectives, how efficiently it satisfies them, and whether it justifies its existence at all. Each of those questions maps onto a deeper level of analysis. The structured approach to evaluation documented in Indian LIS coursework follows this same logic, treating performance and cost as the two basic factors any thorough study must address.

The reason these levels matter is practical. A system can be highly effective and still be a poor choice if it drains a budget. Another system might be cheap but so weak that users abandon it. Evaluating at only one level produces a distorted picture. By moving through all three, an evaluator builds a complete case for keeping, modifying, or replacing a system.

System effectiveness: does it retrieve well?

The first level asks the most fundamental question. Does the system retrieve what users need, and does it leave out what they do not need? This is the domain of performance measurement, and it is where the famous metrics of recall and precision live.

Recall and precision

Recall measures completeness. It is the proportion of all relevant documents in the collection that the system actually retrieves for a query. Precision measures purity. It is the proportion of retrieved documents that are genuinely relevant. A system that pulls every relevant record but buries it under hundreds of irrelevant hits has high recall and low precision. A system that returns only a handful of perfect matches while missing many others has high precision and low recall.

These two measures pull against each other. The landmark Cranfield tests led by Cyril Cleverdon established that there is an inevitable inverse relationship between recall and precision, and that systems typically operate at a recall ratio between 70% and 90% while precision sits much lower, often between 8% and 20%. The same project found that there is an optimum level of exhaustivity in indexing-indexing beyond a certain point does little to improve recall but seriously weakens precision.

Exhaustivity and specificity

Two indexing decisions directly shape these outcomes. Exhaustivity refers to how many concepts in a document the indexer chooses to represent. Specificity refers to how precisely the index terms describe those concepts. High exhaustivity tends to raise recall, while high specificity tends to raise precision. An evaluation at the effectiveness level often traces a system’s recall or precision failures back to choices made about these two variables.

Beyond the two core ratios

Effectiveness is not only about recall and precision. Lancaster proposed a broader set of criteria, including the coverage of the collection, the response time, the form in which output is presented, and the effort the user must invest. The user-oriented criteria for retrieval evaluation note that several of these are easy to measure-response time is directly observable, and user effort can be partly captured through the time spent formulating a query and examining results. Recall and precision remain the hardest to measure because they depend on knowing how many relevant documents exist in the first place.

This level also accommodates the user’s perspective. A search for “best smartphone” that returns thousands of loosely related results scores well on recall but frustrates a user who wanted a quick answer. Effectiveness, then, is not a single number but a balance tuned to what users actually want. A researcher preparing a state-of-the-art review needs high recall and will tolerate noise. A user checking a single fact prefers precision and dislikes clutter.

Cost effectiveness: does it retrieve well at a reasonable price?

A system can be effective and still be impractical. The second level introduces money into the equation. Cost-effectiveness analysis relates the performance of a system to the resources consumed in achieving it. It does not try to convert benefits into rupees. Instead, it compares the cost of competing options against a measurable unit of output, such as the cost per search or the cost per relevant document retrieved.

This distinction is important. Cost-effectiveness analysis is most useful precisely when analysts cannot place a monetary value on the benefits. It expresses cost in money but measures effectiveness in physical units, and because the two are incommensurable, the result is a ratio rather than a single combined figure. In information work, that ratio might be the average cost of running one search at a given monthly search volume.

What counts as a cost

Lancaster identified several components that an honest costing must include. These go well beyond the price of the software. They cover the cost of conducting a search, the user’s effort, the time needed to learn the system, the cost of retrieving the documents themselves, and the time a user spends from submitting a query to receiving an answer. Many of these are indirect and difficult to calculate, which is why economic assessment of retrieval systems is often more complex than measuring effectiveness.

A worked illustration

A concrete study makes this tangible. An evaluation of the PROBE retrieval program searching the ERIC database measured both performance and cost together. It found that descriptor-based searches achieved an average relevance ratio of about 55%, while natural-language searches reached only 42%. On the cost side, a descriptor search cost roughly $14.42 when 150 searches ran each month but dropped to $9.18 per search when 300 searches ran each month. The study concluded that the service was both satisfying users and cost-effective, because at sufficient volume it could become self-supporting at a modest fee.

Two lessons emerge. First, cost per search falls as usage rises, so the cost-effectiveness of a system depends heavily on how intensively it is used. Second, a more effective search method (descriptors) can also be a more economical one, which means effectiveness and cost are not always in conflict. There are, however, genuine trade-offs between input costs-the effort poured into careful indexing-and output costs-the effort users spend wading through poor results.

Cost-benefit evaluation: is the system worth having at all?

The final and most demanding level asks whether the system justifies its existence. Cost-benefit evaluation weighs the total cost of a system against the total value it delivers to its users and to the parent organisation. Where cost-effectiveness stops at performance per rupee, cost-benefit analysis tries to express benefits in the same monetary terms as costs so that the two can be directly compared.

Lancaster drew a clear line between the two in his foundational work. His analysis of cost-effectiveness and cost-benefit approaches distinguished between them and explored the relationship between costs, performance, and benefits, including the trade-offs between input and output costs. Cost-effectiveness looks inward at how efficiently the system performs. Cost-benefit looks outward at whether the system is a worthwhile investment compared to doing nothing or doing something else entirely.

The challenge of valuing benefits

The difficulty is obvious. How do you put a price on a researcher finding a crucial citation, a clinician retrieving the right treatment guideline, or a student locating a vital source? These benefits are real but resist monetisation. This is exactly why cost-effectiveness analysis is often used as a substitute or a first step. As broader economic evaluation literature explains, cost-effectiveness analysis frees the analyst from expressing all benefits in monetary terms and is frequently the opening stage of a fuller cost-benefit study.

The same problem appears across disciplines. In health economics, analysts often prefer cost-effectiveness ratios precisely because they sidestep the ethical and analytical difficulty of assigning a monetary value to outcomes such as a year of life. Indexing systems face an analogous problem: the value of well-organised knowledge is diffuse and long-term, spreading across every successful search a system enables over its lifetime.

Putting the three levels together

The three levels are not rivals but stages of one inquiry. Effectiveness tells you whether the engine runs. Cost-effectiveness tells you the fuel economy. Cost-benefit tells you whether the journey was worth taking. A complete evaluation moves from the technical core outward to the organisational and economic context. A system that is effective, affordable to operate, and demonstrably valuable to its community has cleared all three bars and earns its place. One that fails at any level signals where attention-and possibly redesign-is needed.

For anyone managing an indexing or retrieval system, the practical takeaway is to never settle for a single verdict. Measure performance with recall, precision, and the wider set of user criteria. Cost that performance honestly, including the hidden burden on users. Then step back and ask the hardest question of all: does the benefit to the people who use this system outweigh everything it costs to keep running?

What do you think? If you had to evaluate the search system at your own institution, which level would reveal the most uncomfortable truths-the effectiveness numbers, the real cost per search, or the question of whether its benefits truly justify the investment? And in an age of free web search, how should libraries make the cost-benefit case for maintaining specialised indexing systems at all?

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

References
  1. https://ebooks.inflibnet.ac.in/lisp7/chapter/evaluation-and-measurement-of-information-retrieval-system/
  2. https://www-nlpir.nist.gov/projects/irlib/pubs/cranv1p1/cranv1p1_text/01_002.txt
  3. https://sigir.org/files/museum/introduction_to_modern_information_retrieval/chapter_5.pdf
  4. https://www.betterevaluation.org/methods-approaches/methods/cost-effectiveness-analysis
  5. https://eric.ed.gov/?id=ED121261
  6. https://asistdl.onlinelibrary.wiley.com/doi/abs/10.1002/asi.4630220104
  7. https://www.sciencedirect.com/topics/economics-econometrics-and-finance/cost-effectiveness-analysis
  8. https://www.ncbi.nlm.nih.gov/books/NBK10253/

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

Information Processing & Retrieval

1 Intellectual Organisation of Information

  1. Intellectual Organisation of Information
  2. Meaning of Intellectual Organisation of Information
  3. Why IOI is Necessary?
  4. IOI in Indexing Systems
  5. IOI and Indexing Languages
  6. IOI in User Services
  7. IOI and Content Analysis
  8. Information Retrieval Systems – Changing Environment
  9. Future Trends

2 Indexing Languages–Part I – Concepts and Types, Subject Headings Lists and Thesauri

  1. Indexing and its Types
  2. Indexing Language
  3. Vocabulary Control
  4. Classification Schemes
  5. Subject Headings Lists
  6. Thesaurus
  7. Thesaurofacet
  8. Classaurus
  9. Sears List of Subject Headings
  10. Library of Congress List of Subject Headings

3 Indexing Languages–Part II- Classification Schemes

  1. Dewey Decimal Classification (DDC) Scheme
  2. Universal Decimal Classification (UDC) Scheme
  3. Library of Congress Classification (LCC) Scheme
  4. Colon Classification (CC) Scheme
  5. Bibliographic Classification (BC) Scheme
  6. Library Bibliographical Classification (BBK) Scheme
  7. Broad System of Ordering (BSO) Scheme
  8. Special Classification Systems

4 Indexing Systems and Techniques

  1. Indexing Principles and Process
  2. Pre-Coordinate Indexing Systems
  3. Post-Coordinate Indexing Systems
  4. Automatic Indexing
  5. Non-Conventional Indexing: Citation Indexing
  6. Web Indexing

5 Evaluation of Indexing Systems

  1. Purpose of Evaluation
  2. Levels of Evaluation
  3. Evaluation Criteria
  4. Recall and Precision
  5. Other Performance Measures
  6. Relevance
  7. Evaluation Methodology
  8. Evaluation Experiments

6 Principles and Evolution of Bibliographic Description

  1. Bibliographic Description: An Overview
  2. Scope and Objectives of Bibliographic Description
  3. Evolution of Bibliographic Description
  4. Ranganathan’s Principles
  5. ISBDs
  6. Bibliographic Formats
  7. Electronic Resource Description
  8. Models of Bibliographic Description
  9. Bibliographic Description: Entities, Attributes and Relationships

7 Rules for Bibliographic Description

  1. Bibliographic Description: Its Origin
  2. Development of Anglo-American Code
  3. The International Standard Bibliographic Description (ISBD)
  4. Impact of ISBD on Catalogue Codes
  5. Bibliographic Description for Non-Print Materials
  6. Guidelines for Bibliographic Description of Electronic Resources
  7. Guidelines for Bibliographic Description of Internet Resources
  8. Rules for Description of Electronic Resources in AACR2 Revision 2002

8 Standards for Bibliographic Record Format

  1. International Standard Bibliographic Description (ISBD)
  2. MARC Format
  3. UNIMARC
  4. Common Communication Format (CCF)
  5. Indian Standard

9 Metadata- MARC21-856 Field, Dublin Core, TEI

  1. MARC21 – 856 Field
  2. Dublin Core Metadata Initiative (DCMI)
  3. Text Encoding Initiative (TEI)
  4. Procedure of Electronic Resource Description

10 Norms and Guidelines for Content Development

  1. Introduction
  2. Needs and Guidelines
  3. Standards Related to Electronic Content
  4. W3C Recommendations
  5. Electronic Text Encoding and Interchange
  6. Dynamic Content

11 Introduction to HTML and XML

  1. World Wide Web and Markup Languages
  2. Standard Generalized Markup Language (SGML)
  3. HyperText Markup Language (HTML)
  4. Basic HTML Tags
  5. Linking
  6. URLs
  7. HTML and the Browser
  8. eXtensible Markup Language (XML)
  9. XML Syntax and Semantic Tags
  10. Document Type Definition (DTD)
  11. Implications of XML in Library and Information Activities

12 Web-based Content Development

  1. What can be done with World Wide Web?
  2. Hypertext, Hyperlink, and Hypermedia
  3. Hypertext Markup Language (HTML)
  4. Introduction to Dynamic HTML
  5. Web Interface to Database Linking
  6. Introduction to XML
  7. XML Document Design
  8. Multimedia Web Resources
  9. Web Servers
  10. Website Hosting
  11. Tools for Web Page Designing

13 Multilingual Content Development (Using Unicode)

  1. Character Representation in Computer
  2. American Standard Code for Information Interchange (ASCII)
  3. Indian Scenario and Indian Standard Code for Information Interchange (ISCII)
  4. UNICODE
  5. Web Content Development Through UNICODE
  6. Applications of UNICODE
  7. Applying UNICODE to the Libraries
  8. Problems Associated with UNICODE

14 ISAR Systems- Objectives, Types, Operations and Design

  1. Users and Their Information Needs
  2. Objectives of ISAR Systems
  3. Types of ISAR Systems
  4. Design of ISAR Systems
  5. Evaluation of ISAR Systems

15 Compatibility of ISAR Systems

  1. Need for Compatibility Among ISAR Systems
  2. Scope of Compatibility in ISAR Systems
  3. Areas of Compatibilities in ISAR Systems
  4. Principal Issues of Compatibility in ISAR Systems
  5. Compatibility of Online IR Systems
  6. Approaches Towards Compatibility in ISAR
  7. Quality Control and Compatibility

16 Intelligent Information Retrieval Systems

  1. Introduction
  2. Expert Systems
  3. Expert Systems for Information Processing and Retrieval
  4. Components of Expert Systems
  5. Knowledge Representation
  6. Knowledge Engineering
  7. Artificial Intelligence Based Decision Support Systems (DSS)
  8. Pattern Recognition

17 Information Retrieval Processes and Techniques

  1. Information Retrieval Systems
  2. Databases
  3. Information Retrieval Systems: Purpose, Components, and Functions
  4. Indexing and Information Representation
  5. Vocabulary Control
  6. Searching
  7. Information Seeking and User Interfaces
  8. Web Information Retrieval Systems
  9. Intelligent Information Retrieval

18 Information Retrieval Models and Their Applications

  1. Information Retrieval
  2. Information Retrieval Techniques
  3. Models Based on Input/Output
  4. Models Based on Theories and Tools

19 Search Strategies, Processes and Techinques

  1. Search File – An Essential Component
  2. Search Strategies and Pre-requisites
  3. Search Techniques
  4. The Information Search Process
  5. Online Searching
  6. How the Search Engines Work
  7. Common Search and Retrieval Features of Web Search Engines