Every time you send a message in Hindi, read a Bengali news website, or type an email mixing English and Tamil, a quiet system is working behind the screen to make sure those characters appear correctly. That system is Unicode. Before it existed, sharing text across different computers, operating systems, and languages was a frustrating gamble where words often turned into meaningless symbols. Unicode solved this by giving every character a single, universal identity. For anyone studying information processing, understanding Unicode is essential because it is the foundation that allows digital information to flow across scripts, platforms, and borders without breaking.

Table of Contents

Why the world needed Unicode

To appreciate Unicode, you first need to understand the problem it was built to fix. For decades, the dominant standard was ASCII (American Standard Code for Information Interchange), introduced in 1963. ASCII used only 7 bits, which gave it room for just 128 characters. That was enough for the English alphabet, digits, basic punctuation, and a few control codes, but nothing more.

This limit created serious problems. ASCII could not even represent French accented letters properly, let alone the thousands of characters needed for scripts like Devanagari, Bengali, Arabic, or Chinese. To work around this, computer makers created hundreds of separate “extended” character sets, each using the eighth bit to add another 128 slots. The result was chaos. A document created on one system would often appear as garbled symbols, sometimes called “mojibake,” when opened on a different system that assumed a different character set.

The challenge became impossible to ignore as the internet connected computers across the globe. The Chinese, Japanese, and Korean writing systems alone contain tens of thousands of characters, far beyond what any 8-bit scheme could handle. The world clearly needed a single standard capable of representing every script at once.

The birth of a universal standard

The idea took shape in the late 1980s. While developing a Japanese-capable Macintosh, engineer Mark Davis recognised the need for a far larger and more comprehensive encoding. In 1987, Davis joined with Joe Becker and Lee Collins of Xerox to begin work on a new standard. They called it Unicode. Around the same time, Apple and Xerox were collaborating on building a true successor to ASCII capable of representing the characters of all the world’s languages.

As the project gained support, the Unicode Consortium was formally established in January 1991 as a non-profit organisation. Its members include major technology companies along with universities and governments. The Consortium’s guiding goal remains simple and powerful: assign a unique number to every character, regardless of the platform, the program, or the language. This work was later harmonised with the International Organization for Standardization, producing the shared standard known as ISO/IEC 10646.

How Unicode is structured and designed

The original design of Unicode was built around a 16-bit model. With 16 bits, a system can represent 2 raised to the power of 16, which equals 65,536 unique values. The early designers believed this single space, often described as “wide-body ASCII stretched to 16 bits,” would be enough to cover all the living languages of the world. Each character would receive its own number, called a code point, written in the format U+ followed by a hexadecimal value. For example, the capital letter “A” is U+0041.

This 16-bit space is now known as the Basic Multilingual Plane, or BMP. It runs from U+0000 to U+FFFF and contains the characters for almost all modern scripts in common use today, including Latin, Greek, Cyrillic, Arabic, the major Indic scripts, and a large set of symbols. For most everyday text, the BMP is all that is ever needed.

Growing beyond 65,536 characters

As work progressed, it became clear that 65,536 characters were not enough to hold the world’s living languages, historic scripts, mathematical notation, and later, emoji. The standard was therefore expanded. From version 2.0 onward, Unicode grew to support a much larger space organised into 17 “planes,” where each plane holds 65,536 code points.

This brings the total capacity to 1,114,112 possible code points. Plane 0 is the BMP. The planes above it, called supplementary planes, hold things like historic scripts, rare Chinese characters, musical and mathematical symbols, and emoji. So while the original concept was a clean 16-bit design, the modern standard is far larger. The 65,536 figure remains important historically and describes the size of a single plane, but it is no longer the ceiling of the entire system.

Encoding forms: UTF-8, UTF-16, and UTF-32

It is important to separate two ideas. A code point is the abstract number assigned to a character. An encoding form is the actual method used to store that number as bytes in a file or memory. Unicode defines three main encoding forms, and converting between them is lossless.

UTF-8 is the most widely used by a huge margin. It is a variable-width encoding that uses one to four bytes per character. Its great strength is that it is fully backward-compatible with ASCII: any valid ASCII file is automatically a valid UTF-8 file, because the first 128 characters use exactly the same single-byte values. This design made adoption easy, since older software that handled ASCII could handle basic UTF-8 text without modification. Today, UTF-8 powers the overwhelming majority of web pages.

UTF-16 uses either one or two 16-bit units. Characters in the BMP fit into a single 16-bit unit, while characters in the supplementary planes are represented using a pair of units called a surrogate pair. This encoding is used internally by systems such as Windows, Java, and JavaScript. UTF-32 uses a fixed 4 bytes for every character, which is simple to process but uses more storage space.

Compatibility with ASCII

One of the smartest decisions in Unicode’s design was preserving compatibility with ASCII. The first 128 Unicode code points are identical to the ASCII set, in the same order. This means existing English text and older data did not have to be rebuilt from scratch when systems moved to Unicode. Through UTF-8, that ASCII compatibility carries directly into the modern web, which is a major reason Unicode was adopted so smoothly across the computing world.

Unicode and Indian languages

For users in India, Unicode is especially significant because it brought reliable digital support to a country with many scripts and languages. Before Unicode, the relevant national standard was ISCII (Indian Script Code for Information Interchange), which encoded major scripts including Devanagari, Bengali, Gujarati, Gurmukhi, Kannada, Malayalam, Odia, Tamil, and Telugu. ISCII was clever in design but never saw wide use outside certain government institutions and has largely been replaced by Unicode.

Unicode did not discard this earlier work. Instead, it built upon it. The Devanagari block in Unicode is based on ISCII-1988, and Unicode gives each Indic script its own dedicated block while preserving the ISCII layout within each one. Because the Brahmi-derived scripts share a common structure, comparable characters are placed in the same relative positions across blocks. This parallel arrangement reflects the structural similarities between scripts and was intended to make transliteration between them easier.

Why this matters for information work

This design has real consequences for libraries, archives, and information systems. With Unicode, a single database can store catalogue records in English, Hindi, Bengali, and Tamil together without conflict. Search engines can index multilingual content reliably. Digital repositories can preserve documents in regional scripts knowing they will display correctly years later on any compliant system. For information professionals managing diverse collections, Unicode removed a major barrier to building truly multilingual digital resources.

A standard that keeps evolving

Unicode is not a finished, frozen system. The Consortium reviews proposals from around the world and releases regular updates. Recent versions have added new scripts, characters for historic and lesser-documented languages, and additions to existing blocks. This continuous evolution keeps the standard responsive to how people actually communicate, while the core promise stays the same: one unique number for every character, everywhere.

The shift from a fragmented world of incompatible character sets to a single universal standard is one of the quiet successes of modern computing. Unicode made it possible for text to travel freely between systems and languages, and it remains the backbone of nearly all digital text today.

What do you think? If Unicode had not been developed, how do you think multilingual countries with many scripts would manage digital communication today? And which Indian script or symbol would you most want to see better supported in future versions of the standard?

How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

References
  1. https://en.wikipedia.org/wiki/Unicode
  2. https://www.sciencedirect.com/topics/computer-science/unicode-standard
  3. https://en.wikipedia.org/wiki/Unicode_Consortium
  4. https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-2/
  5. https://en.wikipedia.org/wiki/Plane_(Unicode)
  6. https://learn.microsoft.com/en-us/globalization/encoding/unicode-standard
  7. https://developer.mozilla.org/en-US/docs/Glossary/UTF-16
  8. https://en.wikipedia.org/wiki/Indian_Script_Code_for_Information_Interchange
  9. https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-12/

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

Information Processing & Retrieval

1 Intellectual Organisation of Information

  1. Intellectual Organisation of Information
  2. Meaning of Intellectual Organisation of Information
  3. Why IOI is Necessary?
  4. IOI in Indexing Systems
  5. IOI and Indexing Languages
  6. IOI in User Services
  7. IOI and Content Analysis
  8. Information Retrieval Systems – Changing Environment
  9. Future Trends

2 Indexing Languages–Part I – Concepts and Types, Subject Headings Lists and Thesauri

  1. Indexing and its Types
  2. Indexing Language
  3. Vocabulary Control
  4. Classification Schemes
  5. Subject Headings Lists
  6. Thesaurus
  7. Thesaurofacet
  8. Classaurus
  9. Sears List of Subject Headings
  10. Library of Congress List of Subject Headings

3 Indexing Languages–Part II- Classification Schemes

  1. Dewey Decimal Classification (DDC) Scheme
  2. Universal Decimal Classification (UDC) Scheme
  3. Library of Congress Classification (LCC) Scheme
  4. Colon Classification (CC) Scheme
  5. Bibliographic Classification (BC) Scheme
  6. Library Bibliographical Classification (BBK) Scheme
  7. Broad System of Ordering (BSO) Scheme
  8. Special Classification Systems

4 Indexing Systems and Techniques

  1. Indexing Principles and Process
  2. Pre-Coordinate Indexing Systems
  3. Post-Coordinate Indexing Systems
  4. Automatic Indexing
  5. Non-Conventional Indexing: Citation Indexing
  6. Web Indexing

5 Evaluation of Indexing Systems

  1. Purpose of Evaluation
  2. Levels of Evaluation
  3. Evaluation Criteria
  4. Recall and Precision
  5. Other Performance Measures
  6. Relevance
  7. Evaluation Methodology
  8. Evaluation Experiments

6 Principles and Evolution of Bibliographic Description

  1. Bibliographic Description: An Overview
  2. Scope and Objectives of Bibliographic Description
  3. Evolution of Bibliographic Description
  4. Ranganathan’s Principles
  5. ISBDs
  6. Bibliographic Formats
  7. Electronic Resource Description
  8. Models of Bibliographic Description
  9. Bibliographic Description: Entities, Attributes and Relationships

7 Rules for Bibliographic Description

  1. Bibliographic Description: Its Origin
  2. Development of Anglo-American Code
  3. The International Standard Bibliographic Description (ISBD)
  4. Impact of ISBD on Catalogue Codes
  5. Bibliographic Description for Non-Print Materials
  6. Guidelines for Bibliographic Description of Electronic Resources
  7. Guidelines for Bibliographic Description of Internet Resources
  8. Rules for Description of Electronic Resources in AACR2 Revision 2002

8 Standards for Bibliographic Record Format

  1. International Standard Bibliographic Description (ISBD)
  2. MARC Format
  3. UNIMARC
  4. Common Communication Format (CCF)
  5. Indian Standard

9 Metadata- MARC21-856 Field, Dublin Core, TEI

  1. MARC21 – 856 Field
  2. Dublin Core Metadata Initiative (DCMI)
  3. Text Encoding Initiative (TEI)
  4. Procedure of Electronic Resource Description

10 Norms and Guidelines for Content Development

  1. Introduction
  2. Needs and Guidelines
  3. Standards Related to Electronic Content
  4. W3C Recommendations
  5. Electronic Text Encoding and Interchange
  6. Dynamic Content

11 Introduction to HTML and XML

  1. World Wide Web and Markup Languages
  2. Standard Generalized Markup Language (SGML)
  3. HyperText Markup Language (HTML)
  4. Basic HTML Tags
  5. Linking
  6. URLs
  7. HTML and the Browser
  8. eXtensible Markup Language (XML)
  9. XML Syntax and Semantic Tags
  10. Document Type Definition (DTD)
  11. Implications of XML in Library and Information Activities

12 Web-based Content Development

  1. What can be done with World Wide Web?
  2. Hypertext, Hyperlink, and Hypermedia
  3. Hypertext Markup Language (HTML)
  4. Introduction to Dynamic HTML
  5. Web Interface to Database Linking
  6. Introduction to XML
  7. XML Document Design
  8. Multimedia Web Resources
  9. Web Servers
  10. Website Hosting
  11. Tools for Web Page Designing

13 Multilingual Content Development (Using Unicode)

  1. Character Representation in Computer
  2. American Standard Code for Information Interchange (ASCII)
  3. Indian Scenario and Indian Standard Code for Information Interchange (ISCII)
  4. UNICODE
  5. Web Content Development Through UNICODE
  6. Applications of UNICODE
  7. Applying UNICODE to the Libraries
  8. Problems Associated with UNICODE

14 ISAR Systems- Objectives, Types, Operations and Design

  1. Users and Their Information Needs
  2. Objectives of ISAR Systems
  3. Types of ISAR Systems
  4. Design of ISAR Systems
  5. Evaluation of ISAR Systems

15 Compatibility of ISAR Systems

  1. Need for Compatibility Among ISAR Systems
  2. Scope of Compatibility in ISAR Systems
  3. Areas of Compatibilities in ISAR Systems
  4. Principal Issues of Compatibility in ISAR Systems
  5. Compatibility of Online IR Systems
  6. Approaches Towards Compatibility in ISAR
  7. Quality Control and Compatibility

16 Intelligent Information Retrieval Systems

  1. Introduction
  2. Expert Systems
  3. Expert Systems for Information Processing and Retrieval
  4. Components of Expert Systems
  5. Knowledge Representation
  6. Knowledge Engineering
  7. Artificial Intelligence Based Decision Support Systems (DSS)
  8. Pattern Recognition

17 Information Retrieval Processes and Techniques

  1. Information Retrieval Systems
  2. Databases
  3. Information Retrieval Systems: Purpose, Components, and Functions
  4. Indexing and Information Representation
  5. Vocabulary Control
  6. Searching
  7. Information Seeking and User Interfaces
  8. Web Information Retrieval Systems
  9. Intelligent Information Retrieval

18 Information Retrieval Models and Their Applications

  1. Information Retrieval
  2. Information Retrieval Techniques
  3. Models Based on Input/Output
  4. Models Based on Theories and Tools

19 Search Strategies, Processes and Techinques

  1. Search File – An Essential Component
  2. Search Strategies and Pre-requisites
  3. Search Techniques
  4. The Information Search Process
  5. Online Searching
  6. How the Search Engines Work
  7. Common Search and Retrieval Features of Web Search Engines