Every time you send a message in Hindi, read a Bengali news website, or type an email mixing English and Tamil, a quiet system is working behind the screen to make sure those characters appear correctly. That system is Unicode. Before it existed, sharing text across different computers, operating systems, and languages was a frustrating gamble where words often turned into meaningless symbols. Unicode solved this by giving every character a single, universal identity. For anyone studying information processing, understanding Unicode is essential because it is the foundation that allows digital information to flow across scripts, platforms, and borders without breaking.
Table of Contents
Why the world needed Unicode
To appreciate Unicode, you first need to understand the problem it was built to fix. For decades, the dominant standard was ASCII (American Standard Code for Information Interchange), introduced in 1963. ASCII used only 7 bits, which gave it room for just 128 characters. That was enough for the English alphabet, digits, basic punctuation, and a few control codes, but nothing more.
This limit created serious problems. ASCII could not even represent French accented letters properly, let alone the thousands of characters needed for scripts like Devanagari, Bengali, Arabic, or Chinese. To work around this, computer makers created hundreds of separate “extended” character sets, each using the eighth bit to add another 128 slots. The result was chaos. A document created on one system would often appear as garbled symbols, sometimes called “mojibake,” when opened on a different system that assumed a different character set.
The challenge became impossible to ignore as the internet connected computers across the globe. The Chinese, Japanese, and Korean writing systems alone contain tens of thousands of characters, far beyond what any 8-bit scheme could handle. The world clearly needed a single standard capable of representing every script at once.
The birth of a universal standard
The idea took shape in the late 1980s. While developing a Japanese-capable Macintosh, engineer Mark Davis recognised the need for a far larger and more comprehensive encoding. In 1987, Davis joined with Joe Becker and Lee Collins of Xerox to begin work on a new standard. They called it Unicode. Around the same time, Apple and Xerox were collaborating on building a true successor to ASCII capable of representing the characters of all the world’s languages.
As the project gained support, the Unicode Consortium was formally established in January 1991 as a non-profit organisation. Its members include major technology companies along with universities and governments. The Consortium’s guiding goal remains simple and powerful: assign a unique number to every character, regardless of the platform, the program, or the language. This work was later harmonised with the International Organization for Standardization, producing the shared standard known as ISO/IEC 10646.
How Unicode is structured and designed
The original design of Unicode was built around a 16-bit model. With 16 bits, a system can represent 2 raised to the power of 16, which equals 65,536 unique values. The early designers believed this single space, often described as “wide-body ASCII stretched to 16 bits,” would be enough to cover all the living languages of the world. Each character would receive its own number, called a code point, written in the format U+ followed by a hexadecimal value. For example, the capital letter “A” is U+0041.
This 16-bit space is now known as the Basic Multilingual Plane, or BMP. It runs from U+0000 to U+FFFF and contains the characters for almost all modern scripts in common use today, including Latin, Greek, Cyrillic, Arabic, the major Indic scripts, and a large set of symbols. For most everyday text, the BMP is all that is ever needed.
Growing beyond 65,536 characters
As work progressed, it became clear that 65,536 characters were not enough to hold the world’s living languages, historic scripts, mathematical notation, and later, emoji. The standard was therefore expanded. From version 2.0 onward, Unicode grew to support a much larger space organised into 17 “planes,” where each plane holds 65,536 code points.
This brings the total capacity to 1,114,112 possible code points. Plane 0 is the BMP. The planes above it, called supplementary planes, hold things like historic scripts, rare Chinese characters, musical and mathematical symbols, and emoji. So while the original concept was a clean 16-bit design, the modern standard is far larger. The 65,536 figure remains important historically and describes the size of a single plane, but it is no longer the ceiling of the entire system.
Encoding forms: UTF-8, UTF-16, and UTF-32
It is important to separate two ideas. A code point is the abstract number assigned to a character. An encoding form is the actual method used to store that number as bytes in a file or memory. Unicode defines three main encoding forms, and converting between them is lossless.
UTF-8 is the most widely used by a huge margin. It is a variable-width encoding that uses one to four bytes per character. Its great strength is that it is fully backward-compatible with ASCII: any valid ASCII file is automatically a valid UTF-8 file, because the first 128 characters use exactly the same single-byte values. This design made adoption easy, since older software that handled ASCII could handle basic UTF-8 text without modification. Today, UTF-8 powers the overwhelming majority of web pages.
UTF-16 uses either one or two 16-bit units. Characters in the BMP fit into a single 16-bit unit, while characters in the supplementary planes are represented using a pair of units called a surrogate pair. This encoding is used internally by systems such as Windows, Java, and JavaScript. UTF-32 uses a fixed 4 bytes for every character, which is simple to process but uses more storage space.
Compatibility with ASCII
One of the smartest decisions in Unicode’s design was preserving compatibility with ASCII. The first 128 Unicode code points are identical to the ASCII set, in the same order. This means existing English text and older data did not have to be rebuilt from scratch when systems moved to Unicode. Through UTF-8, that ASCII compatibility carries directly into the modern web, which is a major reason Unicode was adopted so smoothly across the computing world.
Unicode and Indian languages
For users in India, Unicode is especially significant because it brought reliable digital support to a country with many scripts and languages. Before Unicode, the relevant national standard was ISCII (Indian Script Code for Information Interchange), which encoded major scripts including Devanagari, Bengali, Gujarati, Gurmukhi, Kannada, Malayalam, Odia, Tamil, and Telugu. ISCII was clever in design but never saw wide use outside certain government institutions and has largely been replaced by Unicode.
Unicode did not discard this earlier work. Instead, it built upon it. The Devanagari block in Unicode is based on ISCII-1988, and Unicode gives each Indic script its own dedicated block while preserving the ISCII layout within each one. Because the Brahmi-derived scripts share a common structure, comparable characters are placed in the same relative positions across blocks. This parallel arrangement reflects the structural similarities between scripts and was intended to make transliteration between them easier.
Why this matters for information work
This design has real consequences for libraries, archives, and information systems. With Unicode, a single database can store catalogue records in English, Hindi, Bengali, and Tamil together without conflict. Search engines can index multilingual content reliably. Digital repositories can preserve documents in regional scripts knowing they will display correctly years later on any compliant system. For information professionals managing diverse collections, Unicode removed a major barrier to building truly multilingual digital resources.
A standard that keeps evolving
Unicode is not a finished, frozen system. The Consortium reviews proposals from around the world and releases regular updates. Recent versions have added new scripts, characters for historic and lesser-documented languages, and additions to existing blocks. This continuous evolution keeps the standard responsive to how people actually communicate, while the core promise stays the same: one unique number for every character, everywhere.
The shift from a fragmented world of incompatible character sets to a single universal standard is one of the quiet successes of modern computing. Unicode made it possible for text to travel freely between systems and languages, and it remains the backbone of nearly all digital text today.
What do you think? If Unicode had not been developed, how do you think multilingual countries with many scripts would manage digital communication today? And which Indian script or symbol would you most want to see better supported in future versions of the standard?
References
- https://en.wikipedia.org/wiki/Unicode
- https://www.sciencedirect.com/topics/computer-science/unicode-standard
- https://en.wikipedia.org/wiki/Unicode_Consortium
- https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-2/
- https://en.wikipedia.org/wiki/Plane_(Unicode)
- https://learn.microsoft.com/en-us/globalization/encoding/unicode-standard
- https://developer.mozilla.org/en-US/docs/Glossary/UTF-16
- https://en.wikipedia.org/wiki/Indian_Script_Code_for_Information_Interchange
- https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-12/

Leave a Reply