Every time you guess the next word in a sentence, predict tomorrow’s weather, or wait for a message to load, you are dealing with uncertainty. Information theory turns that uncertainty into something we can measure. At the heart of this idea sits a simple but powerful claim: the amount of information carried by an event depends on how likely it is to happen. This is the foundation of probabilistic information, and it explains why a surprising message tells us more than a predictable one. Let us unpack how probability quietly powers the entire field of information theory.
Table of Contents
- The basis of probabilistic information
- Self-information and the logarithm
- Entropy as average information
- Shannon’s classical model
- The five core components
- The discrete memoryless source
- Independent symbols and information rate
- Applications of probabilistic information
- Data compression and the source coding theorem
- Reliable transmission and channel capacity
- Beyond communication engineering
The basis of probabilistic information
Information theory rests on a counter-intuitive insight: information and probability are inversely linked. An event that is almost certain to occur tells us very little when it happens. An event that is rare and unexpected, on the other hand, carries a great deal of information. If a weather app in Chennai predicts a hot day in May, the forecast is barely informative because it was expected. A forecast of snowfall in Chennai would be enormously informative precisely because it is so improbable.
This is why information is modelled mathematically using probability. When Claude Shannon introduced the field in 1948, he defined the information content of an event in terms of its probability. The less likely an outcome, the more information it delivers when observed. In his seminal work, Shannon treated the output of an information source as a random variable, where each possible symbol appears with a fixed probability, and the information it provides is a function of that probability.
Self-information and the logarithm
The information content of a single event is called self-information. For an event with probability p, it is calculated as the negative logarithm of that probability. Using a logarithm with base 2 gives us the answer in bits, the basic unit of information. A coin toss with two equally likely outcomes carries exactly one bit of information. An event that is certain to happen, with a probability of 1, carries zero bits, because there is no uncertainty to resolve.
The logarithm is not an arbitrary choice. It ensures that the information from two independent events adds up neatly. If you learn the outcome of two separate coin tosses, you gain two bits in total. This additive property is exactly what we want from a sensible measure of information, and probability theory combined with the logarithm gives it to us naturally.
Entropy as average information
While self-information measures a single event, we usually care about a whole source that keeps producing symbols. To capture this, Shannon defined entropy, which is the average information per symbol produced by a source. Entropy is computed by weighting the self-information of each possible symbol by its probability and summing across all symbols. According to standard treatments of the concept, entropy measures the average uncertainty associated with the source’s output.
Entropy reaches its maximum when all outcomes are equally likely, because that situation is the most unpredictable. For a source emitting two symbols, entropy peaks at one bit when both symbols have a probability of one-half. If one symbol becomes far more likely than the other, the source becomes more predictable, and its entropy drops. When a single outcome is certain, entropy falls to zero. This relationship between probability distribution and average information is the cornerstone of the entire theory.
Shannon’s classical model
Shannon laid out his ideas in the 1948 paper “A Mathematical Theory of Communication”, published in the Bell System Technical Journal. The framework, later expanded with Warren Weaver and popularly known as the Shannon-Weaver model, became so influential that it is sometimes called the mother of all communication models. It describes communication as a one-directional flow through a sequence of stages.
The five core components
The classical model breaks communication into five basic components. The information source produces the original message. The transmitter encodes that message into a signal suitable for transmission. The channel is the medium that carries the signal, such as a telephone wire, an optical fibre, or the air. The receiver decodes the signal back into a message. Finally, the destination is the person or device for whom the message is intended.
A telephone call illustrates the chain clearly. The caller is the source, the handset acts as the transmitter that converts speech into an electrical signal, the wire is the channel, and the listener’s handset is the receiver that reconstructs the sound for the destination. A sixth element, noise, sits across the channel and represents anything that distorts or corrupts the signal during transmission.
The discrete memoryless source
To make the model mathematically tractable, Shannon focused on a specific kind of source called the discrete memoryless source. The word “discrete” means the source emits symbols from a fixed, finite alphabet, such as the letters of an alphabet or the binary digits 0 and 1. Each symbol is emitted with a defined probability, and all these probabilities together add up to one.
The word “memoryless” is the crucial part. It means that the symbol produced at any moment is statistically independent of every symbol that came before it. The source has no memory of its past output, so knowing the previous symbols gives you no help in predicting the next one. As researchers describe it, each message in a memoryless source is an independent and identically distributed random variable. This assumption greatly simplifies the mathematics while still capturing the essence of many real information sources.
Independent symbols and information rate
Because the symbols are independent, the entropy of the whole source is simply the entropy of a single symbol. There is no need to account for correlations between consecutive symbols, since none exist. The information rate, meaning the average amount of information produced per symbol, equals the entropy of the source under this assumption.
This is a major reason the memoryless model is so widely taught and used. It provides a clean baseline. Real sources, such as natural language text, do have memory, because the letter following “q” in English is very often “u”. Shannon and later researchers extended the theory to handle such cases using stationary and ergodic sources, where the information rate becomes the conditional entropy of a symbol given all previous ones. But the memoryless case remains the essential starting point for understanding how probability governs information flow.
Applications of probabilistic information
The probabilistic model is not merely a theoretical curiosity. It set the limits for two of the most important problems in communication and underpins technologies that billions of people use every day. Shannon proved two landmark results that flow directly from the probabilistic treatment of sources and channels.
Data compression and the source coding theorem
Shannon’s source coding theorem establishes the fundamental limit of lossless data compression. It states that the average number of bits needed to represent each symbol cannot fall below the entropy of the source without losing information. In other words, entropy is the floor for how much you can shrink data. This principle of efficient encoding means that frequent symbols should be assigned short codes and rare symbols longer ones, which minimises the average message length.
This idea drives the compression formats we rely on constantly. Algorithms such as Huffman coding and arithmetic coding aim to approach the entropy limit, and they form the basis of familiar tools. The ZIP format for files and the JPEG format for images both exploit statistical redundancy in data to reduce its size while preserving the essential content. For anyone managing digital archives, repositories, or large document collections, these compression techniques quietly make storage and transfer feasible.
Reliable transmission and channel capacity
The second result is the channel coding theorem, which deals with noise. Every real channel corrupts some of the data passing through it. Shannon introduced the concept of channel capacity, the maximum rate at which information can be transmitted reliably over a noisy channel. As long as the information rate stays below this capacity, a noisy channel coding result shows that the probability of error can be made arbitrarily small by adding carefully designed redundancy. Push beyond the capacity, and reliable communication becomes impossible.
This is why error-correcting codes exist. They add structured redundancy so a receiver can detect and fix corrupted bits. Mobile networks, satellite links, deep-space communication, and data storage on hard drives all depend on this principle to deliver accurate information despite interference.
Beyond communication engineering
The reach of probabilistic information extends well past telephones and modems. In cryptography, entropy measures the unpredictability of keys, which determines how secure they are. In information retrieval and library and information science, probabilistic measures help quantify the informativeness of terms in documents, supporting search ranking and indexing. Entropy-based ideas also appear in machine learning, where they guide decision trees and model evaluation, and in scientometrics, where probability distributions describe how publications, citations, and keywords are spread across a field. Across all these areas, the same insight holds: measuring information means measuring uncertainty, and uncertainty is the language of probability.
What do you think? If a message that is more surprising carries more information, does that mean the most predictable communication is also the least useful? And as our information systems grow more complex, do you think Shannon’s memoryless assumption still serves us well, or do the patterns and memory in real data demand richer models?
Leave a Reply