Information Entropy: The Average Information Content of a Message

Every time you read a text message, scroll a news feed, or stream a video, you are receiving information measured in a precise unit. That unit-and the idea that information can be measured at all-comes from a single concept called information entropy. Introduced by Claude Shannon in 1948, it answers a deceptively simple question: how much information does a message actually carry, on average? For anyone studying library and information science, understanding entropy is essential, because it explains how we quantify, transmit, and preserve information in a world that runs on data.

Understanding entropy: what it is and how it is calculated 🔗

At its most basic level, entropy measures the uncertainty or unpredictability in a set of possible outcomes. The concept was borrowed from physics, where entropy in statistical thermodynamics measures disorder in a system. Shannon adapted the same mathematical idea to messages and signals, and the two formulas turn out to be formally identical.

The key insight is that information is linked to surprise. A message that tells you something you already expected carries little information. A message that tells you something surprising carries a lot. Shannon was the first person to make this relationship mathematically precise, capturing it in a formula that calculates the minimum number of bits needed to communicate a message.

Consider a coin toss. If the coin is fair, heads and tails are equally likely, and you genuinely cannot predict the result. That uncertainty is maximum for a two-outcome event. The information entropy is maximised when the probability is 0.5, which corresponds exactly to the fair coin toss. In this case the entropy works out to 1 bit. That single bit can be communicated with one symbol: 1 for heads, 0 for tails.

Now imagine a trick coin that always lands heads. There is no uncertainty at all, so the outcome carries no information. Its entropy is zero. A variable that can only take one value has zero entropy, because we already know with complete certainty what it will be. This gives us the two boundaries of entropy: it ranges from zero (complete certainty) up to a maximum value set by how many equally likely outcomes exist.

Information content of a single event 🔗

Before averaging across a whole message, we measure the information of one event. This is called self-information or surprisal. The rule is straightforward: rare events carry more information than common ones. If you toss a fair coin and it lands heads, the surprise is modest. If that same coin landed on its edge-an extremely rare event-the information content would be enormous.

Self-information is calculated as the negative logarithm of the event’s probability. A fair coin toss, with a probability of 0.5, has a surprisal value of exactly 1 bit, while a sure event has a surprisal of 0. The logarithm is what gives us a useful additive property: if two independent events occur, the information from both together is simply the sum of their individual information values. This additivity is one reason the logarithmic measure feels so natural for counting information.

The role of entropy in communication systems 🔗

Shannon did not invent entropy as an abstract puzzle. He was working at Bell Telephone Laboratories on a concrete problem: how to send messages reliably and efficiently. His model defines a communication system with three elements: a source of data, a channel, and a receiver. The fundamental problem of communication, as Shannon framed it, is for the receiver to correctly identify what the source sent, based only on the signal that arrives through the channel.

Entropy enters this model in several places. Source entropy describes how much information the source produces on average. Channel capacity describes how much information the channel can reliably carry. Noise adds uncertainty during transmission. The central principle that links them is simple but powerful: reliable communication requires the channel capacity to exceed the source entropy. If you try to push more information through a channel than it can handle, some loss becomes unavoidable.

Why probability is at the centre 🔗

Entropy depends entirely on the probability distribution of the messages a source can produce. A source that sends one symbol almost all the time has low entropy, because most of its output is predictable. A source whose symbols are spread evenly across many possibilities has high entropy, because each new symbol is hard to anticipate.

This is why letter frequency matters in real languages. In English, the letter ‘e’ appears around 12 percent of the time, while ‘z’ appears around 0.07 percent. Because letters are not uniformly distributed, English text carries only about 1 to 1.5 bits per character, far less than the roughly 4.7 bits needed if all 26 letters were equally likely. That gap between actual entropy and maximum possible entropy is called redundancy, and it is precisely what allows text to be compressed.

Shannon’s equation for entropy 🔗

Shannon expressed the average information content of a source with one elegant formula. Written for a source with possible symbols each having probability p, the entropy H is the negative sum of each probability multiplied by the logarithm of that probability:

H(X) = − Σ p(x) log₂ p(x)

Each term in the sum represents one outcome’s contribution: its probability multiplied by its self-information. Adding these contributions gives a weighted average-the expected surprise across the whole distribution. In other words, entropy is nothing more than the expected value of the self-information function for a given probability distribution.

The base of the logarithm sets the unit. Using base 2 gives the answer in bits, the unit used in digital systems. Using the natural logarithm gives nats, and base 10 gives dits. Bits are the standard, because they map directly onto the binary symbols that computers and communication channels use.

A worked example with a biased coin 🔗

The formula becomes clear with numbers. For a fair coin, both outcomes have probability 0.5, so the entropy is (0.5 × 1 + 0.5 × 1) = 1 bit. Now take a biased coin where heads has probability 0.875 and tails has probability 0.125. Plugging these into the equation gives an entropy of roughly 0.544 bits. The biased coin carries less information per toss because its outcome is more predictable. The closer a distribution moves toward certainty, the lower its entropy falls.

Entropy as the limit of compression 🔗

One of Shannon’s most important results connects entropy to data compression. In his source coding theorem, he proved that entropy represents an absolute mathematical limit on how well data from a source can be losslessly compressed. You simply cannot compress data below its entropy without losing some of it.

This is not a theoretical curiosity. Modern compression formats such as ZIP and JPEG are designed around the principles of Shannon entropy, striving to approach the minimum bit count that entropy specifies. The technique is to assign short codes to frequent symbols and longer codes to rare ones, so the average code length approaches the source’s entropy. For libraries and digital archives managing enormous collections, this is the mathematical foundation behind efficient storage.

Maximising entropy for efficient communication 🔗

If entropy measures the average information per symbol, then a source that maximises its entropy is packing the most possible information into each symbol it sends. So when is entropy at its maximum?

The answer is clean: entropy is highest when every outcome is equally likely. The Shannon entropy of a discrete source reaches its maximum when its symbols follow a uniform distribution-the state of highest uncertainty. For a source with n equally likely outcomes, the maximum entropy equals log₂(n) bits. With two outcomes that is 1 bit; with four equally likely outcomes it is 2 bits; with eight it is 3 bits.

The principle of maximum entropy 🔗

This idea extends beyond communication into how we should reason under uncertainty. The principle of maximum entropy says that if nothing is known about a distribution except the class it belongs to, the distribution with the largest entropy should be chosen as the least-informative default. The motivation is that maximising entropy minimises the amount of prior assumption built into our model. We commit only to what we actually know and stay maximally uncertain about everything else.

In practical communication design, this translates into a clear goal. An efficient encoding scheme tries to make the transmitted symbols as close to uniformly distributed as possible, so that no capacity is wasted on predictability. In communication, we aim to transmit the maximum number of bits within a given period, and that means squeezing redundancy out and pushing the effective entropy of the transmitted stream toward its maximum.

Why this matters for information science 🔗

For students of library and information science, entropy is more than an equation. It underpins how search engines weigh terms, how digital repositories compress and store documents, and how networks decide how much data a link can reliably carry. Understanding that information has a measurable lower bound-and that we can never compress or transmit below it-gives a rigorous foundation for evaluating any system that creates, moves, or preserves information.

What do you think? If maximum information is carried when every outcome is equally likely, what does the high redundancy of natural human language tell us about why we communicate the way we do? And in an age of massive digital archives, where should the balance lie between compressing data to its entropy limit and keeping redundancy for safety?

References

1. https://en.wikipedia.org/wiki/Entropy_(information_theory)
2. https://www.quantamagazine.org/how-claude-shannons-concept-of-entropy-quantifies-information-20220906/
3. https://thepalindrome.org/p/shannon-entropy-how-to-measure-information
4. https://medium.com/@mhannan94/information-entropy-3cce2bac62a2
5. http://www.cs.toronto.edu/~yangxu/information-theory-v3.pdf
6. https://www.tensortonic.com/ml-math/information-theory/shannon-entropy
7. https://www.scribd.com/document/73763776/Entropy
8. https://rendazhang.medium.com/information-theory-series-1-entropy-and-shannon-entropy-a20a2101108e
9. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8147137/
10. https://en.wikipedia.org/wiki/Maximum_entropy_probability_distribution
11. https://medium.com/@acamvproducingstudio/channel-capacity-ae7987b99cc5