Key Properties of Shannon’s Information Measure

Every time you read a search result, scan a citation index, or decide whether a new journal is worth adding to a library collection, you are dealing with information. But how do you actually measure it? In 1948, the engineer Claude Shannon answered that question in his paper A Mathematical Theory of Communication, giving the world a precise, mathematical way to quantify how much information an event carries. His measure is now the backbone of information theory, and it underpins much of how we think about data in informetrics and scientometrics. To use it well, you need to understand the rules it obeys. This post walks through three of the most important properties of Shannon’s information measure: non-negativity, nullity with permutation symmetry, and additivity.

The idea behind Shannon’s measure 🔗

Before the properties make sense, the core formula needs a quick explanation. Shannon defined the information content of a single outcome, often called self-information or surprisal, as the negative logarithm of its probability. If an event has probability p, its information content is I = -log p. The smaller the probability of an event, the larger the amount of self-information it carries when it occurs.

The logic is simple. Rare events are surprising, and surprises tell you a lot. Common events are predictable, so learning that they happened tells you little. If a friend in a desert city says “it is sunny today,” that is barely informative; if she says “it is raining today,” that carries far more information because rain there is unlikely. The logarithm turns this intuition into a quantity.

The unit depends on the base of the logarithm. With base 2 the unit is the bit, a term Shannon credited to his Bell Labs colleague John Tukey. With the natural logarithm the unit is the nat, and with base 10 it is the hartley. When you average self-information across all possible outcomes of a random variable, weighted by their probabilities, you get Shannon entropy, written H. Entropy is the expected, or average, surprisal of a source. The three properties below apply both to single outcomes and to the entropy of a whole distribution.

Non-negativity 🔗

The first and most foundational property is that information can never be negative. Shannon entropy is always non-negative for any probability distribution. Mathematically, this is written H ≥ 0.

The reason is built into the formula. A probability always lies between 0 and 1, and the logarithm of a number in that range is zero or negative. Putting a minus sign in front flips it to zero or positive. So every term in the entropy sum is non-negative, and the total cannot drop below zero.

The interpretation is just as important as the algebra. Receiving a message about an experiment cannot make you more ignorant than you were before. At worst, a message tells you nothing new and adds zero information. It can never subtract from what you already know. This is why non-negativity is treated as the bedrock property: it guarantees that information is a quantity you accumulate, never one you lose by observing.

Why this matters for measurement 🔗

Because information is bounded below by zero, any metric built on Shannon’s measure behaves predictably. When you compute the diversity of subject coverage in a database, or the uncertainty in which journal a citation will point to, the result is always a clean, non-negative number. There are no awkward negative scores to interpret. For information professionals building indicators of collection richness or research diversity, this property quietly makes the whole toolkit reliable.

Nullity and permutation symmetry 🔗

The next two properties are closely related and are best understood together. They describe the boundary case where information disappears, and the way the measure ignores the order of events.

Nullity: when information equals zero 🔗

Nullity is the property that information is exactly zero when an outcome is certain. If an event has a probability of 1, its self-information is -log(1) = 0. A constant event with probability 1.0 conveys no information because it is completely unsurprising.

This formalizes plain common sense. Being told something you already know with absolute certainty teaches you nothing. Quanta Magazine illustrates this with a two-headed trick coin: if you flip it twice, communicating the result takes no information at all, because you already have complete certainty that both flips will land heads. There is no uncertainty to resolve, so there is no information to transmit. Nullity is the natural floor that sits right at the edge of the non-negativity property: zero is the minimum, and it is reached precisely when there is nothing left to be surprised about.

Permutation symmetry: order does not matter 🔗

Permutation symmetry states that the amount of information is invariant under a change in the order of events. If you have a set of outcomes with probabilities p₁, p₂, p₃ and so on, you can shuffle those probabilities into any arrangement and the entropy stays exactly the same. The measure is a symmetric function of the probabilities, so arbitrary permutations of them will not alter the value of the information.

This makes sense because entropy depends only on the list of probabilities, not on the labels attached to them. A coin that lands heads with probability 0.7 and tails with 0.3 has the same entropy as one that lands tails with 0.7 and heads with 0.3. The measure cares about the shape of the uncertainty, not about which outcome is sitting in which slot.

For classification and indexing, this property is reassuring. When you assess how evenly documents are spread across subject categories, the information value does not change just because you reorder the categories on a shelf or in a facet list. The retrievability of items in a faceted classification scheme does not hinge on the sequence of the facets. Symmetry tells you that the structure of the distribution, not its presentation order, drives the measurement.

Additivity 🔗

The third major property is additivity, which governs how information combines across multiple events. It is the property that makes Shannon’s measure scale gracefully from single events to large systems.

Additivity for independent events 🔗

When two events are independent, the information from observing both together equals the sum of the information from each one separately. In entropy terms, the entropy of independent random variables is the sum of their individual entropies. If X and Y are independent, then H(X, Y) = H(X) + H(Y).

This follows directly from the logarithm. The joint probability of two independent events is the product of their probabilities, and the logarithm of a product is the sum of the logarithms. So multiplication of probabilities becomes addition of information. Flip two independent fair coins and you gain one bit from each, for two bits total. The measure adds up cleanly, which is exactly the behavior you want from a quantity meant to count information.

Additivity for dependent events 🔗

Real data is rarely fully independent, and Shannon’s framework handles that too through a modified rule. When events are dependent, the total information is the entropy of the first event plus the conditional entropy of the second given the first: H(X, Y) = H(X) + H(Y | X). This is the chain rule of entropy. If the two events happen to be independent, the conditional term collapses back to the plain entropy of Y, and you recover the simple additive form. Dependence reduces the second term because part of the second event’s information is already explained by the first.

This refinement matters for measuring overlapping resources. Suppose a library already subscribes to one citation database and is considering a second that partly covers the same journals. The conditional version of additivity lets you quantify the incremental information the new database adds, after accounting for the overlap. You are not double-counting the shared coverage; you are measuring only the genuinely new content. That distinction is central to evidence-based collection development.

Putting the properties to work in informetrics 🔗

Together these properties give information professionals a dependable analytical foundation. Non-negativity ensures that expanding a collection always adds a non-negative amount to its information content, so even seemingly redundant material contributes something or, at worst, nothing. Nullity reminds us that perfectly predictable holdings carry no new information, which is a useful caution against assuming that sheer volume equals value. Permutation symmetry supports classification and indexing systems where the order of categories should not affect measured retrievability. Additivity lets analysts combine the information value of separate sources, and its conditional form handles the realistic case of overlapping coverage.

These are also the properties that mathematicians used to characterize Shannon’s measure axiomatically. Later work by Khinchin tightened Shannon’s original set of conditions into a formal axiom system, showing that continuity, symmetry, additivity, and a few related properties together pin down entropy as essentially the only measure that behaves the way information should. Understanding these rules is therefore not just academic detail. It is what lets you trust the numbers when you apply scientometric indicators to citation networks, journal coverage, or research output.

What do you think? If non-negativity means adding any resource to a collection can never reduce its information content, does that change how you would justify weeding or deselecting materials? And when two databases heavily overlap, how would you decide whether the small amount of conditional information the second one adds is worth its subscription cost?

References

1. https://www.quantamagazine.org/how-claude-shannons-concept-of-entropy-quantifies-information-20220906/
2. https://en.wikipedia.org/wiki/Self-information
3. https://en.wikipedia.org/wiki/A_Mathematical_Theory_of_Communication
4. https://fiveable.me/statistical-mechanics/unit-10/shannon-entropy/study-guide/n31vaHGy23kHjoiu
5. https://arxiv.org/pdf/1001.3473
6. https://arxiv.org/pdf/2505.23960
7. https://arxiv.org/pdf/cond-mat/0607605
8. https://arxiv.org/pdf/1108.2874