Understanding Entropy and its Role in Information Theory

Few ideas travel across science as widely as entropy. It started in the steam engines of the nineteenth century, moved into the molecular world of physics, and eventually reshaped how we measure information itself. For students of informetrics and scientometrics, this single concept connects the disorder of a hot gas to the uncertainty in a text message. Understanding entropy is the key to understanding what information actually is, why some messages carry more of it than others, and how communication systems fight against the constant pull toward chaos.

Entropy in thermodynamics 🔗

Entropy first appeared in physics as a way to describe energy and disorder. In thermodynamics, entropy is a measure of a system’s disorder and of how much of its energy is unavailable to do useful work. A neat stack of papers has low entropy. The same papers scattered across a room have high entropy. The physical world consistently drifts from the first state toward the second.

This drift is captured by the second law of thermodynamics. The law states that the total entropy of an isolated system never decreases; it either stays constant in an idealised reversible process or increases in any real, irreversible one. Heat flows from hot objects to cold ones, never the reverse on its own. A drop of ink spreads through water but never gathers back into a drop. These everyday observations are all expressions of entropy increasing.

The statistical view of disorder 🔗

The physicist Ludwig Boltzmann gave entropy a deeper meaning by linking it to the number of microscopic arrangements a system can have. A macrostate that can be produced in many different ways is more disordered and has higher entropy. A macrostate with only a few possible arrangements is more ordered and has lower entropy. Melting ice is a clear example: a structured crystal of water molecules becomes a liquid in which molecules have no fixed positions, producing a large increase in entropy. This statistical interpretation matters because it is the bridge that later carried entropy out of physics and into the study of information.

The link between entropy and information 🔗

In 1948, the engineer and mathematician Claude Shannon needed a quantity to measure the uncertainty in a message. He found that the entropy of a random variable quantifies the average level of uncertainty or information tied to that variable’s possible outcomes. The mathematics he derived looked almost identical to Boltzmann’s formula from thermodynamics. On the suggestion of the mathematician John von Neumann, Shannon adopted the same word, entropy, because both ideas describe uncertainty and share a very similar mathematical form.

Here is the central idea. Before you learn the value of something uncertain, entropy measures how much you do not know. After you learn it, the same quantity measures how much information you have gained. These are two complementary views of the same number: entropy as uncertainty beforehand, and entropy as information gained afterward.

Why entropy and information are connected, not identical 🔗

A common confusion is to treat entropy and information as the same thing. They are tightly related but distinct. In Shannon’s framework, entropy and uncertainty can be used interchangeably, but neither of them simply means information. Information is what you gain when uncertainty is resolved. A coin toss with two equally likely outcomes carries one bit of uncertainty. Once the coin lands, that bit of uncertainty is converted into one bit of information.

This also explains the units. When the logarithm in Shannon’s formula is taken to base 2, the resulting units are called bits. A message that could be one of many equally likely possibilities has high entropy and therefore carries a lot of potential information. A message that is completely predictable carries almost none.

Negentropy: information as the opposite of disorder 🔗

If entropy measures disorder, what measures order? The answer is negentropy, short for negative entropy. The concept came from the physicist Erwin Schrödinger in his 1944 book What is Life?, and the French physicist Léon Brillouin later shortened the phrase to negentropy and carried it into information theory. Negentropy represents the measure of order, organisation, and information within a system, standing in direct contrast to entropy’s measure of disorder.

Schrödinger introduced the idea to explain living organisms. A living body maintains a remarkably ordered internal state while the universe around it tends toward disorder. He argued that organisms survive by feeding on order from their surroundings, which he called negative entropy. Brillouin then made the link to information explicit, proposing that information corresponds to a negative term in the total entropy of a system, so that a highly organised structure like the genetic code in DNA has lower entropy than a random collection of molecules.

For information science, this is a powerful reframing. A well-organised library catalogue, a structured database, or a carefully indexed collection are all islands of low entropy. They represent stored order, and that order is exactly what makes information retrievable. Maintaining this order takes continuous effort, because the natural tendency of any unmanaged system is to slide back toward disorder.

Entropy in communication 🔗

Shannon developed his theory at Bell Telephone Laboratories to solve a very practical problem: how much information can be pushed through a wire or radio wave before it degrades into noise. The model he built with Warren Weaver, the Shannon-Weaver model, breaks communication into a source, a transmitter, a channel, noise, and a receiver. Entropy sits at the heart of it.

Organised versus disorganised messages 🔗

In communication, entropy measures the unpredictability of a message. A message where every symbol is equally likely and unrelated to the next carries maximum entropy. A message that is highly predictable carries low entropy. This is where the idea of organised and disorganised message systems becomes practical.

Consider written English. The letters are not used with equal frequency, and some combinations are far more likely than others. After the letter “q”, you can predict the next letter with near certainty. This predictability lowers the entropy of the text. The flip side of predictability is redundancy. In the Shannon-Weaver framework, redundancy is the portion of a message that adds no new information and could be removed without losing meaning. Roughly half the words in ordinary text are, in this technical sense, redundant, as they add no new information and could be dropped without disturbing completeness.

Why redundancy fights noise 🔗

Redundancy may sound wasteful, but it is the main defence against noise. Noise is any interference that distorts the signal as it travels through the channel. Shannon and Weaver proposed redundancy as the solution: if the original message contains redundancy, distortions can be detected and the sender’s original intention can be reconstructed.

This is why pilots and radio operators say “Alpha, Bravo, Charlie” instead of “A, B, C”. The extra information protects accuracy when the channel is unreliable. The same principle influenced modern data compression and error-correcting codes, from ZIP files to satellite links. A balance must be struck: too much entropy makes a message unpredictable and fragile, while too much redundancy makes it inefficient. The art of good communication system design lies in finding the point between the unexpected and the predictable.

The full circle 🔗

This brings the three faces of entropy together. In thermodynamics, entropy measures physical disorder. In information theory, it measures uncertainty and the information that resolving that uncertainty provides. In communication, it measures the unpredictability of messages and sets the limits on how efficiently they can be compressed and transmitted. Negentropy ties them together by naming the order and organisation that information represents. A noisy channel and a melting ice cube are, at a deep level, expressions of the same tendency, and the work of organising information is the work of holding entropy at bay.

For anyone measuring information, whether counting citations, analysing scientific literature, or designing a retrieval system, entropy offers a precise vocabulary. It lets us say not just that one source has more content than another, but that it carries more information in a quantifiable sense. That precision is what turned a thermodynamic curiosity into one of the foundational tools of information science.

What do you think? If a perfectly organised library catalogue represents low entropy and high information, what happens to that information when the catalogue is left unmaintained for years? And in your own writing, where would you draw the line between useful redundancy that guards against misunderstanding and wasteful repetition that adds nothing new?

References

1. https://courses.lumenlearning.com/suny-physics/chapter/15-6-entropy-and-the-second-law-of-thermodynamics-disorder-and-the-unavailability-of-energy/
2. https://phys.libretexts.org/Bookshelves/Conceptual_Physics/Introduction_to_Physics_(Park)/04:_Unit_3-_Classical_Physics_-_Thermodynamics_Electricity_and_Magnetism_and_Light/08:_Thermal_Physics/8.13:_Entropy_and_the_Second_Law_of_Thermodynamics-_Disorder_and_the_Unavailability_of_Energy
3. https://en.wikipedia.org/wiki/Entropy_(information_theory)
4. https://en.wikipedia.org/wiki/Entropy_in_thermodynamics_and_information_theory
5. https://arxiv.org/pdf/quant-ph/0011036
6. https://arxiv.org/pdf/q-bio/0405004
7. https://arxiv.org/pdf/1501.01854
8. https://en.wikipedia.org/wiki/Negentropy
9. https://atahanaslan.medium.com/what-is-the-opposite-of-entropy-negentropy-concept-astronomy-explained-8b0a150b8290
10. https://scienceinsights.org/what-does-negative-entropy-mean-in-biology/
11. https://journalism.university/fundamentals-of-development-and-communication/shannon-weaver-mathematical-communication-model/
12. https://islmblogblog.wordpress.com/wp-content/uploads/2016/05/assignment3shannonandweavermodel.pdf
13. https://en.wikipedia.org/wiki/Shannon%E2%80%93Weaver_model
14. https://journalism.university/introduction-to-journalism-and-mass-communication/shannon-weaver-model-communication-efficiency/