Shannon entropy
Measure uncertainty in a set of possible messages.
Library note. Check the assumptions and further reading before applying a formula.
What Is This?
Entropy measures the average uncertainty of a random outcome. An outcome that is nearly certain tells you little when it happens. A rare outcome tells you more. Shannon entropy averages that surprise over all outcomes. Using base-two logarithms gives bits. The probabilities describe the source; entropy does not measure whether a message is useful, truthful or meaningful.
Try an example
A fair coin has one bit of entropy per toss. A coin that always lands heads has zero. Knowing the next result of the second coin adds no information about its outcome.
Where it needs care
This sum is for a discrete distribution, with zero-probability terms taken as zero. Continuous differential entropy has different properties and is not a direct substitute.
Historical Context
Claude Shannon introduced the framework in "A Mathematical Theory of Communication" (1948) to study communication and coding limits.
Real-World Applications
- Find theoretical limits for lossless compression.
- Measure uncertainty in classification models.