DescTools (version 0.99.19)

Entropy: Shannon Entropy and Mutual Information


Computes Shannon entropy and the mutual information of two variables. The entropy quantifies the expected value of the information contained in a vector. The mutual information is a quantity that measures the mutual dependence of the two random variables.


Entropy(x, y = NULL, base = 2, ...)
MutInf(x, y, base = 2, ...)


a vector or a matrix of numerical or categorical type. If only x is supplied it will be interpreted as contingency table.

a vector with the same type and dimension as x. If y is not NULL then the entropy of table(x, y, ...) will be calculated.

base of the logarithm to be used, defaults to 2.

further arguments are passed to the function table, allowing i.e. to set useNA.



The Shannon entropy equation provides a way to estimate the average minimum number of bits needed to encode a string of symbols, based on the frequency of the symbols. It is given by the formula $H = - \sum(\pi log(\pi))$ where $\pi$ is the probability of character number i showing up in a stream of characters of the given "script". The entropy is ranging from 0 to Inf.


Shannon, Claude E. (July/October 1948). A Mathematical Theory of Communication, Bell System Technical Journal 27 (3): 379-423.

Ihara, Shunsuke (1993) Information theory for continuous systems, World Scientific. p. 2. ISBN 978-981-02-0985-8.

See Also

package entropy which implements various estimators of entropy


Run this code

Entropy(as.matrix(rep(1/8, 8)))

x <- as.factor(c("a","b","a","c","b","c")) 
y <- as.factor(c("b","a","a","c","c","b")) 

Entropy(table(x), base=exp(1))
Entropy(table(y), base=exp(1))
Entropy(x, y, base=exp(1))

# Mutual information is 
Entropy(table(x), base=exp(1)) + Entropy(table(y), base=exp(1)) - Entropy(x, y, base=exp(1))
MutInf(x, y, base=exp(1))

Entropy(table(x)) + Entropy(table(y)) - Entropy(x, y)
MutInf(x, y, base=2)

tab <- matrix(c(60,10000,200,500000), nrow=2, byrow=TRUE)
MutInf(tab, base=2) 

d.frm <- Untable(as.table(tab))
MutInf(d.frm[,1], d.frm[,2])

table(d.frm[,1], d.frm[,2])

MutInf(table(d.frm[,1], d.frm[,2]))

# Ranking mutual information can help to describe clusters
#   r.mi <- MutInf(x, grp)
#   attributes(r.mi)$dimnames <- attributes(tab)$dimnames
#   # calculating ranks of mutual information
#   r.mi_r <- apply( -r.mi, 2, rank, na.last=TRUE )
#   # show only first 6 ranks
#   r.mi_r6 <- ifelse( r.mi_r < 7, r.mi_r, NA) 
#   attributes(r.mi_r6)$dimnames <- attributes(tab)$dimnames
#   r.mi_r6

Run the code above in your browser using DataCamp Workspace