entropy (version 1.2.1)

KL.plugin: Plug-In Estimator of the Kullback-Leibler divergence and of the Chi-Squared Statistic

Description

KL.plugin computes the Kullback-Leiber (KL) divergence from random variable \(X_1\) to \(X_2\). The corresponding probability mass functions are given by freqs1 and freqs2, and the expectation is computed over freqs1.

chi2.plugin computes the chi-squared statistic between an observed \(X_1\) and an expected \(X_2\), where freqs1 and freqs2 are the corresponding probability mass functions.

Usage

KL.plugin(freqs1, freqs2, unit=c("log", "log2", "log10"))
chi2.plugin(freqs1, freqs2, unit=c("log", "log2", "log10"))

Arguments

freqs1

bin frequencies for variable \(X_1\).

freqs2

bin frequencies for variable \(X_2\).

unit

the unit in which entropy is measured. The default is "nats" (natural units). For computing entropy in "bits" set unit="log2".

Value

KL.plugin returns the KL divergence.

chi2.plugin returns the chi-squared statistic.

Details

Kullback-Leibler divergence between the from random variable \(X_1\) to \(X_2\) is given as \( E_{X_1} \log (f(x_1)/f(x_2)) \).

The chi-squared statistic is given \( \sum (f(x_1)-f(x_2))^2/f(x_2) \). It can also be seen as a second-order accurate approximation of twice the KL divergence.

Note that both the KL divergence and the chi-squared statistic are not symmetric in \(X_1\) and \(X_2\).

See Also

KL.Dirichlet, KL.shrink, KL.empirical, mi.plugin, discretize2d.

Examples

# NOT RUN {
# load entropy library 
library("entropy")

# probabilities for two random variables
freqs1 = c(1/5, 1/5, 3/5)
freqs2 = c(1/10, 4/10, 1/2) 

# KL divergence from X1 to X2
KL.plugin(freqs1, freqs2)

# and corresponding (half) chi-squared statistic
0.5*chi2.plugin(freqs1, freqs2)

# }