entropy (version 1.2.0)

KL.plugin: Plug-In Estimator of the Kullback-Leibler divergence and of the Chi-Squared Statistic

Description

KL.plugin computes the Kullback-Leiber (KL) divergence from random variable $X_1$ to $X_2$. The corresponding probability mass functions are given by freqs1 and freqs2, and the expectation is computed over freqs1. chi2.plugin computes the chi-squared statistic between an observed $X_1$ and an expected $X_2$, where freqs1 and freqs2 are the corresponding probability mass functions.

Usage

KL.plugin(freqs1, freqs2, unit=c("log", "log2", "log10"))
chi2.plugin(freqs1, freqs2, unit=c("log", "log2", "log10"))

Arguments

freqs1

bin frequencies for variable $X_1$.

freqs2

bin frequencies for variable $X_2$.

unit

the unit in which entropy is measured. The default is "nats" (natural units). For computing entropy in "bits" set unit="log2".

Value

• KL.plugin returns the KL divergence.

  chi2.plugin returns the chi-squared statistic.

Details

Kullback-Leibler divergence between the from random variable $X_1$ to $X_2$ is given as $E_{X_1} \log (f(x_1)/f(x_2))$.

The chi-squared statistic is given $\sum (f(x_1)-f(x_2))^2/f(x_2)$. It can also be seen as a second-order accurate approximation of twice the KL divergence.

Note that both the KL divergence and the chi-squared statistic are not symmetric in $X_1$ and $X_2$.

See Also

KL.Dirichlet, KL.shrink, KL.empirical, mi.plugin, discretize2d.

Examples

# load entropy library 
library("entropy")

# probabilities for two random variables
freqs1 = c(1/5, 1/5, 3/5)
freqs2 = c(1/10, 4/10, 1/2) 

# KL divergence from X1 to X2
KL.plugin(freqs1, freqs2)

# and corresponding (half) chi-squared statistic
0.5*chi2.plugin(freqs1, freqs2)