loo (version 1.1.0)

loo: Leave-one-out cross-validation (LOO)

Description

Efficient approximate leave-one-out cross-validation for Bayesian models. See loo-package and Vehtari, Gelman, and Gabry (2016a, 2016b) for background.

Usage

loo(x, ...)
# S3 method for matrix
loo(x, ...)
# S3 method for function
loo(x, ..., args)

Arguments

x

A log-likelihood matrix or function. See the Methods (by class) section below for a detailed description.

...

Optional arguments to pass to psislw. Possible arguments and their defaults are:

wcp = 0.2

The proportion of importance weights to use for the generalized Pareto fit. The 100*wcp% largest weights are used as the sample from which to estimate the parameters \(k\) and \(\sigma\) of the generalized Pareto distribution.

wtrunc = 3/4

For truncating very large weights to \(S\)^wtrunc (set to zero for no truncation).

cores = getOption("loo.cores", parallel::detectCores())

The number of cores to use for parallelization. This can be set for an entire R session by options(loo.cores = NUMBER). The default is detectCores().

We recommend using the default values for the psislw arguments unless there are problems (e.g. NA or NaN results).

args

Only required if x is a function. A list containing the data required to specify the arguments to the function. See the Methods (by class) section below for how args should be specified.

Value

A named list with class 'loo' and components:

elpd_loo, se_elpd_loo

Expected log pointwise predictive density and standard error.

p_loo, se_p_loo

Estimated effective number of parameters and standard error.

looic, se_looic

The LOO information criterion (-2*elpd_loo, i.e., converted to deviance scale) and standard error.

pointwise

A matrix containing the pointwise contributions of each of the above measures.

pareto_k

A vector containing the estimates of the shape parameter \(k\) for the generaelized Pareto fit to the importance ratios for each leave-one-out distribution. See PSIS-LOO section in loo-package for details about interpreting \(k\). (By default, the print method for 'loo' objects will also provide warnings about problematic values of \(k\).)

Methods (by class)

• matrix: An \(S\) by \(N\) matrix, where \(S\) is the size of the posterior sample (the number of simulations) and \(N\) is the number of data points. For example, the object returned by extract_log_lik can be used.

• function: A function \(f\) that takes arguments i, data, and draws and returns a vector containing the log-likelihood for the ith observation evaluated at each posterior draw.

  The args argument must also be specified and should be a named list with the following components:

  • draws: An object containing the posterior draws for any parameters needed to compute the pointwise log-likelihood.

  • data: An object containing data (e.g. observed outcome and predictors) needed to compute the pointwise log-likelihood. data should be in an appropriate form so that \(f\)(i=i, data=data[i,,drop=FALSE], draws=draws) returns the S-vector of log-likelihoods for the ith observation.

  • N: The number of observations.

  • S: The size of the posterior sample.

References

Vehtari, A., Gelman, A., and Gabry, J. (2016a). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing. Advance online publication. doi:10.1007/s11222-016-9696-4. (published version, arXiv preprint).

Vehtari, A., Gelman, A., and Gabry, J. (2016b). Pareto smoothed importance sampling. arXiv preprint: http://arxiv.org/abs/1507.02646/

See Also

psislw for the underlying Pareto Smoothed Importance Sampling (PSIS) procedure used for approximating LOO.

pareto-k-diagnostic for convenience functions for looking at diagnostics.

compare for model comparison.

print.loo for the print method for 'loo' objects.

Examples

# NOT RUN {
### Usage with stanfit objects
log_lik1 <- extract_log_lik(stanfit1) # see ?extract_log_lik
loo1 <- loo(log_lik1)
print(loo1, digits = 3)

log_lik2 <- extract_log_lik(stanfit2)
(loo2 <- loo(log_lik2))
compare(loo1, loo2)
# }
# NOT RUN {
### Using log-likelihood function instead of matrix
set.seed(024)

# Simulate data and draw from posterior
N <- 50; K <- 10; S <- 100; a0 <- 3; b0 <- 2
p <- rbeta(1, a0, b0)
y <- rbinom(N, size = K, prob = p)
a <- a0 + sum(y); b <- b0 + N * K - sum(y)
draws <- rbeta(S, a, b)
data <- data.frame(y,K)

llfun <- function(i, data, draws) {
  dbinom(data$y, size = data$K, prob = draws, log = TRUE)
}
loo_with_fn <- loo(llfun, args = nlist(data, draws, N, S), cores = 1)

# Check that we get same answer if using log-likelihood matrix
log_lik_mat <- sapply(1:N, function(i) llfun(i, data[i,, drop=FALSE], draws))
loo_with_mat <- loo(log_lik_mat, cores = 1)
all.equal(loo_with_mat, loo_with_fn)

# }