loo (version 0.1.6)

loo: Leave-one-out cross-validation (LOO)

Description

Efficient approximate leave-one-out cross-validation for Bayesian models.

Usage

loo(x, ...)
"loo"(x, ...)
"loo"(x, ..., args)

Arguments

x

A log-likelihood matrix or function. See the Methods (by class) section below for a detailed description.

...

Optional arguments to pass to psislw. Possible arguments and their defaults are:

wcp = 0.2

The proportion of importance weights to use for the generalized Pareto fit. The 100*wcp% largest weights are used as the sample from which to estimate the parameters $k$ and $\sigma$ of the generalized Pareto distribution.

wtrunc = 3/4

For truncating very large weights to $S$^wtrunc (set to zero for no truncation).

cores = getOption("loo.cores", parallel::detectCores())

The number of cores to use for parallelization. This can be set for an entire R session by options(loo.cores = NUMBER). The default is detectCores().

We recommend using the default values for the psislw arguments unless there are problems (e.g. NA or NaN results).

args

Only required if x is a function. A list containing the data required to specify the arguments to the function. See the Methods (by class) section below for how args should be specified.

Value

A named list with class 'loo' and components:

elpd_loo, se_elpd_loo

Expected log pointwise predictive density and standard error.

p_loo, se_p_loo

Estimated effective number of parameters and standard error.

looic, se_looic

The LOO information criterion (-2*elpd_loo, i.e., converted to deviance scale) and standard error.

pointwise

A matrix containing the pointwise contributions of each of the above measures.

pareto_k

A vector containing the estimates of the shape parameter $k$ for the generaelized Pareto fit to the importance ratios for each leave-one-out distribution. See PSIS-LOO section in loo-package for details about interpreting $k$. (By default, the print method for 'loo' objects will also provide warnings about problematic values of $k$.)

Methods (by class)

• matrix: An $S$ by $N$ matrix, where $S$ is the size of the posterior sample (the number of simulations) and $N$ is the number of data points. Typically (but not restricted to be) the object returned by extract_log_lik.
• function: A function $f$ that takes arguments i, data, and draws and returns a vector containing the log-likelihood for the ith observation evaluated at each posterior draw. The args argument must also be specified and should be a named list with the following components:
  • draws: An object containing the posterior draws for any parameters needed to compute the pointwise log-likelihood.
  • data: An object containing data (e.g. observed outcome and predictors) needed to compute the pointwise log-likelihood. data should be in an appropriate form so that $f$(i=i, data=data[i,,drop=FALSE], draws=draws) returns the S-vector of log-likelihoods for the ith observation.
  • N: The number of observations.
  • S: The size of the posterior sample.

Details

PSIS-LOO We approximate LOO using Pareto Smoothed Importance Sampling (PSIS). See loo-package for details.

Memory For models fit to very large datasets we recommend the loo.function method, which is much more memory efficient than the loo.matrix method. However, the loo.matrix method is typically more convenient, so it is usually worth trying loo.matrix and then switching to loo.function if memory is an issue.

See Also

loo-package for details on the Pareto Smoothed Importance Sampling (PSIS) procedure used for approximating LOO.

print.loo for the print and plot methods for 'loo' objects.

compare for model comparison.

Examples

## Not run: 
# ### Usage with stanfit objects
# log_lik1 <- extract_log_lik(stanfit1) # see ?extract_log_lik
# loo1 <- loo(log_lik1)
# print(loo1, digits = 3)
# 
# log_lik2 <- extract_log_lik(stanfit2)
# (loo2 <- loo(log_lik2))
# compare(loo1, loo2)
# ## End(Not run)

### Using log-likelihood function instead of matrix
set.seed(024)

# Simulate data and draw from posterior
N <- 50; K <- 10; S <- 100; a0 <- 3; b0 <- 2
p <- rbeta(1, a0, b0)
y <- rbinom(N, size = K, prob = p)
a <- a0 + sum(y); b <- b0 + N * K - sum(y)
draws <- rbeta(S, a, b)
data <- data.frame(y,K)

llfun <- function(i, data, draws) {
  dbinom(data$y, size = data$K, prob = draws, log = TRUE)
}
loo_with_fn <- loo(llfun, args = nlist(data, draws, N, S), cores = 1)

# Check that we get same answer if using log-likelihood matrix
log_lik_mat <- sapply(1:N, function(i) llfun(i, data[i,, drop=FALSE], draws))
loo_with_mat <- loo(log_lik_mat, cores = 1)
all.equal(loo_with_mat, loo_with_fn)