loo (version 0.1.4)

compare: Model comparison

Description

Compare fitted models on LOO or WAIC

Usage

compare(...)

Arguments

...

At least two objects returned by loo or waic.

Value

• A vector or matrix with class 'compare.loo'. If ... contains more than two objects then a matrix is returned. This matrix summarizes the objects and also reports model weights (the posterior probability that each model has the best expected out-of-sample predictive accuracy). If ... contains exactly two objects then the difference in expected predictive accuracy and the standard error of the difference are returned (see Details) in addition to model weights.

Details

When comparing two fitted models, we can estimate the difference in their expected predictive accuracy by the difference in elpd_waic or elpd_loo (multiplied by $-2$, if desired, to be on the deviance scale). To compute the standard error of this difference we can use a paired estimate to take advantage of the fact that the same set of $N$ data points was used to fit both models. We think these calculations will be most useful when $N$ is large, because then non-normality of the distribution is not such an issue when estimating the uncertainty in these sums. These standard errors, for all their flaws, should give a better sense of uncertainty than what is obtained using the current standard approach of comparing differences of deviances to a Chi-squared distribution, a practice derived for Gaussian linear models or asymptotically, and which only applies to nested models in any case.

References

Vehtari, A., Gelman, A., and Gabry, J. (2015). Efficient implementation of leave-one-out cross-validation and WAIC for evaluating fitted Bayesian models. http://arxiv.org/abs/1507.04544/ (preprint)

See Also

print.compare.loo

Examples

loo1 <- loo(log_lik1)
loo2 <- loo(log_lik2)
print(compare(loo1, loo2), digits = 3)

waic1 <- waic(log_lik1)
waic2 <- waic(log_lik2)
compare(waic1, waic2)