A list of at least two objects returned by loo or
waic. This argument can be used as an alternative to
specifying the models in ....
Value
A vector or matrix with class 'compare.loo' that has its own
print method. If exactly two objects are provided in ... or
x, then the difference in expected predictive accuracy and the
standard error of the difference are returned (see Details). The
difference will be positive if the expected predictive accuracy for the
second model is higher. If more than two objects are provided then a
matrix of summary information is returned.
Details
When comparing two fitted models, we can estimate the difference in
their expected predictive accuracy by the difference in elpd_waic or
elpd_loo (multiplied by $-2$, if desired, to be on the deviance
scale). To compute the standard error of this difference we can use a
paired estimate to take advantage of the fact that the same set of $N$
data points was used to fit both models. We think these calculations will
be most useful when $N$ is large, because then non-normality of the
distribution is not such an issue when estimating the uncertainty in these
sums. These standard errors, for all their flaws, should give a better
sense of uncertainty than what is obtained using the current standard
approach of comparing differences of deviances to a Chi-squared
distribution, a practice derived for Gaussian linear models or
asymptotically, and which only applies to nested models in any case.
References
Vehtari, A., Gelman, A., and Gabry, J. (2016a). Practical
Bayesian model evaluation using leave-one-out cross-validation and WAIC.
Statistics and Computing. Advance online publication.
doi:10.1007/s11222-016-9696-4. arXiv preprint:
http://arxiv.org/abs/1507.04544/
Vehtari, A., Gelman, A., and Gabry, J. (2016b). Pareto smoothed
importance sampling. arXiv preprint: http://arxiv.org/abs/1507.02646/