Leave-One-Out (LOO) predictive checks. See the Plot Descriptions section below for details.
ppc_loo_pit_overlay(y, yrep, lw, pit, samples = 100, ..., size = 0.25,
alpha = 0.7, trim = FALSE, bw = "nrd0", adjust = 1,
kernel = "gaussian", n_dens = 1024)ppc_loo_pit_qq(y, yrep, lw, pit, compare = c("uniform", "normal"), ...,
size = 2, alpha = 1)
ppc_loo_pit(y, yrep, lw, pit, compare = c("uniform", "normal"), ...,
size = 2, alpha = 1)
ppc_loo_intervals(y, yrep, psis_object, subset = NULL, intervals = NULL,
..., prob = 0.5, prob_outer = 0.9, size = 1, fatten = 3,
order = c("index", "median"))
ppc_loo_ribbon(y, yrep, lw, psis_object, subset = NULL, intervals = NULL,
..., prob = 0.5, prob_outer = 0.9, alpha = 0.33, size = 0.25)
A vector of observations. See Details.
An \(S\) by \(N\) matrix of draws from the posterior
predictive distribution, where \(S\) is the size of the posterior sample
(or subset of the posterior sample used to generate yrep) and
\(N\) is the number of observations (the length of y). The columns
of yrep should be in the same order as the data points in y
for the plots to make sense. See Details for additional
instructions.
A matrix of (smoothed) log weights with the same dimensions as
yrep. See psis and the associated weights
method and the Examples section, below.
For ppc_loo_pit_overlay and ppc_loo_pit_qq,
optionally a vector of precomputed PIT values that can be specified instead
of y, yrep, and lw (these are all ignored if
pit is specified). If not specified the PIT values are computed
internally before plotting.
For ppc_loo_pit_overlay, the number of data sets (each
the same size as y) to simulate from the standard uniform
distribution. The default is 100. The density estimate of each dataset is
plotted as a thin line in the plot, with the density estimate of the LOO
PITs overlaid as a thicker dark line.
Currently unused.
Arguments passed to code geoms to control plot
aesthetics. For ppc_loo_pit_qq and ppc_loo_pit_overlay,
size and alpha are passed to
geom_point and geom_density,
respectively. For ppc_loo_intervals, size and fatten
are passed to geom_pointrange. For
ppc_loo_ribbon, alpha and size are passed to
geom_ribbon.
Passed to stat_density.
Optional arguments passed to
density to override default kernel density estimation
parameters. n_dens defaults to 1024.
For ppc_loo_pit_qq, a string that can be either
"uniform" or "normal". If "uniform" (the default) the
Q-Q plot compares computed PIT values to the standard uniform distribution.
If compare="normal", the Q-Q plot compares standardized PIT values
to the standard normal distribution.
If using loo version 2.0.0 or greater, an
object returned by the psis function (or by the loo function
with argument save_psis set to TRUE).
For ppc_loo_intervals and ppc_loo_ribbon, an
optional integer vector indicating which observations in y (and
yrep) to include. Dropping observations from y and
yrep manually before passing them to the plotting function will not
work because the dimensions will not match up with the dimensions of
psis_object, but if all of y and yrep are passed along
with subset then bayesplot can do the subsetting internally
for y, yrep and psis_object.
See the Examples section for a demonstration.
For ppc_loo_intervals and ppc_loo_ribbon,
optionally a matrix of precomputed LOO predictive intervals
that can be specified instead of yrep and lw (these are both
ignored if intervals is specified). If not specified the intervals
are computed internally before plotting. If specified, intervals
must be a matrix with number of rows equal to the number of data points and
five columns in the following order: lower outer interval, lower inner
interval, median (50%), upper inner interval and upper outer interval
(column names are ignored).
Values between 0 and 1 indicating the desired
probability mass to include in the inner and outer intervals. The defaults
are prob=0.5 and prob_outer=0.9.
For ppc_loo_intervals, a string indicating how to arrange
the plotted intervals. The default ("index") is to plot them in the
order of the observations. The alternative ("median") arranges them
by median value from smallest (left) to largest (right).
A ggplot object that can be further customized using the ggplot2 package.
ppc_loo_pit_qq,ppc_loo_pit_overlayThe calibration of marginal predictions can be assessed using probability integral transformation (PIT) checks. LOO improves the check by avoiding the double use of data. See the section on marginal predictive checks in Gelman et al. (2013, p. 152--153) and section 5 of Gabry et al. (2018) for an example of using bayesplot for these checks.
The LOO PIT values are asymptotically uniform (for continuous data) if the
model is calibrated. The ppc_loo_pit_overlay function creates a plot
comparing the density of the LOO PITs (thick line) to the density estimates
of many simulated data sets from the standard uniform distribution (thin
lines). See Gabry et al. (2018) for an example of interpreting the shape of
the miscalibration that can be observed in these plots.
The ppc_loo_pit_qq function provides an alternative visualization of
the miscalibration with a quantile-quantile (Q-Q) plot comparing the LOO
PITs to the standard uniform distribution. Comparing to the uniform is not
good for extreme probabilities close to 0 and 1, so it can sometimes be
useful to set the compare argument to "normal", which will
produce a Q-Q plot comparing standardized PIT values to the standard normal
distribution that can help see the (mis)calibration better for the extreme
values. However, in most cases we have found that the overlaid density plot
(ppc_loo_pit_overlay) function will provided a clearer picture of
calibration problems that the Q-Q plot.
ppc_loo_intervals, ppc_loo_ribbonSimilar to ppc_intervals and ppc_ribbon but the
intervals are for the LOO predictive distribution.
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. B. (2013). Bayesian Data Analysis. Chapman & Hall/CRC Press, London, third edition. (p. 152--153)
Gabry, J., Simpson, D., Vehtari, A., Betancourt, M., and Gelman, A. (2018). Visualization in Bayesian workflow. Journal of the Royal Statistical Society Series A, accepted for publication. arXiv preprint: http://arxiv.org/abs/1709.01449.
Vehtari, A., Gelman, A., and Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing. 27(5), 1413--1432. doi:10.1007/s11222-016-9696-4. arXiv preprint: http://arxiv.org/abs/1507.04544/
Other PPCs: PPC-discrete,
PPC-distributions,
PPC-errors, PPC-intervals,
PPC-overview,
PPC-scatterplots,
PPC-test-statistics
# NOT RUN {
# }
# NOT RUN {
library(rstanarm)
library(loo)
head(radon)
fit <- stan_lmer(
log_radon ~ floor + log_uranium + floor:log_uranium
+ (1 + floor | county),
data = radon,
iter = 1000,
chains = 2 # ,cores = 2
)
y <- radon$log_radon
yrep <- posterior_predict(fit)
loo1 <- loo(fit, save_psis = TRUE, cores = 2)
psis1 <- loo1$psis_object
lw <- weights(psis1)
# marginal predictive check using LOO probability integral transform
color_scheme_set("orange")
ppc_loo_pit_overlay(y, yrep, lw = lw)
ppc_loo_pit_qq(y, yrep, lw = lw)
ppc_loo_pit_qq(y, yrep, lw = lw, compare = "normal")
# loo predictive intervals vs observations
keep_obs <- 1:50
ppc_loo_intervals(y, yrep, psis_object = psis1, subset = keep_obs)
color_scheme_set("gray")
ppc_loo_intervals(y, yrep, psis_object = psis1, subset = keep_obs,
order = "median")
# }
# NOT RUN {
# }
Run the code above in your browser using DataCamp Workspace