Leave-One-Out (LOO) predictive checks. See the Plot Descriptions section, below, and Gabry et al. (2019) for details.
ppc_loo_pit_overlay(
y,
yrep,
lw = NULL,
...,
psis_object = NULL,
pit = NULL,
samples = 100,
size = 0.25,
alpha = 0.7,
boundary_correction = TRUE,
grid_len = 512,
bw = "nrd0",
trim = FALSE,
adjust = 1,
kernel = "gaussian",
n_dens = 1024
)ppc_loo_pit_data(
y,
yrep,
lw = NULL,
...,
psis_object = NULL,
pit = NULL,
samples = 100,
bw = "nrd0",
boundary_correction = TRUE,
grid_len = 512
)
ppc_loo_pit_qq(
y,
yrep,
lw = NULL,
...,
psis_object = NULL,
pit = NULL,
compare = c("uniform", "normal"),
size = 2,
alpha = 1
)
ppc_loo_pit_ecdf(
y,
yrep,
lw = NULL,
...,
psis_object = NULL,
pit = NULL,
K = NULL,
prob = 0.99,
plot_diff = FALSE,
interpolate_adj = NULL
)
ppc_loo_pit(
y,
yrep,
lw,
pit = NULL,
compare = c("uniform", "normal"),
...,
size = 2,
alpha = 1
)
ppc_loo_intervals(
y,
yrep,
psis_object,
...,
subset = NULL,
intervals = NULL,
prob = 0.5,
prob_outer = 0.9,
alpha = 0.33,
size = 1,
fatten = 2.5,
linewidth = 1,
order = c("index", "median")
)
ppc_loo_ribbon(
y,
yrep,
psis_object,
...,
subset = NULL,
intervals = NULL,
prob = 0.5,
prob_outer = 0.9,
alpha = 0.33,
size = 0.25
)
A ggplot object that can be further customized using the ggplot2 package.
A vector of observations. See Details.
An S by N matrix of draws from the posterior (or prior)
predictive distribution. The number of rows, S, is the size of the
posterior (or prior) sample used to generate yrep. The number of columns,
N is the number of predicted observations (length(y)). The columns of
yrep should be in the same order as the data points in y for the plots
to make sense. See the Details and Plot Descriptions sections for
additional advice specific to particular plots.
A matrix of (smoothed) log weights with the same dimensions as
yrep. See loo::psis() and the associated weights() method as well as
the Examples section, below. If lw is not specified then
psis_object can be provided and log weights will be extracted.
Currently unused.
If using loo version 2.0.0 or greater, an
object returned by the psis() function (or by the loo() function
with argument save_psis set to TRUE).
For ppc_loo_pit_overlay(), ppc_loo_pit_qq(), and
ppc_loo_pit_ecdf() optionally a vector of precomputed PIT values that
can be specified instead of y, yrep, and lw (these are all ignored
if pit is specified). If not specified the PIT values are computed
internally before plotting.
For ppc_loo_pit_overlay(), the number of data sets (each
the same size as y) to simulate from the standard uniform
distribution. The default is 100. The density estimate of each dataset is
plotted as a thin line in the plot, with the density estimate of the LOO
PITs overlaid as a thicker dark line.
Arguments passed to code geoms to control
plot aesthetics. For ppc_loo_pit_qq() and ppc_loo_pit_overlay(),size
and alpha are passed to ggplot2::geom_point() and
ggplot2::geom_density(), respectively. For ppc_loo_intervals(), size
linewidth and fatten are passed to ggplot2::geom_pointrange(). For
ppc_loo_ribbon(), alpha and size are passed to
ggplot2::geom_ribbon().
For ppc_loo_pit_overlay(), when set to TRUE
(the default) the function will compute boundary corrected density values
via convolution and a Gaussian filter, also known as the reflection method
(Boneva et al., 1971). As a result, parameters controlling the standard
kernel density estimation such as adjust, kernel and n_dens are
ignored. NOTE: The current implementation only works well for continuous
observations.
For ppc_loo_pit_overlay(), when boundary_correction is
set to TRUE this parameter specifies the number of points used to
generate the estimations. This is set to 512 by default.
Optional arguments passed to
stats::density() to override default kernel density estimation
parameters. n_dens defaults to 1024.
Passed to ggplot2::stat_density().
For ppc_loo_pit_qq(), a string that can be either
"uniform" or "normal". If "uniform" (the default) the Q-Q plot
compares computed PIT values to the standard uniform distribution. If
compare="normal", the Q-Q plot compares standard normal quantiles
calculated from the PIT values to the theoretical standard normal
quantiles.
For ppc_loo_pit_ecdf() an optional integer defining the number
of equally spaced evaluation points for the PIT-ECDF. Reducing K when
using interpolate_adj = FALSE makes computing the confidence bands
faster. If pit is supplied, defaults to length(pit), otherwise
yrep determines the maximum accuracy of the estimated PIT values and
K is set to min(nrow(yrep) + 1, 1000).
Values between 0 and 1 indicating the desired
probability mass to include in the inner and outer intervals. The defaults
are prob=0.5 and prob_outer=0.9 for ppc_loo_intervals() and
prob = 0.99 for ppc_loo_pit_ecdf().
For ppc_loo_pit_ecdf(), a boolean defining whether to
plot the difference between the observed PIT-ECDF and the theoretical
expectation for uniform PIT values rather than plotting the regular ECDF.
The default is FALSE, but for large samples we recommend setting
plot_diff = TRUE to better use the plot area.
For ppc_loo_pit_ecdf(), a boolean defining if the
simultaneous confidence bands should be interpolated based on precomputed
values rather than computed exactly. Computing the bands may be
computationally intensive and the approximation gives a fast method for
assessing the ECDF trajectory. The default is to use interpolation if K
is greater than 200.
For ppc_loo_intervals() and ppc_loo_ribbon(), an optional
integer vector indicating which observations in y (and yrep) to
include. Dropping observations from y and yrep manually before passing
them to the plotting function will not work because the dimensions will not
match up with the dimensions of psis_object, but if all of y and yrep
are passed along with subset then bayesplot can do the subsetting
internally for y, yrep and psis_object. See the Examples
section for a demonstration.
For ppc_loo_intervals() and ppc_loo_ribbon(), optionally
a matrix of pre-computed LOO predictive intervals that can be specified
instead of yrep (ignored if intervals is specified). If not specified
the intervals are computed internally before plotting. If specified,
intervals must be a matrix with number of rows equal to the number of
data points and five columns in the following order: lower outer interval,
lower inner interval, median (50%), upper inner interval and upper outer
interval (column names are ignored).
For ppc_loo_intervals(), a string indicating how to arrange
the plotted intervals. The default ("index") is to plot them in the
order of the observations. The alternative ("median") arranges them
by median value from smallest (left) to largest (right).
ppc_loo_pit_overlay(), ppc_loo_pit_qq(), ppc_loo_pit_ecdf()The calibration of marginal predictions can be assessed using probability integral transformation (PIT) checks. LOO improves the check by avoiding the double use of data. See the section on marginal predictive checks in Gelman et al. (2013, p. 152--153) and section 5 of Gabry et al. (2019) for an example of using bayesplot for these checks.
The LOO PIT values are asymptotically uniform (for continuous data) if the
model is calibrated. The ppc_loo_pit_overlay() function creates a plot
comparing the density of the LOO PITs (thick line) to the density estimates
of many simulated data sets from the standard uniform distribution (thin
lines). See Gabry et al. (2019) for an example of interpreting the shape of
the miscalibration that can be observed in these plots.
The ppc_loo_pit_qq() function provides an alternative visualization of
the miscalibration with a quantile-quantile (Q-Q) plot comparing the LOO
PITs to the standard uniform distribution. Comparing to the uniform is not
good for extreme probabilities close to 0 and 1, so it can sometimes be
useful to set the compare argument to "normal", which will
produce a Q-Q plot comparing standard normal quantiles calculated from the
PIT values to the theoretical standard normal quantiles. This can help see
the (mis)calibration better for the extreme values. However, in most cases
we have found that the overlaid density plot (ppc_loo_pit_overlay())
function will provide a clearer picture of calibration problems than the
Q-Q plot.
The ppc_loo_pit_ecdf() function visualizes the empirical cumulative
distribution function (ECDF) of the LOO PITs overlaid with simultaneous
confidence intervals for a standard uniform sample. For large samples,
these confidence intervals are visually very narrow. Setting the
plot_diff argument to TRUE transforms the plot to display the
difference of the ECDF and the theoretical expectation, which can aid in
the visual assessment of calibration.
ppc_loo_intervals(), ppc_loo_ribbon()Similar to ppc_intervals() and ppc_ribbon() but the intervals are for
the LOO predictive distribution.
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. B. (2013). Bayesian Data Analysis. Chapman & Hall/CRC Press, London, third edition. (p. 152--153)
Gabry, J. , Simpson, D. , Vehtari, A. , Betancourt, M. and Gelman, A. (2019), Visualization in Bayesian workflow. J. R. Stat. Soc. A, 182: 389-402. doi:10.1111/rssa.12378. (journal version, arXiv preprint, code on GitHub)
Vehtari, A., Gelman, A., and Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing. 27(5), 1413--1432. doi:10.1007/s11222-016-9696-4. arXiv preprint: https://arxiv.org/abs/1507.04544
Boneva, L. I., Kendall, D., & Stefanov, I. (1971). Spline transformations: Three new diagnostic aids for the statistical data-analyst. J. R. Stat. Soc. B (Methodological), 33(1), 1-71. https://www.jstor.org/stable/2986005.
Other PPCs:
PPC-censoring,
PPC-discrete,
PPC-distributions,
PPC-errors,
PPC-intervals,
PPC-overview,
PPC-scatterplots,
PPC-test-statistics
if (FALSE) {
library(rstanarm)
library(loo)
head(radon)
fit <- stan_lmer(
log_radon ~ floor + log_uranium + floor:log_uranium
+ (1 + floor | county),
data = radon,
iter = 100,
chains = 2,
cores = 2
)
y <- radon$log_radon
yrep <- posterior_predict(fit)
loo1 <- loo(fit, save_psis = TRUE, cores = 4)
psis1 <- loo1$psis_object
lw <- weights(psis1) # normalized log weights
# marginal predictive check using LOO probability integral transform
color_scheme_set("orange")
ppc_loo_pit_overlay(y, yrep, lw = lw)
ppc_loo_pit_qq(y, yrep, lw = lw)
ppc_loo_pit_qq(y, yrep, lw = lw, compare = "normal")
# predictive calibration check using LOO probability integral transform
ppc_loo_pit_ecdf(y, yrep, lw)
# With `plot_diff = TRUE` it is easier to assess the calibration.
ppc_loo_pit_ecdf(y, yrep, lw, plot_diff = TRUE)
# can use the psis object instead of lw
ppc_loo_pit_qq(y, yrep, psis_object = psis1)
# loo predictive intervals vs observations
keep_obs <- 1:50
ppc_loo_intervals(y, yrep, psis_object = psis1, subset = keep_obs)
color_scheme_set("gray")
ppc_loo_intervals(y, yrep,
psis_object = psis1, subset = keep_obs,
order = "median"
)
}
Run the code above in your browser using DataLab