Leave-One-Out (LOO) predictive checks. See the Plot Descriptions section, below, and Gabry et al. (2019) for details.
ppc_loo_pit_overlay(
y,
yrep,
lw = NULL,
...,
psis_object = NULL,
pit = NULL,
samples = 100,
size = 0.25,
alpha = 0.7,
boundary_correction = TRUE,
grid_len = 512,
bw = NULL,
trim = FALSE,
adjust = NULL,
kernel = NULL,
n_dens = NULL
)ppc_loo_pit_data(
y,
yrep,
lw = NULL,
...,
psis_object = NULL,
pit = NULL,
samples = 100,
bw = "nrd0",
boundary_correction = TRUE,
grid_len = 512
)
ppc_loo_pit_qq(
y,
yrep,
lw = NULL,
...,
psis_object = NULL,
pit = NULL,
compare = c("uniform", "normal"),
size = 2,
alpha = 1
)
ppc_loo_pit_ecdf(
y,
yrep,
lw = NULL,
...,
psis_object = NULL,
pit = NULL,
K = NULL,
prob = 0.99,
plot_diff = FALSE,
interpolate_adj = NULL,
method = NULL,
test = NULL,
gamma = NULL,
linewidth = NULL,
color = NULL,
help_text = NULL,
pareto_pit = NULL,
help_text_shrinkage = NULL
)
ppc_loo_pit(
y,
yrep,
lw,
pit = NULL,
compare = c("uniform", "normal"),
...,
size = 2,
alpha = 1
)
ppc_loo_intervals(
y,
yrep,
psis_object,
...,
subset = NULL,
intervals = NULL,
prob = 0.5,
prob_outer = 0.9,
alpha = 0.33,
size = 1,
fatten = 2.5,
linewidth = 1,
order = c("index", "median")
)
ppc_loo_ribbon(
y,
yrep,
psis_object,
...,
subset = NULL,
intervals = NULL,
prob = 0.5,
prob_outer = 0.9,
alpha = 0.33,
size = 0.25
)
The plotting functions return a ggplot object that can be further
customized using the ggplot2 package. The functions with suffix
_data() return the data that would have been drawn by the plotting
function.
A vector of observations. See Details.
An S by N matrix of draws from the posterior (or prior)
predictive distribution, or a posterior::draws object. The number of
rows, S, is the size of the posterior (or prior) sample used to generate
yrep. The number of columns, N is the number of predicted observations
(length(y)). The columns of yrep should be in the same order as the
data points in y for the plots to make sense. See the Details and
Plot Descriptions sections for additional advice specific to particular
plots.
A matrix of (smoothed) log weights with the same dimensions as
yrep. See loo::psis() and the associated weights() method as well as
the Examples section, below. If lw is not specified then
psis_object can be provided and log weights will be extracted.
Currently unused.
If using loo version 2.0.0 or greater, an
object returned by the psis() function (or by the loo() function
with argument save_psis set to TRUE).
For ppc_loo_pit_overlay(), ppc_loo_pit_qq(), and
ppc_loo_pit_ecdf(), an optional vector of precomputed LOO-PIT values
(length length(y), values in [0, 1]). If supplied, y, yrep, and
lw / psis_object are ignored. If NULL (default), LOO-PIT values are
computed internally.
For ppc_loo_pit_overlay(), the number of data sets (each
the same size as y) to simulate from the standard uniform
distribution. The default is 100. The density estimate of each dataset is
plotted as a thin line in the plot, with the density estimate of the LOO
PITs overlaid as a thicker dark line.
Arguments passed to code geoms to control
plot aesthetics. For ppc_loo_pit_qq() and ppc_loo_pit_overlay(),size
and alpha are passed to ggplot2::geom_point() and
ggplot2::geom_density(), respectively. For ppc_loo_intervals(), size
linewidth and fatten are passed to ggplot2::geom_pointrange(). For
ppc_loo_ribbon(), alpha and size are passed to
ggplot2::geom_ribbon(). For ppc_loo_pit_ecdf(), linewidth for the ECDF plot. When
method = "correlated", defaults to 0.3. When method = "independent",
if NULL no linewidth is specified for the ECDF line.
For ppc_loo_pit_overlay(), when set to TRUE
(the default) the function will compute boundary corrected density values
via convolution and a Gaussian filter, also known as the reflection method
(Boneva et al., 1971). As a result, parameters controlling the standard
kernel density estimation such as adjust, kernel and n_dens are
ignored. NOTE: The current implementation only works well for continuous
observations.
For ppc_loo_pit_overlay(), when boundary_correction is
set to TRUE this parameter specifies the number of points used to
generate the estimations. This is set to 512 by default.
Optional arguments passed to
stats::density() to override the defaults.
Passed to ggplot2::stat_density().
For ppc_loo_pit_qq(), a string that can be either
"uniform" or "normal". If "uniform" (the default) the Q-Q plot
compares computed PIT values to the standard uniform distribution. If
compare="normal", the Q-Q plot compares standard normal quantiles
calculated from the PIT values to the theoretical standard normal
quantiles.
An optional integer defining the number of equally spaced evaluation
points for the PIT-ECDF. Reducing K when using interpolate_adj = FALSE
makes computing the confidence bands faster. For ppc_loo_pit_ecdf(),
when method = 'independent'. If pit is supplied, defaults to
length(pit), otherwise yrep determines the maximum accuracy of the
estimated PIT values and K is set to min(nrow(yrep) + 1, 1000).
Values between 0 and 1 indicating the desired
probability mass to include in the inner and outer intervals. The defaults
are prob=0.5 and prob_outer=0.9 for ppc_loo_intervals() and
prob = 0.99 for ppc_loo_pit_ecdf().
A boolean defining whether to plot the difference between
the observed PIT-ECDF and the theoretical expectation for uniform PIT
values rather than plotting the regular ECDF. For ppc_loo_pit_ecdf(),
when method = 'independent'. The default is FALSE, but for large
samples we recommend setting plot_diff = TRUE to better use the plot area.
A boolean defining if the simultaneous confidence
bands should be interpolated based on precomputed values rather than
computed exactly. Computing the bands may be computationally intensive and
the approximation gives a fast method for assessing the ECDF trajectory.
For ppc_loo_pit_ecdf() when method = 'independent'.
The default is to use interpolation if K is greater than 200.
The method used to calculate the uniformity test:
"independent": assumes independent PIT values (Säilynoja et al., 2022).
"correlated": accounts for correlated PIT values (Tesso & Vehtari, 2026).
When method = "correlated", which dependence-aware test to use:
"POT", "PRIT", or "PIET". Defaults to "POT".
When method = "correlated", tolerance threshold controlling
how strongly suspicious points are flagged. Larger values (gamma > 0)
emphasize points with larger deviations. If NULL, defaults to 0 and
thus all suspicious points are flagged.
When method = "correlated", a named character vector of
plot colors with elements ecdf and highlight. ecdf is used for the
main ECDF line; highlight for suspicious regions flagged by the
uniformity test. Defaults to c(ecdf = "grey60", highlight = "red").
Values must be valid ggplot2 colors (e.g. R color names or hex codes).
Ignored when method = "independent".
When method = "correlated", a boolean defining whether
to add information about p-value to the plot. Defaults to TRUE.
A boolean defining whether to compute PIT values using
Pareto-PIT method. Defaults to TRUE if test is either "POT" or
"PIET" and no pit values are provided otherwise FALSE. This argument
should normally not be modified by the user, except for development
purposes. If pit is non-NULL, pareto_pit cannot be simultaneously
TRUE.
When method = "correlated", a numeric value
between 0 and 1 defining the factor by which the help-text (p-value
information) is scaled. The default is 0.8.
For ppc_loo_intervals() and ppc_loo_ribbon(), an optional
integer vector indicating which observations in y (and yrep) to
include. Dropping observations from y and yrep manually before passing
them to the plotting function will not work because the dimensions will not
match up with the dimensions of psis_object, but if all of y and yrep
are passed along with subset then bayesplot can do the subsetting
internally for y, yrep and psis_object. See the Examples
section for a demonstration.
For ppc_loo_intervals() and ppc_loo_ribbon(), optionally
a matrix of pre-computed LOO predictive intervals that can be specified
instead of yrep (ignored if intervals is specified). If not specified
the intervals are computed internally before plotting. If specified,
intervals must be a matrix with number of rows equal to the number of
data points and five columns in the following order: lower outer interval,
lower inner interval, median (50%), upper inner interval and upper outer
interval (column names are ignored).
For ppc_loo_intervals(), a string indicating how to arrange
the plotted intervals. The default ("index") is to plot them in the
order of the observations. The alternative ("median") arranges them
by median value from smallest (left) to largest (right).
ppc_loo_pit_data()This function prepares LOO-PIT data for plotting with ggplot2. It is
the data-preparation back end for the LOO-PIT plotting functions
(ppc_loo_pit_overlay(), ppc_loo_pit_qq(), and ppc_loo_pit_ecdf()),
and users can call it directly to create custom LOO-PIT plots using
ggplot2. The function computes the leave-one-out probability integral
transform (LOO-PIT) values and returns a data frame that can be used to
build ggplot objects. This is useful when you want to create custom
visualizations of LOO-PIT values beyond what the built-in plotting
functions provide.
ppc_loo_pit_overlay(), ppc_loo_pit_qq(), ppc_loo_pit_ecdf()The calibration of marginal predictions can be assessed using probability integral transformation (PIT) checks. LOO improves the check by avoiding the double use of data. See the section on marginal predictive checks in Gelman et al. (2013, p. 152--153) and section 5 of Gabry et al. (2019) for an example of using bayesplot for these checks.
The LOO PIT values are asymptotically uniform (for continuous data) if the
model is calibrated. The ppc_loo_pit_overlay() function creates a plot
comparing the density of the LOO PITs (thick line) to the density estimates
of many simulated data sets from the standard uniform distribution (thin
lines). See Gabry et al. (2019) for an example of interpreting the shape of
the miscalibration that can be observed in these plots.
The ppc_loo_pit_qq() function provides an alternative visualization of
the miscalibration with a quantile-quantile (Q-Q) plot comparing the LOO
PITs to the standard uniform distribution. Comparing to the uniform is not
good for extreme probabilities close to 0 and 1, so it can sometimes be
useful to set the compare argument to "normal", which will
produce a Q-Q plot comparing standard normal quantiles calculated from the
PIT values to the theoretical standard normal quantiles. This can help see
the (mis)calibration better for the extreme values. However, in most cases
we have found that the overlaid density plot (ppc_loo_pit_overlay())
function will provide a clearer picture of calibration problems than the
Q-Q plot.
The ppc_loo_pit_ecdf() function visualizes the empirical cumulative
distribution function (ECDF) of the LOO PIT values.
With method = "independent", the plot overlays 100 * prob%
simultaneous confidence intervals for a standard uniform sample.
With method = "correlated", the plot uses a dependence-aware
uniformity assessment and can highlight suspicious regions.
Setting plot_diff = TRUE displays the ECDF minus the theoretical
expectation, which can improve visual assessment of calibration.
Note that the default "independent" method is superseded by
the "correlated" method (Tesso & Vehtari, 2026) which accounts for dependent
LOO-PIT values.
ppc_loo_intervals(), ppc_loo_ribbon()Similar to ppc_intervals() and ppc_ribbon() but the intervals are for
the LOO predictive distribution.
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. B. (2013). Bayesian Data Analysis. Chapman & Hall/CRC Press, London, third edition. (p. 152--153)
Gabry, J. , Simpson, D. , Vehtari, A. , Betancourt, M. and Gelman, A. (2019), Visualization in Bayesian workflow. J. R. Stat. Soc. A, 182: 389-402. doi:10.1111/rssa.12378. (journal version, arXiv preprint, code on GitHub)
Vehtari, A., Gelman, A., and Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing. 27(5), 1413--1432. doi:10.1007/s11222-016-9696-4. arXiv preprint: https://arxiv.org/abs/1507.04544
Boneva, L. I., Kendall, D., & Stefanov, I. (1971). Spline transformations: Three new diagnostic aids for the statistical data-analyst. J. R. Stat. Soc. B (Methodological), 33(1), 1-71. https://www.jstor.org/stable/2986005.
Tesso, H., & Vehtari, A. (2026). LOO-PIT predictive model checking. arXiv preprint https://arxiv.org/abs/2603.02928.
Other PPCs:
PPC-calibration,
PPC-censoring,
PPC-discrete,
PPC-distributions,
PPC-errors,
PPC-intervals,
PPC-overview,
PPC-scatterplots,
PPC-test-statistics
if (FALSE) {
library(rstanarm)
library(loo)
head(radon)
fit <- stan_lmer(
log_radon ~ floor + log_uranium + floor:log_uranium
+ (1 + floor | county),
data = radon,
iter = 100,
chains = 2,
cores = 2
)
y <- radon$log_radon
yrep <- posterior_predict(fit)
loo1 <- loo(fit, save_psis = TRUE, cores = 4)
psis1 <- loo1$psis_object
lw <- weights(psis1) # normalized log weights
# marginal predictive check using LOO probability integral transform
color_scheme_set("orange")
ppc_loo_pit_overlay(y, yrep, lw = lw)
ppc_loo_pit_qq(y, yrep, lw = lw)
ppc_loo_pit_qq(y, yrep, lw = lw, compare = "normal")
# predictive calibration check using LOO probability integral transform
ppc_loo_pit_ecdf(y, yrep, lw)
# With `plot_diff = TRUE` it is easier to assess the calibration.
ppc_loo_pit_ecdf(y, yrep, lw, plot_diff = TRUE)
# can use the psis object instead of lw
ppc_loo_pit_qq(y, yrep, psis_object = psis1)
# loo predictive intervals vs observations
keep_obs <- 1:50
ppc_loo_intervals(y, yrep, psis_object = psis1, subset = keep_obs)
color_scheme_set("gray")
ppc_loo_intervals(y, yrep,
psis_object = psis1, subset = keep_obs,
order = "median"
)
}
Run the code above in your browser using DataLab