This function data averaged posterior
calibrationTest(posteriorList, priorDraws, ...)a list with posterior samples. List items must be of a class that is supported by getSample
a matrix with parameter values, drawn from the prior, that were used to simulate the data underlying the posteriorList. If colnames are provided, these will be used in the plots
arguments to be passed to getSample. Consider in particular the thinning option.
The purpose of this function is to evaluate the results of a simulation-based calibration of an MCMC analysis.
Briefly, the idea is to repeatedly
1. sample parameters from the prior, 2. simulate new data based on these parameters, 3. calculate the posterior for these data
If the sampler and the likelihood are implemented correctly, the average of over all the posterior distribution should then again yield the prior (e.g. Cook et al., 2006).
To test if this is the case, we implement the methods suggested by Talts et al., which is to calculate the rank statistics between the parameter draws and the posterior draws, which we then formally evaluate with a qq unif plot, and a ks.test
We speculate that a ks.test between the two distribution would likely give an identical result, but this is not noted in Talts et al.
Cook, S. R., Gelman, A. and Rubin, D. B. (2006). Validation of Software for Bayesian Models Using Posterior Quantiles. J. Comput. Graph. Stat. 15 675-692.
Talts, Sean, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. "Validating Bayesian Inference Algorithms with Simulation-Based Calibration." arXiv preprint arXiv:1804.06788 (2018).