mc.ci: Diagnostic plots for model choice coverage output

Description

Plots credible interval estimates for raw model choice output from cov.mc. This is used to investigate whether the coverage property holds and validate whether diagnostic statistics are acting as intended.

Usage

mc.ci(raw, tol, eps, modname, modtrue, nbins=5, 
bintype=c("interval", "quantile"), bw=FALSE, ...)

Arguments

raw

The raw item from the list output by cov.mc.

tol

The value of tol to test.

eps

The value of eps to test. This is used when tol is missing. One of eps and tol must be supplied.

modname

The name of the model to test.

modtrue

Vector containing the true models generating the pseudo-observed test data. i.e. modtrue[i] is the model generating dataset i.

nbins

Number of bins to display.

bintype

How to choose the bins (see Details).

Whether to produce a black and white image. Default is FALSE. Colour is used to make different bins stand out.

...

Additional plotting arguments - anything that can be used by plot.

Details

This function provides a plot which can be used as an informal test of the model choice coverage hypothesis for a particular value of eps or tol and choice of model. The plot is more flexible than the diagnostics, but not suitable as the basis of a formal test.

For each pseudo-observed data set, the ABC probability that the model is modname is taken from raw, and the true model is taken from modtrue. The probabilities are binned into nbins intervals, either of equal length or based on nbins+1 equally spaced empirical quantiles. The function estimates the observed probability of modname within each bin using Bayesian inference for a binomial proportion under a uniform prior. The plot shows the mean and 95% credible interval plotted against predicted probabilities. Informally, the coverage property should be rejected if predicted values are too unlikely given the observed values.

References

Nunes, M. A. and Prangle, D. (2016) abctools: an R package for tuning approximate Bayesian computation analyses. The R Journal 7, Issue 2, 189--205. Prangle D., Blum M. G. B., Popovic G., Sisson S. A. (2014) Diagnostic tools of approximate Bayesian computation using the coverage property. Australian and New Zealand Journal of Statistics 56, Issue 4, 309--329.

Examples

Run this code

  ##The examples below are chosen to run relatively quickly (<5 mins)
  ##and do not represent recommended tuning choices.
  ## Not run: 
#   index <- sample(1:2, 1E4, replace=TRUE)
#   sumstat <- ifelse(index==1, rnorm(1E4,0,1), rnorm(1E4,0,rexp(1E4,1)))
#   sumstat <- data.frame(ss=sumstat)
#   covdiag <- cov.mc(index=index, sumstat=sumstat, testsets=1:100, 
#   tol=seq(0.1,1,by=0.1), diagnostics=c("freq"))
#   mc.ci(covdiag$raw, tol=0.5, modname=1, modtrue=index[1:100])
#   ## End(Not run)