gp.model.adequacy: Model assessment based on Deviance information criterion (DIC), logarithmic pointwise predictive density (lppd), and logarithmic joint predictive density (ljpd).

Description

This function computes effective number of parameters (pD), deviance information criterion (DIC), logarithmic pointwise predictive density (lppd), and logarithmic joint predictive density (ljpd). For detailed introduction of these metrics, see Chapter 7 of Gelman et al. (2013).

The deviance function for a model with a vector of parameters $\boldsymbol \theta$ is defined as $$ D(\boldsymbol \theta) = -2\log p(\mathbf{y} \mid \boldsymbol \theta), $$ where $\mathbf{y}:=(y(\mathbf{x}_1), \ldots, y(\mathbf{x}_n))^\top$ is a vector of $n$ observations.

The effective number of parameters (see p.172 of Gelman et al. 2013) is defined as $$ pD = E_{\boldsymbol \theta| \mathbf{y}}[D(\boldsymbol \theta)] - D(\hat{ \boldsymbol \theta }), $$ where $\hat{\boldsymbol \theta} = E_{\boldsymbol \theta | \mathbf{y}}[\boldsymbol \theta]. $ The interpretation is that the effective number of parameters is the ``expected" deviance minus the ``fitted" deviance. Higher $pD$ implies more over-fitting with estimate $\hat{\boldsymbol \theta}$.
The Deviance information criteria (DIC) (see pp. 172-173 of Gelman et al. 2013) is $$DIC = E_{\boldsymbol \theta | \mathbf{y}}[D(\boldsymbol \theta)] + pD. $$ DIC approximates Akaike information criterion (AIC) and is more appropriate for hierarchical models than AIC and BIC.
The log predictive density (lpd) is defined as $$ p(y(\mathbf{x}_0) \mid \mathbf{y}) = \int p(y(\mathbf{x}_0) \mid \boldsymbol \theta, \mathbf{y}) p(\boldsymbol \theta \mid \mathbf{y}) d \boldsymbol \theta, $$ where $\mathbf{y}:=(y(\mathbf{x}_1), \ldots, y(\mathbf{x}_n))^\top$ is a vector of $n$ observations. $\boldsymbol \theta$ contains correlation parameters and nugget parameter. This predictive density should be understood as an update of the likelihood since data is treated as prior information now. With a set of prediction locations $\mathcal{X}:=\{x_0^i: i=1, \ldots, m\}$. The log pointwise predictive density (lppd) is defined as $$lppd = \sum_{i=1}^m \log p(y(\mathbf{x}_0^i) \mid \mathbf{y}).$$ The log joint predictive density (ljpd) is defined as $$ljpd = \log p(y(\mathcal{X})). $$ The lppd is connected to cross-validation, while the ljpd measures joint uncertainty across prediction locations.

Usage

gp.model.adequacy(
  obj,
  testing.input,
  testing.output,
  pointwise = TRUE,
  joint = TRUE
)

Value

a list containingg pD, DIC, lppd, ljpd.

Arguments

obj: a gp object.
testing.input: a matrix of testing inputs
testing.output: a vector of testing outputs
pointwise: a logical value with default value TRUE. If it is TRUE, lppd is calculated.
joint: a logical value with default value TRUE. If it is TRUE, ljpd is calculated.

Author

Pulong Ma mpulong@gmail.com

References