kcv: Perform k-fold cross validation

Description

Perform k-fold cross validation with options for computationally efficient approximations for big data. Generalizes loocv() (leave-one-out cross validation) to leaving out k folds of (approximately) equal size instead of single observations.

Usage

kcv(object, ...)
# S3 method for splm
kcv(
  object,
  k = 5,
  cv_predict = FALSE,
  se.fit = FALSE,
  local,
  interval = c("none", "prediction"),
  level = 0.95,
  folds_index,
  ...
)
# S3 method for spautor
kcv(
  object,
  k = 5,
  cv_predict = FALSE,
  se.fit = FALSE,
  local,
  interval = c("none", "prediction"),
  level = 0.95,
  folds_index,
  ...
)
# S3 method for spglm
kcv(
  object,
  k = 5,
  cv_predict = FALSE,
  type = c("link", "response"),
  se.fit = FALSE,
  delta = FALSE,
  local,
  folds_index,
  ...
)
# S3 method for spgautor
kcv(
  object,
  k,
  cv_predict = FALSE,
  type = c("link", "response"),
  se.fit = FALSE,
  delta = FALSE,
  local,
  folds_index,
  ...
)

Value

If cv_predict = FALSE and se.fit = FALSE, a fit statistics tibble (with bias, MSPE, RMSPE, and cor2; see Details). If cv_predict = TRUE or se.fit = TRUE, a list with elements: stats, a fit statistics tibble (with bias, MSPE, RMSPE, and cor2; see Details); cv_predict, a numeric vector with k-fold cross validation predictions for each observation (if cv_predict = TRUE); and se.fit, a numeric vector with k-fold cross validation prediction standard errors for each observation (if se.fit = TRUE). When object is from splm() or spautor() and interval = "prediction", the fit statistics tibble also has a cover.XX column (e.g. cover.95

for level = 0.95; see Details).

Arguments

object: A fitted model object from splm(), spautor(), spglm(), or spgautor().
...: Other arguments. Not used (needed for generic consistency).
k: The number of folds. Must be a whole number at least 2 and no more than the sample size (the number of non-missing observations in object). The default is 5. If k equals the sample size, kcv() is equivalent to (and calls) loocv(). Ignored when folds_index is supplied.
cv_predict: A logical indicating whether the k-fold cross validation fitted values should be returned. Defaults to FALSE. If object is from spglm() or spgautor(), the fitted values returned are on the link scale.
se.fit: A logical indicating whether the k-fold cross validation prediction standard errors should be returned. Defaults to FALSE. If object is from spglm() or spgautor(), the standard errors correspond to the fitted values returned on the link scale.
local: A list or logical. If a list, specific list elements described in predict.spmodel() control the big data approximation behavior. If a logical, TRUE chooses default list elements for the list version of local as specified in predict.spmodel(). Defaults to FALSE, which performs exact computations.
interval: Whether to also report empirical k-fold cross validation prediction interval coverage in the returned fit statistics. "none" (the default) omits it; "prediction" reports it (see Details). Only available for splm()/spautor() objects.
level: The prediction interval level (e.g. 0.95) used to compute prediction interval coverage when interval = "prediction". Ignored otherwise. The default is 0.95.
folds_index: An optional vector, the same length as the number of non-missing observations in object and in the same order, assigning each observation to a fold (at least two distinct values required). When supplied, this fold assignment is used as-is, implying k is ignored and no random fold assignment is performed. The default is NULL, which randomly assigns k (approximately) equally sized folds.
type: The scale (response or link) of predictions obtained when cv_predict = TRUE and using spglm() or spgautor objects.
delta: A logical indicating whether to return delta method standard errors on the response scale when se.fit = TRUE and type = "response". The default is FALSE.

Details

Observations are randomly partitioned into k folds of (approximately) equal size. Each fold is held out from the data set in turn and the remaining data are used to make predictions for the held-out fold. This is compared to the true values of the held-out observations and several fit statistics are (sometimes optionally) computed: bias, mean-squared-prediction error (MSPE), root-mean-squared-prediction error (RMSPE), and the squared correlation (cor2) between the observed data and k-fold cross validation predictions (regarded as a prediction version of r-squared appropriate for comparing across spatial and nonspatial models), , and prediction interval coverage (cover.XX). Generally, bias should be near zero and prediction interval coverage at the intended level for well-fitting models. The lower the MSPE and RMSPE, the better the model fit (according to the k-fold cross validation criterion). The higher the cor2, the better the model fit (according to the k-fold cross validation criterion). cor2 and cover.XX are not returned when object was fit using spglm() or spgautor() because we do not observe the underlying latent mean.

When object is from splm() or spautor(), setting interval = "prediction" additionally reports the empirical coverage of the level (e.g. 95\ the proportion of held-out observations whose true value falls within fit +/- qnorm(1 - (1 - level) / 2) * se.fit (the same normal-quantile interval predict.spmodel() uses by default). This is only available for splm()/spautor() objects, since spglm()/spgautor() have no observed-scale latent mean to compare against.

Examples

Run this code

spmod <- splm(z ~ water + tarp,
  data = caribou,
  spcov_type = "exponential", xcoord = x, ycoord = y
)
kcv(spmod)
kcv(spmod, k = 10, cv_predict = TRUE, se.fit = TRUE)

Run the code above in your browser using DataLab