trendeval (version 0.1.0)

evaluate_resampling: Resampling approach for model evaluation

Description

evaluate_resampling() uses repeated K-fold cross-validation and the Root Mean Square Error (RMSE) of testing sets to measure the predictive power of a single model. Methods are provided for trending::trending_model (and lists of these) objects.

Usage

evaluate_resampling(x, ...)
# S3 method for default
evaluate_resampling(x, ...)
# S3 method for trending_model
evaluate_resampling(
  x,
  data,
  metric = c("rmse", "rsq", "mae"),
  metric_arguments = list(na.rm = TRUE),
  v = nrow(data),
  repeats = 1,
  ...
)
# S3 method for list
evaluate_resampling(
  x,
  data,
  metric = c("rmse", "rsq", "mae"),
  metric_arguments = list(na.rm = TRUE),
  v = nrow(data),
  repeats = 1,
  ...
)

Arguments

x

An R object.

...

Not currently used.

data

a data.frame containing data (including the response variable and all predictors) used in the specified model.

metric

One of "rmse" (see calculate_rmse), "mae" (see calculate_mae) and "rsq" (see calculate_rsq).

metric_arguments

A named list of arguments passed to the underlying functions that calculate the metrics.

v

the number of equally sized data partitions to be used for K-fold cross-validation; v cross-validations will be performed, each using v - 1 partition as training set, and the remaining partition as testing set. Defaults to the number of row in data, so that the method uses leave-one-out cross validation, akin to Jackknife except that the testing set (and not the training set) is used to compute the fit statistics.

repeats

the number of times the random K-fold cross validation should be repeated for; defaults to 1; larger values are likely to yield more reliable / stable results, at the expense of computational time

Details

These functions wrap around existing functions from several packages. evaluate_resampling.trending_model() and evaluate_resampling.list() both use rsample::vfold_cv() for sampling and, for the calculating the different metrics, the yardstick package.

See Also

calculate_aic(), calculate_rmse(), calculate_mae() and calculate_rsq().

Examples

x <- rnorm(100, mean = 0)
y <- rpois(n = 100, lambda = exp(x + 1))
dat <- data.frame(x = x, y = y)
model <- trending::glm_model(y ~ x, poisson)
models <- list(
  poisson_model = trending::glm_model(y ~ x, poisson),
  linear_model = trending::lm_model(y ~ x)
)

evaluate_resampling(model, dat)
evaluate_resampling(models, dat)