fdm2id (version 1.0.1)

performance: Performance estimation

Description

Estimate the performance of classification or regression methods using bootstrap or crossvalidation (accuracy, ROC curves, confusion matrices, ...)

Usage

performance(
  methods,
  train.x,
  train.y,
  test.x = NULL,
  test.y = NULL,
  train.size = round(0.7 * nrow(train.x)),
  type = c("evaluation", "confusion", "roc", "cost", "scatter", "avsp"),
  protocol = c("bootstrap", "crossvalidation", "loocv", "holdout", "train"),
  eval = ifelse(is.factor(train.y), "accuracy", "r2"),
  nruns = 10,
  nfolds = 10,
  new = TRUE,
  lty = 1,
  seed = NULL,
  methodparameters = NULL,
  names = NULL,
  fuzzy = FALSE,
  positive = NULL,
  stratify = TRUE,
  ...
)

Value

The evaluation of the predictions (numeric value).

Arguments

methods

The classification or regression methods to be evaluated.

train.x

The dataset (description/predictors), a matrix or data.frame.

train.y

The target (class labels or numeric values), a factor or vector.

test.x

The test dataset (description/predictors), a matrix or data.frame. Giving test.x and test.y is the simplest use of this function: each method is fitted on (train.x, train.y), used to predict test.x, and its predictions are compared with test.y -- no resampling at all. protocol then defaults to "holdout", and any other protocol raises an error, since it would ignore the test set.

test.y

The (test) target (class labels or numeric values), a factor or vector.

train.size

The size of the training set, for protocol = "holdout" without an explicit test set: either a number of observations, or a proportion between 0 and 1.

type

The type of evaluation (confusion matrix, ROC curve, ...)

protocol

How the performance is estimated.

"bootstrap"

(default) nruns bootstrap samples of the training set, each model evaluated on the observations left out of its sample.

"crossvalidation"

nruns repetitions of a nfolds-fold cross-validation of the training set.

"loocv"

leave-one-out cross-validation.

"holdout"

a single train/test split. Uses test.x/test.y when they are given -- see test.x -- and otherwise draws a training set of train.size observations and evaluates on the rest.

"train"

evaluates each model on the very data it was fitted on. Optimistic by construction; useful to show students exactly that.

eval

The evaluation functions.

nruns

The number of bootstrap runs.

nfolds

The number of folds (crossvalidation estimation).

new

A logical value indicating whether a new plot should be created or not (cost curves or ROC curves).

lty

The line type (and color) specified as an integer (cost curves or ROC curves).

seed

A specified seed for random number generation (useful for testing different method with the same bootstap samplings).

methodparameters

Method parameters (if null tuning is done by cross-validation).

names

Method names.

fuzzy

Used by type = "roc" and type = "cost" only. FALSE by default: the curves are built from the hard class labels, which reduces them to three points. Pass fuzzy = TRUE to build them from the estimated probabilities of the positive class, which is what a ROC curve is meant to show. See roc.curves.

positive

The label of the positive class. Used by type = "roc" and type = "cost" to orient the curves, and passed on to evaluation for the criteria that are defined on one class -- precision, recall, the F-measure and the other measures taking an average argument -- so that a two-class problem can be scored on either of its classes. Defaults to the first level of the target, which is worth setting explicitly whenever the class of interest is not the first one.

stratify

Whether the splits should preserve the proportions of the classes (TRUE, the default), for protocol = "crossvalidation" and for protocol = "holdout" when it draws its own split. Ignored for a numeric target, and for the bootstrap and leave-one-out protocols, which have no split to stratify.

...

Other specific parameters for the leaning method.

See Also

confusion, evaluation, cost.curves, roc.curves

Examples

if (FALSE) {
require ("datasets")
data (iris)
# The simplest use: a training set, a test set, and the score of the model fitted on the
# first and evaluated on the second. Same thing as
# evaluation.accuracy (predict (NB (d$train.x, d$train.y), d$test.x), d$test.y).
d = splitdata (iris, 5, seed = 0)
performance (NB, d$train.x, d$train.y, d$test.x, d$test.y)
# Several methods and criteria at once
performance (c (NB, LDA, CART), d$train.x, d$train.y, d$test.x, d$test.y,
             eval = c ("accuracy", "kappa"))
# One method, one evaluation criterion, bootstrap estimation
performance (NB, iris [, -5], iris [, 5], seed = 0)
# One method, two evaluation criteria, train set estimation
performance (NB, iris [, -5], iris [, 5], eval = c ("accuracy", "kappa"),
             protocol = "train", seed = 0)
# Three methods, ROC curves, LOOCV estimation
data (linsep)
performance (c (NB, LDA, LR), linsep [, -3], linsep [, 3], type = "roc",
             protocol = "loocv", seed = 0)
# Same curves, read from the hard predicted labels instead of the class-membership
# scores: each method collapses to a single operating point.
performance (c (NB, LDA, LR), linsep [, -3], linsep [, 3], type = "roc",
             protocol = "loocv", seed = 0, fuzzy = FALSE)
# Choosing the positive class explicitly
performance (NB, linsep [, -3], linsep [, 3], type = "roc", protocol = "loocv",
             seed = 0, positive = levels (linsep [, 3]) [2])
# List of methods in a variable, confusion matrix, hodout estimation
classif = c (NB, LDA, LR)
performance (classif, iris [, -5], iris [, 5], type = "confusion",
             protocol = "holdout", seed = 0, names = c ("NB", "LDA", "LR"))
# List of strings (method names), scatterplot evaluation, crossvalidation estimation
classif = c ("NB", "LDA", "LR")
performance (classif, iris [, -5], iris [, 5], type = "scatter",
             protocol = "crossvalidation", seed = 0)
# Actual vs. predicted
data (trees)
performance (LINREG, trees [, -3], trees [, 3], type = "avsp")
}