Given two numeric vectors of data, the mean squared error and R-squared are calculated. For two factors, the overall agreement rate and Kappa are determined.

`defaultSummary(data, lev = NULL, model = NULL)`postResample(pred, obs)

twoClassSummary(data, lev = NULL, model = NULL)

mnLogLoss(data, lev = NULL, model = NULL)

multiClassSummary(data, lev = NULL, model = NULL)

prSummary(data, lev = NULL, model = NULL)

data

a data frame with columns `obs`

and
`pred`

for the observed and predicted outcomes. For metrics
that rely on class probabilities, such as
`twoClassSummary`

, columns should also include predicted
probabilities for each class. See the `classProbs`

argument
to `trainControl`

.

lev

a character vector of factors levels for the
response. In regression cases, this would be `NULL`

.

model

a character string for the model name (as taken
from the `method`

argument of `train`

.

pred

A vector of numeric data (could be a factor)

obs

A vector of numeric data (could be a factor)

A vector of performance estimates.

`postResample`

is meant to be used with `apply`

across a matrix. For numeric data the code checks to see if the
standard deviation of either vector is zero. If so, the
correlation between those samples is assigned a value of zero.
`NA`

values are ignored everywhere.

Note that many models have more predictors (or parameters) than
data points, so the typical mean squared error denominator (n -
p) does not apply. Root mean squared error is calculated using
`sqrt(mean((pred - obs)^2`

. Also, \(R^2\) is calculated
wither using as the square of the correlation between the
observed and predicted outcomes when `form = "corr"`

. when
`form = "traditional"`

, $$ R^2 = 1-\frac{\sum (y_i -
\hat{y}_i)^2}{\sum (y_i - \bar{y}_i)^2} $$. Mean absolute error
is calculated using `mean(abs(pred-obs))`

.

`defaultSummary`

is the default function to compute
performance metrics in `train`

. It is a wrapper
around `postResample`

. The first argument is `data`

,
which is `data.frame`

with columns named `obs`

and
`pred`

for the observed and predicted outcome values
(either numeric data for regression or character values for
classification). The second argument is `lev`

, a character
string that has the outcome factor levels or NULL for a
regression model. The third parameter is `model`

, which can
be used if a summary metric is specific to a model function. If
other columns from the data are required to compute the summary
statistics, but should not be used in the model, the
`recipe`

method for `train`

can be used.

`twoClassSummary`

computes sensitivity, specificity and
the area under the ROC curve. `mnLogLoss`

computes the
minus log-likelihood of the multinomial distribution (without
the constant term): $$ -logLoss = \frac{-1}{n}\sum_{i=1}^n
\sum_{j=1}^C y_{ij} \log(p_{ij}) $$ where the `y`

values are
binary indicators for the classes and `p`

are the predicted
class probabilities.

`prSummary`

(for precision and recall) computes values for
the default 0.50 probability cutoff as well as the area under
the precision-recall curve across all cutoffs and is labelled as
`"AUC"`

in the output. If assumes that the first level of
the factor variables corresponds to a relevant result but the
`lev`

argument can be used to change this.

`multiClassSummary`

computes some overall measures of for
performance (e.g. overall accuracy and the Kappa statistic) and
several averages of statistics calculated from "one-versus-all"
configurations. For example, if there are three classes, three
sets of sensitivity values are determined and the average is
reported with the name ("Mean_Sensitivity"). The same is true
for a number of statistics generated by
`confusionMatrix`

. With two classes, the basic
sensitivity is reported with the name "Sensitivity".

To use `twoClassSummary`

and/or `mnLogLoss`

, the
`classProbs`

argument of `trainControl`

should
be `TRUE`

. `multiClassSummary`

can be used without
class probabilities but some statistics (e.g. overall log loss
and the average of per-class area under the ROC curves) will not
be in the result set.

Other functions can be used via the `summaryFunction`

argument of `trainControl`

. Custom functions must
have the same arguments as`defaultSummary`

.

The function `getTrainPerf`

returns a one row data frame
with the resampling results for the chosen model. The statistics
will have the prefix "`Train`

" (i.e. "`TrainROC`

").
There is also a column called "`method`

" that echoes the
argument of the call to `trainControl`

of the same
name.

Kvalseth. Cautionary note about \(R^2\). American Statistician (1985) vol. 39 (4) pp. 279-285

# NOT RUN { predicted <- matrix(rnorm(50), ncol = 5) observed <- rnorm(10) apply(predicted, 2, postResample, obs = observed) classes <- c("class1", "class2") set.seed(1) dat <- data.frame(obs = factor(sample(classes, 50, replace = TRUE)), pred = factor(sample(classes, 50, replace = TRUE)), class1 = runif(50)) dat$class2 <- 1 - dat$class1 defaultSummary(dat, lev = classes) twoClassSummary(dat, lev = classes) prSummary(dat, lev = classes) mnLogLoss(dat, lev = classes) # }

Run the code above in your browser using DataCamp Workspace