confusionMatrix: Create a confusion matrix

Description

Calculates a cross-tabulation of observed and predicted classes with associated statistics.

Usage

confusionMatrix(data, ...)
# S3 method for default
confusionMatrix(
  data,
  reference,
  positive = NULL,
  dnn = c("Prediction", "Reference"),
  prevalence = NULL,
  mode = "sens_spec",
  ...
)
# S3 method for matrix
confusionMatrix(
  data,
  positive = NULL,
  prevalence = NULL,
  mode = "sens_spec",
  ...
)
# S3 method for table
confusionMatrix(
  data,
  positive = NULL,
  prevalence = NULL,
  mode = "sens_spec",
  ...
)

Value

a list with elements

table: the results of table on data and reference
positive: the positive result level
overall: a numeric vector with overall accuracy and Kappa statistic values
byClass: the sensitivity, specificity, positive predictive value, negative predictive value, precision, recall, F1, prevalence, detection rate, detection prevalence and balanced accuracy for each class. For two class systems, this is calculated once using the positive argument

Arguments

data: a factor of predicted classes (for the default method) or an object of class table.
...: options to be passed to table. NOTE: do not include dnn here
reference: a factor of classes to be used as the true results
positive: an optional character string for the factor level that corresponds to a "positive" result (if that makes sense for your data). If there are only two factor levels, the first level will be used as the "positive" result. When mode = "prec_recall", positive is the same value used for relevant for functions precision, recall, and F_meas.table.
dnn: a character vector of dimnames for the table
prevalence: a numeric value or matrix for the rate of the "positive" class of the data. When data has two levels, prevalence should be a single numeric value. Otherwise, it should be a vector of numeric values with elements for each class. The vector should have names corresponding to the classes.
mode: a single character string either "sens_spec", "prec_recall", or "everything"

Author

Max Kuhn

Details

The functions requires that the factors have exactly the same levels.

For two class problems, the sensitivity, specificity, positive predictive value and negative predictive value is calculated using the positive argument. Also, the prevalence of the "event" is computed from the data (unless passed in as an argument), the detection rate (the rate of true events also predicted to be events) and the detection prevalence (the prevalence of predicted events).

Suppose a 2x2 table with notation

	Reference
Predicted	Event	No Event
Event	A	B
No Event	C	D

The formulas used here are: $S e n s i t i v i t y = A / (A + C)$ $S p e c i f i c i t y = D / (B + D)$ $P r e v a l e n c e = (A + C) / (A + B + C + D)$ $P P V = (s e n s i t i v i t y * p r e v a l e n c e) / ((s e n s i t i v i t y * p r e v a l e n c e) + ((1 - s p e c i f i c i t y) * (1 - p r e v a l e n c e)))$ $N P V = (s p e c i f i c i t y * (1 - p r e v a l e n c e)) / (((1 - s e n s i t i v i t y) * p r e v a l e n c e) + ((s p e c i f i c i t y) * (1 - p r e v a l e n c e)))$ $D e t e c t i o n R a t e = A / (A + B + C + D)$ $D e t e c t i o n P r e v a l e n c e = (A + B) / (A + B + C + D)$ $B a l a n c e d A c c u r a c y = (s e n s i t i v i t y + s p e c i f i c i t y) / 2$

$P r e c i s i o n = A / (A + B)$ $R e c a l l = A / (A + C)$ $F 1 = (1 + b e t a^{2}) * p r e c i s i o n * r e c a l l / ((b e t a^{2} * p r e c i s i o n) + r e c a l l)$

where beta = 1 for this function.

See the references for discussions of the first five formulas.

For more than two classes, these results are calculated comparing each factor level to the remaining levels (i.e. a "one versus all" approach).

The overall accuracy and unweighted Kappa statistic are calculated. A p-value from McNemar's test is also computed using mcnemar.test (which can produce NA values with sparse tables).

The overall accuracy rate is computed along with a 95 percent confidence interval for this rate (using binom.test) and a one-sided test to see if the accuracy is better than the "no information rate," which is taken to be the largest class percentage in the data.

References

Kuhn, M. (2008), ``Building predictive models in R using the caret package, '' Journal of Statistical Software, (tools:::Rd_expr_doi("10.18637/jss.v028.i05")).

Altman, D.G., Bland, J.M. (1994) ``Diagnostic tests 1: sensitivity and specificity,'' British Medical Journal, vol 308, 1552.

Altman, D.G., Bland, J.M. (1994) ``Diagnostic tests 2: predictive values,'' British Medical Journal, vol 309, 102.

Velez, D.R., et. al. (2008) ``A balanced accuracy function for epistasis modeling in imbalanced datasets using multifactor dimensionality reduction.,'' Genetic Epidemiology, vol 4, 306.

Examples

Run this code


###################
## 2 class example

lvs <- c("normal", "abnormal")
truth <- factor(rep(lvs, times = c(86, 258)),
                levels = rev(lvs))
pred <- factor(
               c(
                 rep(lvs, times = c(54, 32)),
                 rep(lvs, times = c(27, 231))),
               levels = rev(lvs))

xtab <- table(pred, truth)

confusionMatrix(xtab)
confusionMatrix(pred, truth)
confusionMatrix(xtab, prevalence = 0.25)

###################
## 3 class example

confusionMatrix(iris$Species, sample(iris$Species))

newPrior <- c(.05, .8, .15)
names(newPrior) <- levels(iris$Species)

confusionMatrix(iris$Species, sample(iris$Species))

Run the code above in your browser using DataLab

Data Engineering and BI courses are free this week!