confMatrixMetrics: Obtain several metrics based on the confusion matrix

Description

Calculates a range of metrics based upon the confusion matrix: accuracy, true positive rate (TPR; sensitivity or recall), true negative rate (specificity), false postive rate (FPR), false negative rate (FPR), F1-score , with the optional ability to dynamically determine an incidence-based cutoff value using validation sample predictions.

Usage

confMatrixMetrics(predTest, depTest, cutoff = 0.5, dyn.cutoff = FALSE,
  predVal = NULL, depVal = NULL)

Arguments

predTest

Vector with predictions (real-valued or discrete)

depTest

Vector with real class labels

cutoff

Threshold for converting real-valued predictions into class predictions. Default 0.5.

dyn.cutoff

Logical indicator to enable dynamic threshold determination using validation sample predictions. In this case, the function determines, using validation data, the incidence (occurrence percentage of the customer behavior or characterstic of interest) and chooses a cutoff value so that the number of predicted positives is equal to the number of true positives. If TRUE, then the value for the cutoff parameter is ignored.

predVal

Vector with predictions (real-valued or discrete). Only used if dyn.cutoff is TRUE.

depVal

Optional vector with true class labels for validation data. Only used if dyn.cutoff is TRUE.

Value

A list with the following items:

accuracy

accuracy value

truePostiveRate

TPR or true positive rate

trueNegativeRate

TNR or true negative rate

falsePostiveRate

FPR or false positive rate

falseNegativeRate

FNR or false negative rate

F1Score

F1-score

cutoff

the threshold value used to convert real-valued predictions to class predictions

References

Witten, I.H., Frank, E. (2005): Data Mining: Practical Machine Learning Tools and Techniques, Second Edition. Chapter 5. Morgan Kauffman.

Examples

Run this code

# NOT RUN {
## Load response modeling data set
data("response")
## Apply confMatrixMetrics function to obtain confusion matrix-based performance metrics
## achieved on the test sample. Use validation sample predictions to dynamically
## determine a cutoff value.
cmm<-confMatrixMetrics(response$test[,2],response$test[,1],dyn.cutoff=TRUE,
predVal=response$val[,2],depVal=response$val[,1])
## Retrieve F1-score
print(cmm$F1Score)

# }