precrec (version 0.24.0)

evalmod: Evaluate models and calculate performance evaluation metrics

Description

The evalmod function calculates ROC and Precision-Recall curves for specified prediction scores and binary labels. It also calculate several basic performance evaluation metrics, such as accuracy, error rate, and precision, by specifying mode as "basic".

Usage

evalmod(
  mdat,
  mode = NULL,
  scores = NULL,
  labels = NULL,
  modnames = NULL,
  dsids = NULL,
  posclass = NULL,
  na_worst = TRUE,
  ties_method = "equiv",
  calc_avg = TRUE,
  cb_alpha = 0.05,
  raw_curves = FALSE,
  x_bins = 1000,
  interpolate = TRUE,
  beta = 1,
  on_single_class = "error",
  metrics = NULL,
  cost_fp = 1,
  cost_fn = 1,
  basic_ties = "split",
  ...
)

Value

The evalmod function returns an S3 object that contains performance evaluation metrics. The number of models and the number of datasets can be controlled by modnames and dsids. For example, the number of models is "single" and the number of test datasets is "multiple" when modnames = c("m1", "m1", "m1")

and dsids = c(1, 2, 3) are specified.

Different S3 objects have different default behaviors of S3

generics, such as plot(), autoplot(), and fortify().

1. The evalmod function returns one of the following S3

   objects when mode is "prcroc". The objects contain ROC and Precision-Recall curves.

   S3 object	# of models	# of test datasets
   sscurves	single	single
   mscurves	multiple	single
   smcurves	single	multiple
   mmcurves	multiple	multiple

2. The evalmod function returns one of the following S3

   objects when mode is "basic". They contain the per-rank basic evaluation metrics; error rate, accuracy, specificity, sensitivity, precision, Matthews correlation coefficient, F-score, balanced accuracy, negative predictive value, informedness, markedness, and Cohen's kappa.

   S3 object	# of models	# of test datasets
   sspoints	single	single
   mspoints	multiple	single
   smpoints	single	multiple
   mmpoints	multiple	multiple

3. The evalmod function returns the aucroc S3 object

   when mode is "aucroc", which can be used with 'print' and 'as.data.frame'.

Arguments

mdat

An S3 object created by the mmdata() function. It contains formatted scores and labels. The evalmod function ignores the following arguments when mdat is specified.

• scores

• labels

• modnames

• dsids

• posclass

• na_worst

• ties_method

These arguments are internally passed to the mmdata() function when mdat is unspecified. In that case, both scores and labels must be at least specified.

mode

A string that specifies the types of evaluation metrics that the evalmod function calculates.

"rocprc"

ROC and Precision-Recall curves

"prcroc"

Same as above

"basic"

Normalized ranks vs. accuracy, error rate, specificity, sensitivity, precision, Matthews correlation coefficient, F-score, balanced accuracy, negative predictive value, informedness, markedness, and Cohen's kappa.

"aucroc"

Fast AUC(ROC) calculation with the U statistic

scores

A numeric dataset of predicted scores. It can be a vector, a matrix, an array, a data frame, or a list. The join_scores() function can be useful to make scores with multiple datasets.

labels

A numeric, character, logical, or factor dataset of observed labels. It can be a vector, a matrix, an array, a data frame, or a list. The join_labels() function can be useful to make labels with multiple datasets.

modnames

A character vector for the names of the models. The evalmod function automatically generates default names as "m1", "m2", "m3", and so on when it is NULL.

dsids

A numeric vector for test dataset IDs. The evalmod function automatically generates the default ID as 1 when it is NULL.

posclass

A scalar value to specify the label of positives in labels. It must be the same data type as labels. For example, posclass = -1 changes the positive label from 1 to -1 when labels contains 1 and -1. The positive label will be automatically detected when posclass is NULL.

na_worst

A Boolean value for controlling the treatment of NAs in scores.

TRUE

All NAs are treated as the worst scores

FALSE

All NAs are treated as the best scores

ties_method

A string for controlling ties in scores.

"equiv"

Ties are equivalently ranked

"first"

Ties are ranked in an increasing order as appeared

"random"

Ties are ranked in random order

calc_avg

A logical value to specify whether average curves should be calculated. It is effective only when dsids contains multiple dataset IDs. For instance, the function calculates the average for the model "m1" when modnames is c("m1", "m1", "m1") and dsids is c(1, 2, 3). The calculation points are defined by x_bins.

cb_alpha

A numeric value with range [0, 1] to specify the alpha value of the point-wise confidence bounds calculation. It is effective only when calc_avg is set to TRUE. For example, it should be 0.05 for the 95% confidence level. The calculation points are defined by x_bins.

raw_curves

A logical value to specify whether all raw curves should be discarded after the average curves are calculated. It is effective only when calc_avg is set to TRUE.

x_bins

An integer value to specify the number of minimum bins on the x-axis. It is then used to define supporting points For instance, the x-values of the supporting points will be c(0, 0.5, 1) and c(0, 0.25, 0.5, 0.75, 1) when x_bins = 2 and x_bins = 4, respectively. All corresponding y-values of the supporting points are calculated. x_bins places supporting points only when mode is set to rocprc or prcroc; with mode = "basic" there is no interpolation to place them on, and the value is instead the number of points kept per metric when a plot or a data frame is asked for with reduce_points = TRUE. It must be 1e6 or smaller; every stage sized by it allocates a vector of that length per curve, and a million supporting points is already finer than a plot resolves.

interpolate

A Boolean value to specify whether or not interpolation of ROC and precision-recall curves are performed. x_bins and calc_avg are ignored and when x_bins is set to FALSE. interpolate is effective only when mode is set to rocprc or prcroc.

beta

A numeric value to specify the beta of the F-beta score, which weights recall beta times as heavily as precision. The default 1 gives the F1 score. beta is effective only when mode is set to basic.

on_single_class

A string that specifies what the evalmod function does with a dataset in which every label belongs to the same class.

"error"

Raise an error (default)

"na"

Warn, and return NA for the metrics that are undefined

ROC and precision-recall curves are undefined for such a dataset, so on_single_class is effective only when mode is set to rocprc, prcroc, or aucroc. mode = "basic" always warns and calculates what it can, because accuracy and error rate are still defined.

metrics

A character vector that names the basic evaluation metrics to calculate in addition to the default set, or the string "all" for every metric precrec knows. The default NULL is the fourteen metrics evalmod has always returned: score, label, error, accuracy, specificity, sensitivity, precision, mcc, fscore, balanced_accuracy, npv, informedness, markedness and kappa.

The metrics that can be added are fpr, fnr, false_discovery_rate, false_omission_rate, predicted_positive_rate, predicted_negative_rate, lift, odds, mi, chisq, cost and sar. They are the metrics ROCR provides that precrec did not, and each of them also answers to the identifier ROCR uses for it - fall, miss, pcfall, pcmiss, rpp, rnp and mutual_information - and to its standard abbreviation where it has one.

roc_dist and sedi can be added on the same footing. roc_dist is the distance from (1 - specificity, sensitivity) to the perfect corner of ROC space, and is the one metric here that is better when it is smaller. sedi is the symmetric extremal dependence index, a skill score built to stay informative when the positive class is rare.

jaccard, positive_likelihood_ratio and negative_likelihood_ratio come from scikit-learn. jaccard is the Jaccard index, also called the critical success index: TP / (TP + FP + FN), the confusion matrix with its true negative corner left out, which is the same omission precision and sensitivity make. The two likelihood ratios are sensitivity / fpr and fnr / specificity; odds, the diagnostic odds ratio, is their quotient.

They are not calculated by default because each is another vector the size of the dataset, and because plot and autoplot draw one panel per metric the object holds. A metric that was not asked for cannot be plotted; metrics is effective only when mode is set to basic.

cost_fp

A numeric value for the cost of a false positive, used by the cost metric. cost is not normalized, following ROCR: it is cost_fp * FP / n + cost_fn * FN / n, which with the default weights of 1 is the error rate. cost_fp is effective only when mode is set to basic and metrics asks for cost.

cost_fn

A numeric value for the cost of a false negative.

basic_ties

A string that specifies what the basic evaluation metrics report at the cutoffs inside a run of tied scores. basic_ties is effective only when mode is set to basic.

"split"

Spread the true and false positives of the run evenly over its cutoffs (default). This is the interpolation the ROC and precision-recall curves need, and is what precrec has always done.

"hold"

Give every cutoff in the run the counts it has once the whole run is taken, so tied instances share one value of every metric.

A cutoff inside a tied run splits instances that share a score, so no threshold produces it. With "split" a perfect classifier whose scores are all 0 or 1 reports sensitivity climbing from 0 to 1 across the positives rather than reaching 1 at once. "hold" makes each metric a step function that changes only where the score does. The two agree whenever the scores are all distinct, and "split" is kept as the default because it is what every published precrec result was computed with.

...

These additional arguments are passed to mmdata() for data preparation. multiclass = "ovr" asks for a one-vs-rest evaluation of a dataset with more than two classes; see mmdata().

See Also

plot() for plotting curves with the general R plot. autoplot() and fortify() for plotting curves with ggplot2. mmdata() for formatting input data. join_scores() and join_labels() for formatting scores and labels with multiple datasets. format_nfold() for creating n-fold cross validation dataset from data frame. create_sim_samples() for generating random samples for simulations.

Examples

##################################################
### Single model & single test dataset
###

## Load a dataset with 10 positives and 10 negatives
data(P10N10)

## Generate an sscurve object that contains ROC and Precision-Recall curves
sscurves <- evalmod(scores = P10N10$scores, labels = P10N10$labels)
sscurves

## Generate an sspoints object that contains basic evaluation metrics
sspoints <- evalmod(
  mode = "basic", scores = P10N10$scores,
  labels = P10N10$labels
)
sspoints

## Let tied scores share one value of every basic metric
tiedpoints <- evalmod(
  mode = "basic", scores = round(P10N10$scores, 1),
  labels = P10N10$labels, basic_ties = "hold"
)
tiedpoints


##################################################
### Multiple models & single test dataset
###

## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(1, 100, 100, "all")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
  modnames = samps[["modnames"]]
)

## Generate an mscurve object that contains ROC and Precision-Recall curves
mscurves <- evalmod(mdat)
mscurves

## Generate an mspoints object that contains basic evaluation metrics
mspoints <- evalmod(mdat, mode = "basic")
mspoints


##################################################
### Single model & multiple test datasets
###

## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(4, 100, 100, "good_er")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
  modnames = samps[["modnames"]],
  dsids = samps[["dsids"]]
)

## Generate an smcurve object that contains ROC and Precision-Recall curves
smcurves <- evalmod(mdat)
smcurves

## Generate an smpoints object that contains basic evaluation metrics
smpoints <- evalmod(mdat, mode = "basic")
smpoints


##################################################
### Multiple models & multiple test datasets
###

## Create sample datasets with 100 positives and 100 negatives
samps <- create_sim_samples(4, 100, 100, "all")
mdat <- mmdata(samps[["scores"]], samps[["labels"]],
  modnames = samps[["modnames"]],
  dsids = samps[["dsids"]]
)

## Generate an mmcurve object that contains ROC and Precision-Recall curves
mmcurves <- evalmod(mdat)
mmcurves

## Generate an mmpoints object that contains basic evaluation metrics
mmpoints <- evalmod(mdat, mode = "basic")
mmpoints


##################################################
### N-fold cross validation datasets
###

## Load test data
data(M2N50F5)

## Speficy nessesary columns to create mdat
cvdat <- mmdata(
  nfold_df = M2N50F5, score_cols = c(1, 2),
  lab_col = 3, fold_col = 4,
  modnames = c("m1", "m2"), dsids = 1:5
)

## Generate an mmcurve object that contains ROC and Precision-Recall curves
cvcurves <- evalmod(cvdat)
cvcurves

## Generate an mmpoints object that contains basic evaluation metrics
cvpoints <- evalmod(cvdat, mode = "basic")
cvpoints

## Specify mmdata arguments from evalmod
cvcurves2 <- evalmod(
  nfold_df = M2N50F5, score_cols = c(1, 2),
  lab_col = 3, fold_col = 4,
  modnames = c("m1", "m2"), dsids = 1:5
)
cvcurves2


##################################################
### AUC with the U statistic
###

## mode = "aucroc" returns 'aucroc' S3 object
data(P10N10)

# 'aucroc' S3 object
uauc1 <- evalmod(
  scores = P10N10$scores, labels = P10N10$labels,
  mode = "aucroc"
)

# print 'aucroc'
uauc1

# as.data.frame 'aucroc'
as.data.frame(uauc1)

## It is 2-3 times faster than mode = "rocprc"
# A sample of 100,000
samp1 <- create_sim_samples(1, 50000, 50000)

# a function to test mode = "rocprc"
func_evalmod_rocprc <- function(samp) {
  curves <- evalmod(scores = samp$scores, labels = samp$labels)
  aucs <- auc(curves)
}

# a function to test mode = "aucroc"
func_evalmod_aucroc <- function(samp) {
  uaucs <- evalmod(
    scores = samp$scores, labels = samp$labels,
    mode = "aucroc"
  )
  as.data.frame(uaucs)
}

# Process time
system.time(res1 <- func_evalmod_rocprc(samp1))
system.time(res2 <- func_evalmod_aucroc(samp1))

# AUCs
res1
res2


##################################################
### Multiclass evaluation
###

## Load a 3-class dataset with one score column per class
data(C3N150)

## Each class is evaluated against the rest
mccurves <- evalmod(scores = C3N150$scores, labels = C3N150$labels)
mccurves

## Per-class AUCs, plus their macro-average
auc(mccurves)