This function is used to the test the performance of an authorship analysis method.
performance(training, test = NULL, by = "case", progress = TRUE)The function returns a list containing a data frame with performance statistics, including an object that can be used to make a tippet plot using the tippet.plot() function of the ROC package (https://github.com/davidavdav/ROC).
The data frame with the results to evaluate, typically the output of an authorship analysis function, such as impostors(). If only training is present then the function will perform a leave-one-out cross-validation.
Optional data frame of results. If present then a calibration model is extracted from training and its performance is evaluated on this data set.
Either "case" or "author". If the performance is evaluated leave-one-out, then "case" would go through the table row by row while, if "author" is selected, then the performance is calculated after taking out each author (identified as a value of the K column).
Logical. If TRUE (default) then a progress bar is diplayed.
Before applying a method to a real authorship case, it is good practice to test it on known ground truth data. This function performs this test by taking as input either a single table of results or two tables, one for training and one for the test, and then returning as output a list with the following performance statistics: the log-likelihood ratio cost (both \(C_{llr}\) and \(C_{llr}^{min}\)), Equal Error Rate (ERR), the mean values of the log-likelihood ratio for both the same-author (TRUE) and different-author (FALSE) cases, the Area Under the Curve (AUC), Balanced Accuracy, Precision, Recall, F1, and the full confusion matrix. The binary classification statistics are all calculated considering a Log-Likelihood Ratio score of 0 as a threshold.
results <- data.frame(score = c(0.5, 0.2, 0.8, 0.01), target = c(TRUE, FALSE, TRUE, FALSE))
perf <- performance(results)
perf$evaluation
Run the code above in your browser using DataLab