Learn R Programming

⚠️There's a newer version (2.1.2) of this package.Take me there.

RJafroc (version 2.0.1)

Artificial Intelligence Systems and Observer Performance

Description

Analyzing the performance of artificial intelligence (AI) systems/algorithms characterized by a "search-and-report" strategy. While historically observer performance has dealt with measuring radiologists' performance in search tasks – i.e., searching for lesions in medical images and reporting them - the software described here applies equally to any task involving searching for and reporting arbitrary targets in images. The package can be used to analyze the performance of AI systems, compare AI performance to a group of human readers or optimize the reporting threshold of an AI system. In addition to performing conventional receiver operating characteristic (ROC) analysis (localization information ignored), the software also performs free-response receiver operating characteristic (FROC) analysis, where lesion localization information is integral to the analyzed data. A book using the software has been published: Chakraborty DP: Observer Performance Methods for Diagnostic Imaging - Foundations, Modeling, and Applications with R-Based Examples, Taylor-Francis LLC; 2017. An online update of this book is at . Illustrations of the software (vignettes) are at . Supported data collection paradigms are the ROC, FROC and the location ROC (LROC). ROC data consists of single ratings per images, where a rating is the perceived confidence level that the image is that of a diseased patient. An ROC curve is a plot of true positive fraction vs. false positive fraction. FROC data consists of a variable number (zero or more) of mark-rating pairs per image, where a mark is the location of a reported suspicious region and the rating is the confidence level that it is a real lesion. LROC data consists of a rating and a location of the most suspicious region, for every image. Four models of observer performance, and curve-fitting software, are implemented: the binormal model (BM), the contaminated binormal model (CBM), the correlated contaminated binormal model (CORCBM), and the radiological search model (RSM). Unlike the binormal model, CBM, CORCBM and RSM predict "proper" ROC curves that do not inappropriately cross the chance diagonal. Additionally, RSM parameters are related to search performance (not measured in conventional ROC analysis) and classification performance. Search performance refers to finding lesions, i.e., true positives, while simultaneously not finding false positive locations. Classification performance measures the ability to distinguish between true and false positive locations. Knowing these separate performances allows principled optimization of reader or AI system performance. RJafroc supersedes Windows JAFROC (jackknife alternative FROC) software V4.2.1, . Package functions are organized as follows. Data file related function names are preceded by "Df", curve fitting functions by "Fit", included data sets by "dataset", plotting functions by "Plot", significance testing functions by "St", sample size related functions by "Ss", data simulation functions by "Simulate" and utility functions by "Util". Implemented are figures of merit (FOMs) for quantifying performance and functions for visualizing empirical or fitted operating characteristics: e.g., ROC, FROC, alternative FROC (AFROC) and weighted AFROC (wAFROC) curves. For fully crossed study designs significance testing of reader-averaged FOM differences between modalities is implemented via either Dorfman-Berbaum-Metz or the Obuchowski-Rockette methods. Also implemented is single treatment analysis, which allows comparison of performance of a group of radiologists to a specified value, or comparison of AI to a group of radiologists interpreting the same cases. Crossed-modality analysis is implemented wherein there are two crossed treatment factors and the aim is to determined performance in each treatment factor averaged over all levels of the second factor. Sample size estimation tools are provided for ROC and FROC studies; these use estimates of the relevant variances from a pilot study to predict required numbers of readers and cases in a pivotal study to achieve the desired power. Utility and data file manipulation functions allow data to be read in any of the currently used input formats, including Excel, and the results of the analysis can be viewed in text or Excel output files. The methods are illustrated with several included datasets from the author's collaborations. This version corrects bugs, simplifies usage of the software and updates the dataset structure. All changes are noted in NEWS.

Copy Link

Version

Install

install.packages('RJafroc')

Monthly Downloads

260

Version

2.0.1

License

GPL-3

Maintainer

Dev Chakraborty

Last Published

December 15th, 2020

Functions in RJafroc (2.0.1)

DfCreateCorCbmDataset

Create paired dataset for testing FitCorCbm
DfFroc2Roc

Convert an FROC dataset to an ROC dataset
ChisqrGoodnessOfFit

Compute the chisquare goodness of fit statistic for ROC fitting model
DfFroc2Lroc

Simulates an "AUC-equivalent" LROC dataset from an FROC dataset
Df2RJafrocDataset

Convert ratings arrays to an RJafroc dataset
Compare3ProperRocFits

Compare three proper-ROC curve fitting models
DfExtractCorCbmDataset

Extract two arms of a pairing from an MRMC ROC dataset
DfLroc2Froc

Simulates an "AUC-equivalent" FROC dataset from an LROC dataset
DfExtractDataset

Extract a subset of treatments and readers from a dataset
DfBinDataset

Returns a binned dataset
DfReadDataFile

Read a data file
FitCbmRoc

Fit the contaminated binormal model (CBM) to selected treatment and reader in an ROC dataset
FitBinormalRoc

Fit the binormal model to selected treatment and reader in an ROC dataset
FitCorCbm

Fit CORCBM to a paired ROC dataset
SsFrocNhRsmModel

RSM fitted model for FROC sample size
RJafroc-package

Artificial Intelligence Systems and Observer Performance
SsPowerGivenJK

Statistical power for specified numbers of readers and cases
FitRsmRoc

Fit the radiological search model (RSM) to an ROC dataset
SimulateCorCbmDataset

Simulate paired binned data for testing FitCorCbm
PlotEmpiricalOperatingCharacteristics

Plot empirical operating characteristics, ROC, FROC or LROC
DfSaveDataFile

Save ROC data file in a different format
DfLroc2Roc

Convert an LROC dataset to a ROC dataset
PlotBinormalFit

Plot binormal fit
DfReadCrossedModalities

Read a crossed-treatment data file
PlotCbmFit

Plot CBM fitted curve
StSignificanceTestingCrossedModalities

Perform significance testing using crossed treatments analysis
UtilAucBinormal

Binormal model AUC function
SimulateFrocFromLrocDataset

Simulates an "AUC-equivalent" FROC dataset from an LROC dataset
SimulateFrocDataset

Simulates an MRMC uncorrelated FROC dataset using the RSM
SsPowerGivenJKDbmVarCom

Power given J, K and Dorfman-Berbaum-Metz variance components
PlotRsmOperatingCharacteristics

RSM predicted operating characteristics, ROC highest rating pdfs and FOMs, for FROC data
SsPowerGivenJKOrVarCom

Power given J, K and Obuchowski-Rockette variance components
UtilAucPROPROC

PROPROC AUC function
UtilVarComponentsDBM

Utility for Dorfman-Berbaum-Metz variance components
dataset01

TONY FROC dataset
UtilDBM2ORVarCom

Convert from DBM to OR variance components
datasetBinned124

Binned dataset suitable for checking FitCorCbm; seed = 124
UtilAucCBM

CBM AUC function
UtilLesionDistr

Lesion distribution of a dataset or as specified by a one-dimensional array
UtilLesionWeightsDistr

Lesion weights distribution
datasetBinned125

Binned dataset suitable for checking FitCorCbm; seed = 125
SsPowerTable

Generate a power table using the OR method
datasetCrossedModality

John Thompson crossed treatment FROC dataset
StSignificanceTesting

Performs DBM or OR significance testing for factorial or split-plot A,C datasets
SimulateRocDataset

Simulates a binormal model ROC dataset
SimulateLrocDataset

Simulates an uncorrelated FLROC FrocDataset using the RSM
datasetDegenerate

Simulated degenerate ROC dataset (for testing purposes)
UtilORVarComponentsFactorial

Utility for estimating Obuchowski-Rockette variance components for factorial datasets
UtilPhysical2IntrinsicRSM

Convert from physical to intrinsic RSM parameters
SsSampleSizeKGivenJ

Number of cases, for specified number of readers, to achieve desired power
UtilOutputReport

Generate a text formatted report file or an Excel file
dataset06

Magnus FROC dataset
dataset07

Lucy Warren FROC dataset
dataset02

Van Dyke ROC dataset
dataset03

Franken ROC dataset
datasetBinned123

Binned dataset suitable for checking FitCorCbm; seed = 123
dataset14

Federica Zanca real (as opposed to inferred) ROC dataset
UtilFigureOfMerit

Calculate empirical figures of merit (FOMs) for specified dataset
UtilAnalyticalAucsRSM

RSM ROC/AFROC/wAFROC AUC calculator
UtilIntrinsic2PhysicalRSM

Convert from intrinsic to physical RSM parameters
StSignificanceTestingCadVsRad

Significance testing: standalone CAD vs. radiologists
UtilMeanSquares

Calculate mean squares for factorial dataset
dataset04

Federica Zanca FROC dataset
dataset05

John Thompson FROC dataset
datasetCadLroc

Nico Karssemeijer LROC dataset (CAD vs. radiologists)
dataset12

Dobbins 2 ROC dataset
dataset13

Dobbins 3 FROC dataset
datasetFROCSpC

Simulated FROC SPLIT-PLOT-C dataset
UtilOR2DBMVarCom

Convert from OR to DBM variance components
dataset08

Monica Penedo ROC dataset
dataset09

Nico Karssemeijer ROC dataset (CAD vs. radiologists)
isBinnedDataset

Determine if a dataset is binned
isValidDataset

Check the validity of a dataset
UtilPseudoValues

Pseudovalues for given dataset and FOM
dataset11

Dobbins 1 FROC dataset
dataset10

Marc Ruschin ROC dataset
datasetROI

Simulated ROI dataset
datasetCadSimuFroc

Simulated FROC CAD vs. RAD dataset