RJafroc (version 1.2.0)

StSignificanceTestingCadVsRadiologists: Significance testing, CAD vs. radiologists

Description

Significance testing, comparing CAD vs. a group of radiologists interpreting the same cases, an example of single treatment analysis

Usage

StSignificanceTestingCadVsRadiologists(dataset, FOM = "Wilcoxon",
  option = "RRRC", method = "singleModality", FPFValue = 0.2)

Arguments

dataset

The dataset must be ROC or LROC.

FOM

The desired FOM, default is "Wilcoxon" for ROC data, or ROC data inferred from LROC data; for LROC data the choices are "PCL" and "ALROC".

option

The desired generalization, the default is "RRRC"; another possibility is "RRFC".

method

"singleModality", the default, or "dualModality", see details.

FPFValue

Only needed for LROC data; where to evaluate a partial curve based figure of merit, see details. The default is 0.2.

## potential project for summer student

Value

If method = "singleModality" the return value is a list with the following elements:

fomCAD

The observed FOM for CAD

fomRAD

The observed FOM array for the readers

avgRadFom

The average FOM of the readers

avgDiffFom

The mean of the difference FOM, RAD - CAD

ciAvgDiffFom

The 95-percent CI of the average difference, RAD - CAD

varR

The variance of the radiologists

varError

The variance of the error term in the single-treatment multiple-reader OR model

cov2

The covariance of the error term

tstat

The observed value of the t-statistic; it's square is equivalent to an F-statistic

df

The degrees of freedom of the t-statistic

pval

The p-value for rejecting the NH

Plots

Empirical operating characteristic plots corresponding to specified FOM

If method = "dualModality" the return value is a list with the following elements:

fomCAD

The observed FOM for CAD

fomRAD

The observed FOM array for the readers

avgRadFom

The average FOM of the readers

avgDiffFom

The mean of the difference FOM, RAD - CAD

ciDiffFom

A data frame containing the statistics associated with the average difference, RAD - CAD

ciAvgRdrEachTrt

A data frame containing the statistics associated with the average FOM in each treatment

varR

The variance of the pure reader term in the OR model

varTR

The variance of the treatment-reader term error term in the OR model

cov1

The covariance1 of the error term - same reader, different treatments

cov2

The covariance2 of the error term - different readers, same treatment

cov3

The covariance3 of the error term - different readers, different treatments

varError

The variance of the pure error term in the OR model

Fstat

The observed value of the F-statistic

ndf

The numerator degrees of freedom of the F-statistic

ddf

The denominator degrees of freedom of the F-statistic

pval

The p-value for rejecting the NH

Plots

Empirical operating characteristic plots corresponding to specified FOM, i.e., if FOM = "Wilcoxon" an ROC plot is produced where reader 1 is CAD. If an LROC FOM is selected, an LROC plot is displayed.

Details

PCL is the probability of a correct localization. The LROC is the plot of PCL (ordinate) vs. FPF. For LROC data "PCL" means interpolated PCL value at specified "FPFValue". "ALROC" is the trapezoidal area under the LROC from FPF = 0 to FPF = FPFValue. If method = "singleModality" the first reader is assumed to be CAD. If method = "dualModality" the first treatment is assumed to be CAD. The NH is that the FOM of CAD equals the average of the readers. The method = "singleModality" analysis uses an adaptation of the single-treatment multiple-reader Obuchowski Rockette (OR) model described in a paper by Hillis (2007), section 5.3. The adaptation is characterized by 3 parameters VarR, Var and Cov2, which are returned by the function. The method = "dualModality" analysis replicates CAD data as many times as necessary so as to form one "treatment" of an MRMC pairing, the other "treatment" being the radiologists. Standard RRRC DBMH/ORH analysis is applied. The method, described in Kooi et al gives exactly the same final results (F-statistic, ddf and p-value) as "singleModality" but the intermediate quantities are questionable. The method is characterized by 6 OR parameters VarR, VarTR, Var, Cov1, Cov2 and Cov3, which are returned by the function.

References

Hillis SL (2007) A comparison of denominator degrees of freedom methods for multiple observer ROC studies, Statistics in Medicine. 26:596-619.

Chakraborty DP (2017) Observer Performance Methods for Diagnostic Imaging - Foundations, Modeling, and Applications with R-Based Examples, CRC Press, Boca Raton, FL. https://www.crcpress.com/Observer-Performance-Methods-for-Diagnostic-Imaging-Foundations-Modeling/Chakraborty/p/book/9781482214840

Hupse R, Samulski M, Lobbes M, et al (2013) Standalone computer-aided detection compared to radiologists performance for the detection of mammographic masses, Eur Radiol. 23(1):93-100.

Kooi T, Gubern-Merida A, et al. (2016) A comparison between a deep convolutional neural network and radiologists for classifying regions of interest in mammography. Paper presented at: International Workshop on Digital Mammography, Malmo, Sweden.

Examples

# NOT RUN {
ret1M <- StSignificanceTestingCadVsRadiologists (dataset09, 
FOM = "Wilcoxon", method = "singleModality")

# }
# NOT RUN {
## takes longer than 5 sec on OSX
ret2M <- StSignificanceTestingCadVsRadiologists (dataset09, 
FOM = "Wilcoxon", method = "dualModality")

retLroc1M <- StSignificanceTestingCadVsRadiologists (datasetCadLroc, 
FOM = "PCL", option = "RRRC", method = "singleModality", FPFValue = 0.05)

retLroc2M <- StSignificanceTestingCadVsRadiologists (datasetCadLroc, 
FOM = "PCL", option = "RRRC", method = "dualModality", FPFValue = 0.05)

## test with fewer readers
dataset09a <- DfExtractDataset(dataset09, rdrs = seq(1:7))
ret1M7 <- StSignificanceTestingCadVsRadiologists (dataset09a, 
FOM = "Wilcoxon", method = "singleModality")
ret2M7 <- StSignificanceTestingCadVsRadiologists (dataset09a, 
FOM = "Wilcoxon", method = "dualModality")

datasetCadLroc7 <- DfExtractDataset(datasetCadLroc, rdrs = seq(1:7))
ret1MLroc7 <- StSignificanceTestingCadVsRadiologists (datasetCadLroc7, 
FOM = "PCL", option = "RRRC", method = "singleModality", FPFValue = 0.05)

ret2MLroc7 <- StSignificanceTestingCadVsRadiologists (datasetCadLroc7, 
FOM = "PCL", option = "RRRC", method = "dualModality", FPFValue = 0.05)
# }
# NOT RUN {
# }