Significance testing, comparing CAD vs. a group of radiologists interpreting the same cases, an example of single modality analysis
StSignificanceTestingCadVsRadiologists (dataset, FOM = "Wilcoxon",
option = "RRRC", method = "singleModality", FPFValue = 0.2)The dataset must be ROC or LROC.
The desired FOM, default is "Wilcoxon" for ROC data, or ROC data
inferred from LROC data;
for LROC data the choices are "PCL" and "ALROC".
The desired generalization, the default is "RRRC";
another possibility is "RRFC".
"singleModality", the default, or "dualModality",
see details.
Only needed for LROC data; where to evaluate a partial curve based figure of merit, see details. The default is 0.2.
## potential project for summer student
If method = "singleModality" the return value is a
list with the following elements:
The observed FOM for CAD
The observed FOM array for the readers
The average FOM of the readers
The mean of the difference FOM, RAD - CAD
The 95-percent CI of the average difference, RAD - CAD
The variance of the radiologists
The variance of the error term in the single-modality multiple-reader OR model
The covariance of the error term
The observed value of the t-statistic; it's square is equivalent to an F-statistic
The degrees of freedom of the t-statistic
The p-value for rejecting the NH
Empirical operating characteristic plots corresponding to specified FOM
If method = "dualModality" the return value is a list with the following elements:
The observed FOM for CAD
The observed FOM array for the readers
The average FOM of the readers
The mean of the difference FOM, RAD - CAD
A data frame containing the statistics associated with the average difference, RAD - CAD
A data frame containing the statistics associated with the average FOM in each treatment
The variance of the pure reader term in the OR model
The variance of the treatment-reader term error term in the OR model
The covariance1 of the error term - same reader, different treatments
The covariance2 of the error term - different readers, same treatment
The covariance3 of the error term - different readers, different treatments
The variance of the pure error term in the OR model
The observed value of the F-statistic
The numerator degrees of freedom of the F-statistic
The denominator degrees of freedom of the F-statistic
The p-value for rejecting the NH
Empirical operating characteristic plots corresponding
to specified FOM, i.e., if FOM = "Wilcoxon" an ROC plot
is produced where reader 1 is CAD. If an LROC FOM is selected, an LROC
plot is displayed.
PCL is the probability of a correct localization. The LROC is the plot of PCL
(ordinate) vs. FPF. For LROC data "PCL" means interpolated PCL value
at specified "FPFValue". "ALROC" is the trapezoidal area
under the LROC
from FPF = 0 to FPF = FPFValue. If method = "singleModality"
the first reader is assumed to be CAD. If
method = "dualModality" the first modality is assumed to be CAD.
The NH is that the FOM of CAD equals the average of the readers. The
method = "singleModality" analysis uses an adaptation of the
single-modality multiple-reader Obuchowski Rockette (OR) model described in a
paper by Hillis (2007), section 5.3. The adaptation is characterized by 3
parameters
VarR, Var and Cov2, which are returned by the function.
The method = "dualModality" analysis replicates CAD data as many times as
necessary so as to form one "modality" of an MRMC pairing, the other
"modality" being the
radiologists. Standard RRRC DBMH/ORH analysis is applied. The
method, described
in Kooi et al gives exactly the same final results (F-statistic, ddf and p-value)
as "singleModality" but the intermediate quantities are questionable.
The method is characterized by 6 OR parameters VarR, VarTR,
Var, Cov1, Cov2 and Cov3, which are returned
by the function.
Hillis SL (2007) A comparison of denominator degrees of freedom methods for multiple observer ROC studies, Statistics in Medicine. 26:596-619.
Chakraborty DP (2017) Observer Performance Methods for Diagnostic Imaging - Foundations, Modeling, and Applications with R-Based Examples, CRC Press, Boca Raton, FL. https://www.crcpress.com/Observer-Performance-Methods-for-Diagnostic-Imaging-Foundations-Modeling/Chakraborty/p/book/9781482214840
Hupse R, Samulski M, Lobbes M, et al (2013) Standalone computer-aided detection compared to radiologists performance for the detection of mammographic masses, Eur Radiol. 23(1):93-100.
Kooi T, Gubern-Merida A, et al. (2016) A comparison between a deep convolutional neural network and radiologists for classifying regions of interest in mammography. Paper presented at: International Workshop on Digital Mammography, Malmo, Sweden.
# NOT RUN {
ret1 <- StSignificanceTestingCadVsRadiologists (dataset09,
FOM = "Wilcoxon", method = "singleModality")
# }
# NOT RUN {
ret2 <- StSignificanceTestingCadVsRadiologists (dataset09,
FOM = "Wilcoxon", method = "dualModality")
ret1 <- StSignificanceTestingCadVsRadiologists (datasetCadLroc,
FOM = "PCL", option = "RRRC", method = "singleModality", FPFValue = 0.05)
ret2 <- StSignificanceTestingCadVsRadiologists (datasetCadLroc,
FOM = "PCL", option = "RRRC", method = "dualModality", FPFValue = 0.05)
# }
# NOT RUN {
# }
Run the code above in your browser using DataLab