The function val.prob.ci.2 is an adaptation of val.prob from Frank Harrell's rms package,
https://cran.r-project.org/package=rms. Hence, the description of some of the functions of val.prob.ci.2
come from the the original val.prob.
The key feature of val.prob.ci.2 is the generation of logistic and flexible calibration curves and related statistics.
When using this code, please cite: Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M.J., Steyerberg,
E.W. (2016). A calibration hierarchy for risk models was defined: from utopia to empirical data. Journal of Clinical Epidemiology,
74, pp. 167-176
val.prob.ci.2(
p,
y,
logit,
group,
weights = rep(1, length(y)),
normwt = FALSE,
pl = TRUE,
smooth = c("loess", "rcs", "none"),
CL.smooth = "fill",
CL.BT = FALSE,
lty.smooth = 1,
col.smooth = "black",
lwd.smooth = 1,
nr.knots = 5,
logistic.cal = FALSE,
lty.log = 1,
col.log = "black",
lwd.log = 1,
xlab = "Predicted probability",
ylab = "Observed proportion",
xlim = c(-0.02, 1),
ylim = c(-0.15, 1),
m,
g,
cuts,
emax.lim = c(0, 1),
legendloc = c(0.5, 0.27),
statloc = c(0, 0.85),
dostats = TRUE,
cl.level = 0.95,
method.ci = "pepe",
roundstats = 2,
riskdist = "predicted",
cex = 0.75,
cex.leg = 0.75,
connect.group = FALSE,
connect.smooth = TRUE,
g.group = 4,
evaluate = 100,
nmin = 0,
d0lab = "0",
d1lab = "1",
cex.d01 = 0.7,
dist.label = 0.04,
line.bins = -0.05,
dist.label2 = 0.03,
cutoff,
las = 1,
length.seg = 1,
y.intersp = 1,
lty.ideal = 1,
col.ideal = "red",
lwd.ideal = 1,
allowPerfectPredictions = FALSE,
argzLoess = alist(degree = 2),
...
)An object of type CalibrationCurve with the following slots:
the matched call.
a vector containing performance measures of calibration.
the confidence level used.
contains the calibration intercept and slope, together with their confidence intervals.
the value of the c-statistic, together with its confidence interval.
if any, the warning messages that were printed while running the function.
The coordinates for plotting the calibration curves.
predicted probability
vector of binary outcomes
predicted log odds of outcome. Specify either p or logit.
a grouping variable. If numeric this variable is grouped into
g.group quantile groups (default is quartiles). Set group=TRUE to
use the group algorithm but with a single stratum for val.prob.
an optional numeric vector of per-observation weights (usually frequencies),
used only if group is given.
set to TRUE to make weights sum to the number of non-missing observations.
TRUE to plot the calibration curve(s). If FALSE no calibration curves will be plotted,
but statistics will still be computed and outputted.
"loess" generates a flexible calibration curve based on loess,
"rcs" generates a calibration curves based on restricted cubic splines (see rcs and
rcspline.plot), "none" suppresses the flexible curve. We recommend to use loess unless N is large,
for example N>5000. Default is "loess".
"fill" shows pointwise 95% confidence limits for the flexible calibration curve with a gray
area between the lower and upper limits, TRUE shows pointwise 95% confidence limits for the flexible calibration curve
with dashed lines, FALSE suppresses the confidence limits. Default is "fill".
TRUE uses confidence limits based on 2000 bootstrap samples, FALSE uses closed form confidence limits.
Default is FALSE.
the linetype of the flexible calibration curve. Default is 1.
the color of the flexible calibration curve. Default is "black".
the line width of the flexible calibration curve. Default is 1.
specifies the number of knots for rcs-based calibration curve. The default as well as the highest allowed value is 5. In case the specified number of knots leads to estimation problems, then the number of knots is automatically reduced to the closest value without estimation problems.
TRUE plots the logistic calibration curve, FALSE suppresses this curve.
Default is FALSE.
if logistic.cal=TRUE, the linetype of the logistic calibration curve. Default is 1.
if logistic.cal=TRUE, the color of the logistic calibration curve. Default is "black".
if logistic.cal=TRUE, the line width of the logistic calibration curve. Default is 1.
x-axis label, default is "Predicted Probability".
y-axis label, default is "Observed proportion".
numeric vectors of length 2, giving the x and y coordinates ranges (see plot.window)
If grouped proportions are desired, minimum no. observations per group
If grouped proportions are desired, number of quantile groups
If grouped proportions are desired, actual cut points for constructing
intervals, e.g. c(0,.1,.8,.9,1) or seq(0,1,by=.2)
Vector containing lowest and highest predicted probability over which to
compute Emax.
if pl=TRUE, list with components x,y or vector c(x,y) for bottom right corner of legend for
curves and points. Default is c(.50, .27) scaled to lim. Use locator(1) to use the mouse, FALSE to suppress legend.
the "abc" of model performance (Steyerberg et al., 2011)-calibration intercept, calibration slope,
and c statistic-will be added to the plot, using statloc as the upper left corner of a box (default is c(0,.85).
You can specify a list or a vector. Use locator(1) for the mouse, FALSE to suppress statistics. This is plotted after
the curve legends.
specifies whether and which performance measures are shown in the figure.
TRUE shows the "abc" of model performance (Steyerberg et al., 2011): calibration intercept, calibration slope,
and c-statistic. TRUE is default.
FALSE suppresses the presentation of statistics in the figure. A c() list of specific stats shows the specified
stats. The key stats which are also mentioned in this paper are "C (ROC)" for the c statistic, "Intercept" for the
calibration intercept, "Slope" for the calibration slope, and "ECI" for the estimated calibration index
(Van Hoorde et al, 2015). The full list of possible statistics is taken from val.prob
and augmented with the estimated calibration index: "Dxy", "C (ROC)", "R2", "D", "D:Chi-sq", "D:p", "U", "U:Chi-sq",
"U:p", "Q", "Brier", "Intercept", "Slope", "Emax", "Brier scaled", "Eavg", "ECI". These statistics are always returned by the function.
if dostats=TRUE, the confidence level for the calculation of the confidence intervals of the calibration intercept,
calibration slope and c-statistic. Default is 0.95.
method to calculate the confidence interval of the c-statistic. The argument is passed to auc.nonpara.mw from
the auRoc-package and possible methods to compute the confidence interval are "newcombe", "pepe", "delong" or
"jackknife". Bootstrap-based methods are not available. The default method is "pepe" and here, the confidence interval is
the logit-transformation-based confidence interval as documented in Qin and Hotilovac (2008). See auc.nonpara.mw for
more information on the other methods.
specifies the number of decimals to which the statistics are rounded when shown in the plot. Default is 2.
Use "calibrated" to plot the relative frequency distribution of
calibrated probabilities after dividing into 101 bins from lim[1] to
lim[2].
Set to "predicted" (the default as of rms 4.5-1) to use raw assigned risk, FALSE to omit risk distribution.
Values are scaled so that highest bar is 0.15*(lim[2]-lim[1]).
controls the font size of the statistics (cex) or plot legend (cex.leg). Default is 0.75
Defaults to FALSE to only represent group fractions as triangles.
Set to TRUE to also connect with a solid line.
Defaults to TRUE to draw smoothed estimates using a line. Set to FALSE to instead use dots at individual estimates
number of quantile groups to use when group is given and variable is
numeric.
number of points at which to store the lowess-calibration curve.
Default is 100. If there are more than evaluate unique predicted
probabilities, evaluate equally-spaced quantiles of the unique
predicted probabilities, with linearly interpolated calibrated values,
are retained for plotting (and stored in the object returned by
val.prob.
applies when group is given. When nmin \(> 0\), val.prob will not
store coordinates of smoothed calibration curves in the outer tails,
where there are fewer than nmin raw observations represented in
those tails. If for example nmin=50, the plot function will only
plot the estimated calibration curve from \(a\) to \(b\), where there are
50 subjects with predicted probabilities \(< a\) and \(> b\).
nmin is ignored when computing accuracy statistics.
controls the labels for events and non-events (i.e. outcome y) for the histograms.
Defaults are d1lab="1" for events and d0lab="0" for non-events.
controls the size of the labels for events and non-events. Default is 0.7.
controls the horizontal position of the labels for events and non-events. Default is 0.04.
controls the horizontal (y-axis) position of the histograms. Default is -0.05.
controls the vertical distance between the labels for events and non-events. Default is 0.03.
puts an arrow at the specified risk cut-off(s). Default is none.
controls whether y-axis values are shown horizontally (1) or vertically (0).
controls the length of the histogram lines. Default is 1.
character interspacing for vertical line distances of the legend (legend)
linetype of the ideal line. Default is 1.
controls the color of the ideal line on the plot. Default is "red".
controls the line width of the ideal line on the plot. Default is 1.
Logical, indicates whether perfect predictions (i.e. values of either 0 or 1) are allowed. Default is FALSE, since we transform
the predictions using the logit transformation to calculate the calibration measures. In case of 0 and 1, this results in minus infinity and infinity, respectively. if
allowPerfectPredictions = TRUE, 0 and 1 are replaced by 1e-8 and 1 - 1e-8, respectively.
a list with arguments passed to the loess function
When using the predicted probabilities of an uninformative model (i.e. equal probabilities for all observations), the model has no predictive value. Consequently, where applicable, the value of the performance measure corresponds to the worst possible theoretical value. For the ECI, for example, this equals 1 (Edlinger et al., 2022).
Edlinger, M, van Smeden, M, Alber, HF, Wanitschek, M, Van Calster, B. (2022). Risk prediction models for discrete ordinal outcomes: Calibration and the impact of the proportional odds assumption. Statistics in Medicine, 41( 8), pp. 1334– 1360
Qin, G., & Hotilovac, L. (2008). Comparison of non-parametric confidence intervals for the area under the ROC curve of a continuous-scale diagnostic test. Statistical Methods in Medical Research, 17(2), pp. 207-21
Steyerberg, E.W., Van Calster, B., Pencina, M.J. (2011). Performance measures for prediction models and markers : evaluation of predictions and classifications. Revista Espanola de Cardiologia, 64(9), pp. 788-794
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina M., Steyerberg E.W. (2016). A calibration hierarchy for risk models was defined: from utopia to empirical data. Journal of Clinical Epidemiology, 74, pp. 167-176
Van Hoorde, K., Van Huffel, S., Timmerman, D., Bourne, T., Van Calster, B. (2015). A spline-based tool to assess and visualize the calibration of multiclass risk predictions. Journal of Biomedical Informatics, 54, pp. 283-93
# Load package
library(CalibrationCurves)
set.seed(1783)
# Simulate training data
X = replicate(4, rnorm(5e2))
p0true = binomial()$linkinv(cbind(1, X) %*% c(0.1, 0.5, 1.2, -0.75, 0.8))
y = rbinom(5e2, 1, p0true)
Df = data.frame(y, X)
# Fit logistic model
FitLog = lrm(y ~ ., Df)
# Simulate validation data
Xval = replicate(4, rnorm(5e2))
p0true = binomial()$linkinv(cbind(1, Xval) %*% c(0.1, 0.5, 1.2, -0.75, 0.8))
yval = rbinom(5e2, 1, p0true)
Pred = binomial()$linkinv(cbind(1, Xval) %*% coef(FitLog))
# Default calibration plot
val.prob.ci.2(Pred, yval)
# Adding logistic calibration curves and other additional features
val.prob.ci.2(Pred, yval, CL.smooth = TRUE, logistic.cal = TRUE, lty.log = 2,
col.log = "red", lwd.log = 1.5)
val.prob.ci.2(Pred, yval, CL.smooth = TRUE, logistic.cal = TRUE, lty.log = 9,
col.log = "red", lwd.log = 1.5, col.ideal = colors()[10], lwd.ideal = 0.5)
Run the code above in your browser using DataLab