Probability Calibration Plot
For classification models, this function creates a 'calibration plot' that describes how consistent model probabilities are with observed event rates.
calibration(x, ...)"calibration"(x, data = NULL, class = NULL, cuts = 11, subset = TRUE, lattice.options = NULL, ...)"xyplot"(x, data, ...)panel.calibration(...)
xyplotfor syntax) where the left-hand side of the formula is a factor class variable of the observed outcome and the right-hand side specifies one or model columns corresponding to a numeric ranking variable for a model (e.g. class probabilities). The classification variable should have two levels.
calibration.formula, a data frame (or more precisely, anything that is a valid
eval, e.g., a list or an environment) containing values for any variables in the formula, as well as
subsetif applicable. If not found in
data, or if
datais unspecified, the variables are looked for in the environment of the formula. This argument is not used for
- a character string for the class of interest
If a single number this indicates the number of splits of the data are used to create the plot. By default, it uses as many cuts as there are rows in
data. If a vector, these are the actual cuts that will be used.
An expression that evaluates to a logical or integer indexing vector. It is evaluated in
data. Only the resulting rows of
dataare used for the plot.
A list that could be supplied to
options to pass through to
xyplotor the panel function (not used in
calibration.formula is used to process the data and
xyplot.calibration is used to create the plot.
To construct the calibration plot, the following steps are used for each model:
- The data are split into
cuts - 1roughly equal groups by their class probabilities
- the number of samples with true results equal to
- the event rate is determined for each bin
xyplot.calibration produces a plot of the observed event rate by the mid-point of the bins.
This implementation uses the lattice function
xyplot, so plot elements can be changed via panel functions,
trellis.par.set or other means.
calibration uses the panel function
panel.calibration by default, but it can be changed by passing that argument into
The following elements are set by default in the plot but can be changed by passing new values into
xlab = "Bin Midpoint",
ylab = "Observed Event Percentage",
type = "o",
ylim = extendrange(c(0, 100)),
xlim = extendrange(c(0, 100)) and
panel = panel.calibration
- the data used for plotting
- the number of cuts
- the event class
- the names of the model probabilities
calibration.formulareturns a list with elements:
xyplot.calibrationreturns a lattice object
## Not run: # data(mdrr) # mdrrDescr <- mdrrDescr[, -nearZeroVar(mdrrDescr)] # mdrrDescr <- mdrrDescr[, -findCorrelation(cor(mdrrDescr), .5)] # # # inTrain <- createDataPartition(mdrrClass) # trainX <- mdrrDescr[inTrain[], ] # trainY <- mdrrClass[inTrain[]] # testX <- mdrrDescr[-inTrain[], ] # testY <- mdrrClass[-inTrain[]] # # library(MASS) # # ldaFit <- lda(trainX, trainY) # qdaFit <- qda(trainX, trainY) # # testProbs <- data.frame(obs = testY, # lda = predict(ldaFit, testX)$posterior[,1], # qda = predict(qdaFit, testX)$posterior[,1]) # # calibration(obs ~ lda + qda, data = testProbs) # # calPlotData <- calibration(obs ~ lda + qda, data = testProbs) # calPlotData # # xyplot(calPlotData, auto.key = list(columns = 2)) # ## End(Not run)