mlr (version 2.17.1)

generateCritDifferencesData: Generate data for critical-differences plot.

Description

Generates data that can be used to plot a critical differences plot. Computes the critical differences according to either the "Bonferroni-Dunn" test or the "Nemenyi" test. "Bonferroni-Dunn" usually yields higher power as it does not compare all algorithms to each other, but all algorithms to a baseline instead. Learners are drawn on the y-axis according to their average rank. For test = "nemenyi" a bar is drawn, connecting all groups of not significantly different learners. For test = "bd" an interval is drawn arround the algorithm selected as a baseline. All learners within this interval are not signifcantly different from the baseline. Calculation: $$CD = q_{\alpha} \sqrt{\left(\frac{k(k+1)}{6N}\right)}$$ Where \(q_\alpha\) is based on the studentized range statistic. See references for details.

Usage

generateCritDifferencesData(
  bmr,
  measure = NULL,
  p.value = 0.05,
  baseline = NULL,
  test = "bd"
)

Arguments

bmr

(BenchmarkResult) Benchmark result.

measure

(Measure) Performance measure. Default is the first measure used in the benchmark experiment.

p.value

(numeric(1)) P-value for the critical difference. Default: 0.05

baseline

(character(1)): (learner.id) Select a learner.id as baseline for the test = "bd" ("Bonferroni-Dunn") critical differences diagram. The critical difference interval will then be positioned arround this learner. Defaults to best performing algorithm. For test = "nemenyi", no baseline is needed as it performs all pairwise comparisons.

test

(character(1)) Test for which the critical differences are computed. “bd” for the Bonferroni-Dunn Test, which is comparing all classifiers to a baseline, thus performing a comparison of one classifier to all others. Algorithms not connected by a single line are statistically different from the baseline. “nemenyi” for the PMCMR::posthoc.friedman.nemenyi.test which is comparing all classifiers to each other. The null hypothesis that there is a difference between the classifiers can not be rejected for all classifiers that have a single grey bar connecting them.

Value

(critDifferencesData). List containing:

data

(data.frame) containing the info for the descriptive part of the plot

friedman.nemenyi.test

(list) of class pairwise.htest contains the calculated PMCMR::posthoc.friedman.nemenyi.test

cd.info

(list) containing info on the critical difference and its positioning

baseline

baseline chosen for plotting

p.value

p.value used for the PMCMR::posthoc.friedman.nemenyi.test and for computation of the critical difference

See Also

Other generate_plot_data: generateCalibrationData(), generateFeatureImportanceData(), generateFilterValuesData(), generateLearningCurveData(), generatePartialDependenceData(), generateThreshVsPerfData(), plotFilterValues()

Other benchmark: BenchmarkResult, batchmark(), benchmark(), convertBMRToRankMatrix(), friedmanPostHocTestBMR(), friedmanTestBMR(), getBMRAggrPerformances(), getBMRFeatSelResults(), getBMRFilteredFeatures(), getBMRLearnerIds(), getBMRLearnerShortNames(), getBMRLearners(), getBMRMeasureIds(), getBMRMeasures(), getBMRModels(), getBMRPerformances(), getBMRPredictions(), getBMRTaskDescs(), getBMRTaskIds(), getBMRTuneResults(), plotBMRBoxplots(), plotBMRRanksAsBarChart(), plotBMRSummary(), plotCritDifferences(), reduceBatchmarkResults()