"Bonferroni-Dunn" test or the "Nemenyi" test.
"Bonferroni-Dunn" usually yields higher power as it does not
compare all algorithms to each other, but all algorithms to a
baseline instead.
Learners are drawn on the y-axis according to their average rank.
For test = "nemenyi" a bar is drawn, connecting all groups of not
significantly different learners.
For test = "bd" an interval is drawn arround the algorithm selected
as baseline. All learners within this interval are not signifcantly different
from the baseline.
Calculation:
$$ CD = q_{\alpha} \sqrt{(\frac{k(k+1)}{6N})}$$
Where $q_\alpha$ is based on the studentized range statistic.
See references for details.
generateCritDifferencesData(bmr, measure = NULL, p.value = 0.05, baseline = NULL, test = "bd")BenchmarkResult]
Benchmark result.Measure]
Performance measure.
Default is the first measure used in the benchmark experiment.numeric(1)]
P-value for the critical difference. Default: 0.05character(1)]: [learner.id]
Select a learner.id as baseline for the test = "bd"
("Bonferroni-Dunn") critical differences
diagram.The critical difference Interval will then be positioned arround this learner.
Defaults to best performing algorithm.
For test = "nemenyi", no baseline is needed as it performs all pairwise
comparisons.character(1)]
Test for which the critical differences are computed.
“bd” for the Bonferroni-Dunn Test, which is comparing all
classifiers to a baseline, thus performing a comparison
of one classifier to all others.
Algorithms not connected by a single line are statistically different.
then the baseline.
“nemenyi” for the posthoc.friedman.nemenyi.test
which is comparing all classifiers to each other. The null hypothesis that
there is a difference between the classifiers can not be rejected for all
classifiers that have a single grey bar connecting them.critDifferencesData]. List containing:
data.frame] containing the info for the descriptive
part of the plotlist] of class pairwise.htest
contains the calculated
posthoc.friedman.nemenyi.testlist] containing info on the critical difference
and its positioningbaseline chosen for plottingBenchmarkResult,
benchmark,
convertBMRToRankMatrix,
friedmanPostHocTestBMR,
friedmanTestBMR,
getBMRAggrPerformances,
getBMRFeatSelResults,
getBMRFilteredFeatures,
getBMRLearnerIds,
getBMRLearnerShortNames,
getBMRLearners,
getBMRMeasureIds,
getBMRMeasures, getBMRModels,
getBMRPerformances,
getBMRPredictions,
getBMRTaskIds,
getBMRTuneResults,
plotBMRBoxplots,
plotBMRRanksAsBarChart,
plotBMRSummary,
plotCritDifferencesOther generate_plot_data: generateCalibrationData,
generateFilterValuesData,
generateLearningCurveData,
generatePartialPredictionData,
generateThreshVsPerfData,
getFilterValues