An R6 class for aggregated benchmark results.
data(data.table::data.table) Aggregated data.
learners(character()) Unique learner names.
tasks(character()) Unique task names.
measures(character()) Unique measure names.
nlrns(integer()) Number of learners.
ntasks(integer()) Number of tasks.
nmeas(integer()) Number of measures.
nrow(integer()) Number of rows.
col_roles(character())
Column roles, currently cannot be changed after construction.
new()Creates a new instance of this R6 class.
BenchmarkAggr$new( dt, task_id = "task_id", learner_id = "learner_id", independent = TRUE, strip_prefix = TRUE, ... )
dt(matrix(1)) '
matrix like object coercable to data.table::data.table, should
include column names "task_id" and "learner_id", and at least one measure (numeric).
If ids are not already factors then coerced internally.
task_id(character(1))
String specifying name of task id column.
learner_id(character(1))
String specifying name of learner id column.
independent(logical(1))
Are tasks independent of one another? Affects which tests can be used for analysis.
strip_prefix(logical(1))
If TRUE (default) then mlr prefixes, e.g. regr., classif., are automatically
stripped from the learner_id.
...ANY
Additional arguments, currently unused.
print()Prints the internal data via data.table::print.data.table.
BenchmarkAggr$print(...)
...ANY Passed to data.table::print.data.table.
summary()Prints the internal data via data.table::print.data.table.
BenchmarkAggr$summary(...)
...ANY Passed to data.table::print.data.table.
rank_data()Ranks the aggregated data given some measure.
BenchmarkAggr$rank_data(meas = NULL, minimize = TRUE, task = NULL, ...)
meas(character(1))
Measure to rank the data against, should be in $measures. Can be NULL if only one measure
in data.
minimize(logical(1))
Should the measure be minimized? Default is TRUE.
task(character(1))
If NULL then returns a matrix of ranks where columns are tasks and rows are
learners, otherwise returns a one-column matrix of a specified task, should
be in $tasks.
...ANY ANY Passed to data.table::frank().
friedman_test()Computes Friedman test over all tasks, assumes datasets are independent.
BenchmarkAggr$friedman_test(meas = NULL, p.adjust.method = NULL)
meas(character(1))
Measure to rank the data against, should be in $measures. If no measure is provided
then returns a matrix of tests for all measures.
p.adjust.method(character(1))
Passed to p.adjust if meas = NULL for multiple testing correction. If NULL
then no correction applied.
friedman_posthoc()Posthoc Friedman Nemenyi tests. Computed with
PMCMR::posthoc.friedman.nemenyi.test. If global $friedman_test is non-significant then
this is returned and no post-hocs computed. Also returns critical difference
BenchmarkAggr$friedman_posthoc(meas = NULL, p.value = 0.05)
meas(character(1))
Measure to rank the data against, should be in $measures. Can be NULL if only one measure
in data.
p.value(numeric(1))
p.value for which the global test will be considered significant.
subset()Subsets the data by given tasks or learners. Returns data as data.table::data.table.
BenchmarkAggr$subset(task = NULL, learner = NULL)
task(character())
Task(s) to subset the data by.
learner(character())
Learner(s) to subset the data by.
clone()The objects of this class are cloneable with this method.
BenchmarkAggr$clone(deep = FALSE)
deepWhether to make a deep clone.
This class is used to easily carry out and guide analysis of models after aggregating
the results after resampling. This can either be constructed using mlr3 objects,
for example the result of mlr3::BenchmarkResult$aggregate or via as.BenchmarkAggr,
or by passing in a custom dataset of results. Custom datasets must include at the very least,
a character column for learner ids, a character column for task ids, and numeric columns for
one or more measures.
Currently supported for multiple independent datasets only.
`r format_bib("demsar_2006")
# NOT RUN {
# Not restricted to mlr3 objects
df = data.frame(tasks = rep(c("A", "B"), each = 5),
learners = paste0("L", 1:5),
RMSE = runif(10), MAE = runif(10))
as.BenchmarkAggr(df, task_id = "tasks", learner_id = "learners")
if (requireNamespaces(c("mlr3", "rpart"))) {
library(mlr3)
task = tsks(c("boston_housing", "mtcars"))
learns = lrns(c("regr.featureless", "regr.rpart"))
bm = benchmark(benchmark_grid(task, learns, rsmp("cv", folds = 2)))
# coercion
as.BenchmarkAggr(bm)
}
# }
Run the code above in your browser using DataLab