mlr3 (version 0.1.0-9000)

BenchmarkResult: Container for Results of benchmark()

Description

This is the result container object returned by benchmark().

Note that all stored objects are accessed by reference. Do not modify any object without cloning it first.

Format

R6::R6Class object.

Construction

bmr = BenchmarkResult$new(data)

• data :: data.table::data.table() Table with data for one resampling iteration per row: Task, Learner, Resampling, iteration (integer(1)), Prediction, and the hash (character(1)) of the corresponding ResampleResult.

Fields

• data :: data.table::data.table() Internal data storage. We discourage users to directly work with this field.

• tasks :: data.table::data.table() Table of used tasks with three columns: "task_hash" (character(1)), "task_id" (character(1)) and "task" (Task).

• learners :: data.table::data.table() Table of used learners with three columns: "learner_hash" (character(1)), "learner_id" (character(1)) and "learner" (Learner).

• resamplings :: data.table::data.table() Table of used resamplings with three columns: "resampling_hash" (character(1)), "resampling_id" (character(1)) and "resampling" (Resampling).

Methods

• aggregate(measures = NULL, ids = TRUE, params = FALSE, warnings = FALSE, errors = FALSE) (list() of Measure, logical(1), logical(1)) -> data.table::data.table() Returns a result table where resampling iterations are aggregated together into ResampleResults. Arguments control the number of additional columns:

  • ids :: logical(1) Adds object ids ("task_id", "learner_id", "resampling_id") as extra character columns.

  • params :: logical(1) Adds the hyperparameter values as extra list column "params". You can unnest them with mlr3misc::unnest().

  • warnings :: logical(1) Adds the number of resampling iterations with at least one recorded warning as extra integer column "warnings".

  • errors :: logical(1) Adds the number of resampling iterations with at least one recorded error as extra integer column "errors".

• performance(measures = NULL, ids = TRUE) (list() of Measure, logical(1)) -> data.table::data.table() Returns a table with one row for each resampling iteration, including all involved objects. Additionally calculates the provided performance measures and binds the performance as extra column. If no measure is provided, defaults to the measure defined in mlr_reflections$default_measures (mlr_measures_classif.ce for classification and mlr_measures_regr.mse for regression). If ids is TRUE, character column of id names are added to the table for convenient filtering.

• best(measure) (Measure) -> ResampleResult Returns the ResampleResult with the best performance according to Measure.

• resample_result(hash) (character(1) -> ResampleResult) Retrieve the ResampleResult with hash.

• combine(bmr) BenchmarkResult -> self Fuses a second BenchmarkResult into itself.

S3 Methods

• as.data.table(bmr) BenchmarkResult -> data.table::data.table() Returns a copy of the internal data.

Examples

# NOT RUN {
set.seed(123)
tasks = mlr_tasks$mget(c("sonar", "spam"))
learners = mlr_learners$mget(c("classif.featureless", "classif.rpart"), predict_type = "prob")
resamplings = mlr_resamplings$get("cv3")
design = expand_grid(tasks = tasks, learners = learners, resamplings = resamplings)
print(design)

bmr = benchmark(design)
print(bmr)

bmr$tasks
bmr$learners

# first 5 individual resamplings
head(as.data.table(bmr, measures = c("classif.acc", "classif.auc")), 5)

# aggregate results
bmr$aggregate()

# aggregate results with hyperparameters as separate columns
mlr3misc::unnest(bmr$aggregate(params = TRUE), "params")

# extract resample result for classif.rpart
rr = bmr$aggregate()[learner_id == "classif.rpart", resample_result][[1]]
print(rr)

# access the confusion matrix of the first resampling iteration
rr$data$prediction[[1]]$confusion
# }