mlr (version 2.8)

makeCustomResampledMeasure: Construct your own resampled performance measure.

Description

Construct your own performance measure, used after resampling. Note that individual training / test set performance values will be set to NA, you only calculate an aggregated value. If you can define a function that makes sense for every single training / test set, implement your own Measure.

Usage

makeCustomResampledMeasure(measure.id, aggregation.id, minimize = TRUE, properties = character(0L), fun, extra.args = list(), best = NULL, worst = NULL, measure.name = measure.id, aggregation.name = aggregation.id, note = "")

Arguments

measure.id

[character(1)] Short name of measure.

aggregation.id

[character(1)] Short name of aggregation.

minimize

[logical(1)] Should the measure be minimized? Default is TRUE.

properties

[character] Set of measure properties. Some standard property names include:

classif

Is the measure applicable for classification?

classif.multi

Is the measure applicable for multi-class classification?

regr

Is the measure applicable for regression?

surv

Is the measure applicable for survival?

costsens

Is the measure applicable for cost-sensitive learning?

req.pred

Is prediction object required in calculation? Usually the case.

req.truth

Is truth column required in calculation? Usually the case.

req.task

Is task object required in calculation? Usually not the case

req.model

Is model object required in calculation? Usually not the case.

req.feats

Are feature values required in calculation? Usually not the case.

req.prob

Are predicted probabilites required in calculation? Usually not the case, example would be AUC.

Default is character(0).

fun

[function(task, group, pred, extra.args)] Calculates performance value from ResamplePrediction object. For rare cases you can also use the task, the grouping or the extra arguments extra.args.

task [Task]

The task.

group [factor]

Grouping of resampling iterations. This encodes whether specific iterations 'belong together' (e.g. repeated CV).

pred [Prediction]

Prediction object.

extra.args [list]

See below.

extra.args

[list] List of extra arguments which will always be passed to fun. Default is empty list.

best

[numeric(1)] Best obtainable value for measure. Default is -Inf or Inf, depending on minimize.

worst

[numeric(1)] Worst obtainable value for measure. Default is Inf or -Inf, depending on minimize.

measure.name

[character(1)] Long name of measure. Default is measure.id.

aggregation.name

[character(1)] Long name of the aggregation. Default is aggregation.id.

note

[character] Description and additional notes for the measure. Default is “”.

Value

[Measure].

See Also

Other performance: estimateRelativeOverfitting, makeCostMeasure, makeMeasure, measures, performance