The technical details of this method are described in Kuhn (2014).
Racing methods are efficient approaches to grid search. Initially, the
function evaluates all tuning parameters on a small initial set of
resamples. The burn_in
argument of control_race()
sets the number of
initial resamples.
The performance statistics from the current set of resamples are converted
to win/loss/tie results. For example, for two parameters (j
and k
) in a
classification model that have each been resampled three times:
| area under the ROC curve |
-----------------------------
resample | parameter j | parameter k | winner
---------------------------------------------
1 | 0.81 | 0.92 | k
2 | 0.95 | 0.94 | j
3 | 0.79 | 0.81 | k
---------------------------------------------
After the third resample, parameter k
has a 2:1 win/loss ratio versus j
.
Parameters with equal results are treated as a half-win for each setting.
These statistics are determined for all pairwise combinations of the
parameters and a Bradley-Terry model is used to model these win/loss/tie
statistics. This model can compute the ability of a parameter combination to
win overall. A confidence interval for the winning ability is computed and
any settings whose interval includes zero are retained for future resamples
(since it is not statistically different form the best results).
The next resample is used with the remaining parameter combinations and the
statistical analysis is updated. More candidate parameters may be excluded
with each new resample that is processed.
The control_race()
function contains are parameter for the significance cutoff
applied to the Bradley-Terry model results as well as other relevant arguments.
Censored regression models
With dynamic performance metrics (e.g. Brier or ROC curves), performance is
calculated for every value of eval_time
but the first evaluation time
given by the user (e.g., eval_time[1]
) is analyzed during racing.
Also, values of eval_time
should be less than the largest observed event
time in the training data. For many non-parametric models, the results beyond
the largest time corresponding to an event are constant (or NA
).