Simulated annealing is a global optimization method. For model tuning, it
can be used to iteratively search the parameter space for optimal tuning
parameter combinations. At each iteration, a new parameter combination is
created by perturbing the current parameters in some small way so that they
are within a small neighborhood. This new parameter combination is used to
fit a model and that model's performance is measured using resampling (or a
simple validation set).
If the new settings have better results than the current settings, they are
accepted and the process continues.
If the new settings has worse performance, a probability threshold is
computed for accepting these sub-optimal values. The probability is a
function of how sub-optimal the results are as well as how many iterations
have elapsed. This is referred to as the "cooling schedule" for the
algorithm. If the sub-optimal results are accepted, the next iterations
settings are based on these inferior results. Otherwise, new parameter
values are generated from the previous iteration's settings.
This process continues for a pre-defined number of iterations and the
overall best settings are recommended for use. The control_sim_anneal()
function can specify the number of iterations without improvement for early
stopping. Also, that function can be used to specify a restart threshold;
if no globally best results have not be discovered within a certain number
if iterations, the process can restart using the last known settings that
globally best.
Creating new settings
For each numeric parameter, the range of possible values is known as well
as any transformations. The current values are transformed and scaled to
have values between zero and one (based on the possible range of values). A
candidate set of values that are on a sphere with random radii between
rmin
and rmax
are generated. Infeasible values are removed and one value
is chosen at random. This value is back transformed to the original units
and scale and are used as the new settings. The argument radius
of
control_sim_anneal()
controls the range neighborhood sizes.
For categorical and integer parameters, each is changes with a pre-defined
probability. The flip
argument of control_sim_anneal()
can be used to
specify this probability. For integer parameters, a nearby integer value is
used.
Simulated annealing search may not be the preferred method when many of the
parameters are non-numeric or integers with few unique values. In these
cases, it is likely that the same candidate set may be tested more than
once.
Cooling schedule
To determine the probability of accepting a new value, the percent
difference in performance is calculated. If the performance metric is to be
maximized, this would be d = (new-old)/old*100
. The probability is
calculated as p = exp(d * coef * iter)
were coef
is a user-defined
constant that can be used to increase or decrease the probabilities.
The cooling_coef
of control_sim_anneal()
can be used for this purpose.
Termination criterion
The restart counter is reset when a new global best results is found.
The termination counter resets when a new global best is located or when a
suboptimal result is improved.
Parallelism
The tune
and finetune
packages currently parallelize over resamples.
Specifying a parallel back-end will improve the generation of the initial
set of sub-models (if any). Each iteration of the search are also run in
parallel if a parallel backend is registered.
Censored regression models
With dynamic performance metrics (e.g. Brier or ROC curves), performance is
calculated for every value of eval_time
but the first evaluation time
given by the user (e.g., eval_time[1]
) is used to guide the optimization.
Also, values of eval_time
should be less than the largest observed event
time in the training data. For many non-parametric models, the results beyond
the largest time corresponding to an event are constant (or NA
).