FutureExecutionConfigParallel execution through future. A plan is required, and is filled in
from getOption("future.plan", "mirai_multisession") when not given.
setup_FutureExecution(
n_workers = NULL,
n_workers_outer = NULL,
n_workers_tuning = NULL,
n_workers_algorithm = NULL,
device = NULL,
future_plan = getOption("future.plan", "mirai_multisession"),
on_error = c("continue", "stop", "stop_outer"),
seed = NULL,
shared_memory = c("auto", "none", "always")
)FutureExecutionConfig object.
Integer [1, Inf): Number of parallel workers. Left NULL,
it follows the named dispatch levels, or default_n_workers() if none is named.
Optional Integer [1, Inf): Workers for outer resampling, overriding the automatic assignment.
Optional Integer [1, Inf): Workers for tuning, overriding the automatic assignment.
Optional Integer [1, Inf): Threads for a self-parallelizing algorithm, overriding the automatic assignment.
Optional DeviceConfig: Compute device, as a type name --
"cpu", "cuda", "mps" (the Apple silicon GPU) or "opencl" (a LightGBM
GPU build) -- or setup_CUDA to name which GPUs. NULL selects one per
algorithm: "cuda" where the machine has it and the algorithm can use it,
else the CPU. An algorithm that cannot use the requested device runs on the
CPU, and the run's resources line says so; a device the machine lacks is an
error when an algorithm tries to use it. "mps" is never selected
automatically: in rtemis benchmarks on an Apple M5 the CPU was faster for
every network narrower than about 1024 units or trained in batches smaller
than about 2048 cases, often by 2 to 5 times, and "mps" was faster by up to
about 25% only with both and thousands of features. It also slows down across
many fits in one R session, and a seed does not reach dropout on it.
Character: Future plan to use. Defaults to
getOption("future.plan", "mirai_multisession").
Character {"continue", "stop", "stop_outer"}: Failure policy.
"continue" makes grid cells and unscorable hyperparameter combinations
non-fatal (recorded, warned, and excluded), failing only when nothing is scorable or
the final model fails; "stop" aborts on any error; "stop_outer" tolerates grid-cell
failures but aborts on an outer-fold failure.
Optional Integer [0, Inf): Master seed for the run's computation RNG,
from which one independent substream per parallel task is derived. Left NULL, a
seed is drawn from the current RNG stream and recorded on the returned object, so
every run is reproducible and the seed it used is auditable. Distinct from a
ResamplerConfig seed, which governs how the data is split.
Character {"none", "auto", "always"}: Whether to hand workers
the training data through OS shared memory instead of serializing a copy to each.
"auto" is the default: it shares whenever it can -- workers parallel and on this
machine, mori installed -- and quietly does not when it cannot, falling back to
the ordinary transport. "none" disables it. "always" is a demand: it shares even
when the run is sequential, so a run can be compared against its own shared
counterpart, and raises an error when the request cannot be met, for a caller relying
on sharing to stay inside a memory budget.
EDG
Worker levels
There are three levels that can absorb workers: a self-parallelizing algorithm,
tuning, and outer resampling. Left alone, n_workers is assigned to exactly one of
them, in that order of priority, so no two levels compete for the same cores.
Naming any of n_workers_outer, n_workers_tuning or n_workers_algorithm takes
over that assignment: the automatic one is switched off entirely and any level not
named gets 1, so a run never mixes a hand-picked level with an inferred one.
Outer resampling and tuning dispatch to worker processes, and an outer fold runs
inside one of those processes, so only one of the two can be parallel -- setting both
above 1 is an error. n_workers_algorithm is threads within a worker, so it combines
with either.
Reproducibility
Substreams are assigned by task index, so a run gives the same answer under every backend and worker count, and the parallel result matches the sequential one exactly.
Shared memory needs mori, whose minimum R version (4.3) is above
rtemis's own (4.1), so "auto" is the portable setting for a script that may
run anywhere. It is local RAM: workers on another machine cannot map it.
setup_FutureExecution(n_workers = 4L, future_plan = "multisession")
Run the code above in your browser using DataLab