tar_make_clustermq: Run a pipeline of targets in parallel with persistent `clustermq` workers.

Description

This function is like tar_make() except that targets run in parallel with persistent clustermq workers. It requires that you set global options like clustermq.scheduler and clustermq.template inside the _targets.R script. clustermq is not a strict dependency of targets, so you must install clustermq yourself.

Usage

tar_make_clustermq(
  names = NULL,
  reporter = Sys.getenv("TAR_MAKE_REPORTER", unset = "verbose"),
  workers = 1L,
  log_worker = FALSE,
  callr_function = callr::r,
  callr_arguments = targets::callr_args_default(callr_function, reporter)
)

Arguments

names

Names of the targets to build or check. Set to NULL to check/build all the targets (default). Otherwise, you can supply symbols, a character vector, or tidyselect helpers like starts_with().

reporter

Character of length 1, name of the reporter to user. Controls how messages are printed as targets run in the pipeline. Defaults to the TAR_MAKE_REPORTER environment variable if set and "verbose" otherwise. Choices:

"verbose": print one message for each target that runs (default).
"silent": print nothing.
"timestamp": print a time-stamped message for each target that runs.
"summary": print a running total of the number of each targets in each status category (queued, started, skipped, build, canceled, or errored). Also show a timestamp ("%H:%M %OS2" strptime() format) of the last time the progress changed and printed to the screen.

workers

Positive integer, number of persistent clustermq workers to create.

log_worker

Logical, whether to write a log file for each worker. Same as the log_worker argument of clustermq::Q() and clustermq::workers().

callr_function

A function from callr to start a fresh clean R process to do the work. Set to NULL to run in the current session instead of an external process (but restart your R session just before you do in order to clear debris out of the global environment). callr_function needs to be NULL for interactive debugging, e.g. tar_option_set(debug = "your_target"). However, callr_function should not be NULL for serious reproducible work.

callr_arguments

A list of arguments to callr_function.

Value

NULL except if callr_function = callr::r_bg(), in which case a handle to the callr background process is returned. Either way, the value is invisibly returned.

Details

To use with a cluster, you will need to set the global options clustermq.scheduler and clustermq.template inside _targets.R. To read more about configuring clustermq for your scheduler, visit https://mschubert.github.io/clustermq/articles/userguide.html#configuration # nolint and navigate to the appropriate link under "Setting up the scheduler". Wildcards in the template file are filled in with elements from tar_option_get("resources").

Examples

Run this code

# NOT RUN {
if (!identical(tolower(Sys.info()[["sysname"]]), "windows")) {
if (identical(Sys.getenv("TAR_LONG_EXAMPLES"), "true")) {
tar_dir({ # tar_dir() runs code from a temporary directory.
tar_script({
  options(clustermq.scheduler = "multicore") # Does not work on Windows.
  tar_option_set()
  list(tar_target(x, 1 + 1))
}, ask = FALSE)
tar_make_clustermq()
})
}
}
# }