slurmR (version 0.5-4)

Slurm_Map: The Slurm version of the *apply family of functions.

Description

The Slurm version of the *apply family of functions.

Usage

Slurm_Map(
  f,
  ...,
  njobs = 2L,
  mc.cores = 1L,
  job_name = opts_slurmR$get_job_name(),
  tmp_path = opts_slurmR$get_tmp_path(),
  plan = "collect",
  sbatch_opt = list(),
  rscript_opt = list(),
  seeds = NULL,
  compress = TRUE,
  export = NULL,
  export_env = NULL,
  libPaths = .libPaths(),
  hooks = NULL,
  overwrite = TRUE,
  preamble = NULL
)
Slurm_lapply(
  X,
  FUN,
  ...,
  njobs = 2L,
  mc.cores = 1L,
  job_name = opts_slurmR$get_job_name(),
  tmp_path = opts_slurmR$get_tmp_path(),
  plan = "collect",
  sbatch_opt = list(),
  rscript_opt = list(),
  seeds = NULL,
  compress = TRUE,
  export = NULL,
  export_env = NULL,
  libPaths = .libPaths(),
  hooks = NULL,
  overwrite = TRUE,
  preamble = NULL
)
Slurm_sapply(X, FUN, ..., simplify = TRUE, USE.NAMES = TRUE)

Value

If plan == "collect", then whatever the analogous function returns, otherwise, an object of class slurm_job.

Arguments

njobs

Integer. Number of jobs to use in the job-array. This specifies the number of R sessions to initialize. This does not specify the number of cores to be used.

job_name

Character. Name of the job to be passed to Slurm.

tmp_path

Character. Path to the directory where all the data (including scripts) will be stored. Notice that this path must be accessible by all the nodes in the network (See opts_slurmR).

plan

A character scalar. (See the_plan).

sbatch_opt

List of options to be passed to sbatch. This is usually done by adding the flags #SBATCH to the bash file.

rscript_opt

List. Options to be passed to Rscript.

seeds

Integer vector of length njobs. Seeds to be passed to each job. When NULL (default), seeds will be picked automatically (see new_rscript).

compress

Logical scalar (default TRUE). Passed to saveRDS. Setting this value to FALSE can be useful when the user requires faster read/write of R objects on disk.

export

A named list with objects to be included in the Spawned sessions.

export_env

An environment. Environment where the objects listed in export are located (default parent.frame()).

libPaths

A character vector. See .libPaths.

hooks

A list of functions (passed to new_slurm_job).

overwrite

Logical scalar. When TRUE, if the path specified by tmp_path/job_name already exists, it will overwrite it, otherwise the function returns with an error.

preamble

Character vector. Each element is then added to the Slurm batch file between the #SBATCH options and the script's main call. A common example is adding required modules, e.g. c("module load gcc/6.1.1").

X, FUN, f, mc.cores, ...

Arguments passed to either parallel::mclapply or parallel::mcMap.

simplify, USE.NAMES

Logical scalar. See sapply.

Details

The function Slurm_lapply will submit njobs to the queue and distribute X according to parallel::splitIndices. For example, if X is list with 1,000 elements, and njobs = 2, then Slurm_lapply will submit 2 jobs with 500 elements of X each (2 chunks of data). The same principle applies to Slurm_sapply and Slurm_Map, this is, the data is split by chunks so all the information is sent at once when the job is submitted.

Just like sapply is to lapply, Slurm_sapply is just a wrapper of Slurm_lapply with an extra argument, simplify. When TRUE, once the job is collected, the function simplify2array is called.

References

Job Array Support https://slurm.schedmd.com/job_array.html

See Also

For resubmitting a job, see the example in sbatch.

Examples

if (FALSE) {
  # A job drawing 1e6 uniforms on 10 jobs (array)
  # The option plan = "wait" makes it return only once the job is completed.
  job1 <- Slurm_lapply(1:20, function(i) runif(1e6), njobs=10, plan = "wait")

  # To collect
  ans <- Slurm_collect(job1)

  # As before, but this time not waiting, and now we are passing more
  # arguments to the function
  # plan = "none" only creates the job object (and the files), we submit
  # later
  job1 <- Slurm_lapply(1:20, function(i, a) runif(1e6, a), a = -1, njobs=10,
      plan = "none")

  # We submit
  job1 <- sbatch(job1)

  # In order to cancel a job
  scancel(job1)

  # How to clean up
  Slurm_clean(job1)
}