taskqueue (version 0.2.0)

worker_slurm: Create a worker on slurm cluster

Description

Create a worker on slurm cluster

Usage

worker_slurm(
  project,
  resource,
  fun,
  rfile,
  module_r = "R/4.3.1",
  module_pg = "postgresql/16.0",
  modules = NULL,
  pkgs = rev(.packages()),
  submit = TRUE,
  ...
)

Value

no return

Arguments

project

Project name.

resource

Resource name.

fun

Function running on workers. See details.

rfile

R script file path. See details.

module_r

Module name for R.

module_pg

Module name for postgresql. See details.

modules

extra modules to load in slurm. See details.

pkgs

A character vector containing the names of packages that must be loaded on worker including all packages in default when worker_slurm is called.

submit

Whether to submit to slurm cluster (TRUE in default). See details.

...

Extra arguments for fun.

Details

There are two ways to pass R scripts into workers (i.e. fun or file). * fun is used for general and simple case which takes the task id as the first argument. A new r script is created in the log folder and running in the workers. The required packages are passed using pkgs. Extra arguments are specified through .... taskqueue_options() is passed into workers. * rfile is used more complicated case. Function worker has to be called at the end of file. No taskqueue_options() is passed into workers. * fun is higher priority with file. A submit file is created in the log folder for each project/resource with random file name. Then system command ssh is used to connect remote slurm host if submit = TRUE.

Examples

if (FALSE) {
# Not run:
fun_test <- function(i, prefix) {
    Sys.sleep(runif(1) * 2)
}
worker_slurm("test_project", "slurm", fun = fun_test)
worker_slurm("test_project", "slurm", fun = fun_test, prefix = "a")
worker_slurm("test_project", "slurm", rfile = "rfile.R")
worker_slurm("test_project", "slurm", fun = fun_test, submit = FALSE)
}