clustermq (version 0.8.6)

Q: Queue function calls on the cluster

Description

Queue function calls on the cluster

Usage

Q(fun, ..., const = list(), export = list(), seed = 128965,
  memory = NULL, template = list(), n_jobs = NULL, job_size = NULL,
  split_array_by = -1, rettype = "list", fail_on_error = TRUE,
  workers = NULL, log_worker = FALSE, chunk_size = NA,
  timeout = Inf)

Arguments

fun

A function to call

...

Objects to be iterated in each function call

const

A list of constant arguments passed to each function call

export

List of objects to be exported to the worker

seed

A seed to set for each function call

memory

Short for template=list(memory=value)

template

A named list of values to fill in template

n_jobs

The number of LSF jobs to submit; upper limit of jobs if job_size is given as well

job_size

The number of function calls per job

split_array_by

The dimension number to split any arrays in `...`; default: last

rettype

Return type of function call (vector type or 'list')

fail_on_error

If an error occurs on the workers, continue or fail?

workers

Optional instance of QSys representing a worker pool

log_worker

Write a log file for each worker

chunk_size

Number of function calls to chunk together defaults to 100 chunks per worker or max. 10 kb per chunk

timeout

Maximum time in seconds to wait for worker (default: Inf)

Value

A list of whatever `fun` returned

Examples

# NOT RUN {
# Run a simple multiplication for numbers 1 to 3 on a worker node
fx = function(x) x * 2
Q(fx, x=1:3, n_jobs=1)
# list(2,4,6)

# Run a mutate() call in dplyr on a worker node
iris %>%
    mutate(area = Q(`*`, e1=Sepal.Length, e2=Sepal.Width, n_jobs=1))
# iris with an additional column 'area'
# }