DockerParallel (version 1.0.4)

makeDockerCluster: Create a docker cluster

Description

Create a docker cluster. The user needs to provide a cloud provider and a worker container to make it work.

Usage

makeDockerCluster(
  cloudProvider = NULL,
  workerContainer = NULL,
  workerNumber = 1,
  workerCpu = 1024,
  workerMemory = 2048,
  workerHardwareId = character(0),
  serverCpu = 256,
  serverMemory = 2048,
  serverHardwareId = character(0),
  jobQueueName = "DockerParallelQueue",
  privateServerData = NULL,
  serverContainer = getServerContainer(workerContainer),
  stopClusterOnExit = TRUE,
  verbose = 1
)

Arguments

cloudProvider

A CloudProvider object, the cloud that the container will be deployed

workerContainer

A DockerContainer object, the object that defines the worker container

workerNumber

Integer, the number of workers in the cluster

serverCpu, workerCpu

Integer, the CPU unit used by the server or each worker. 1024 CPU unit corresponds to a physical CPU core.

serverMemory, workerMemory

Integer, the memory used by the server or each worker in MB

serverHardwareId, workerHardwareId

Character, the ID of the hardware, this argument might be ignored by some cloud providers.

jobQueueName

Character, the job queue name used by the cluster to send the job.

privateServerData

A data object made from CloudPrivateServer. If this object is provided, the cluster server should be from another source and the cloud provider will not deploy the server container.

serverContainer

A DockerContainer object, the object that defines the server container.

stopClusterOnExit

Logical, whether to stop the cluster when the cluster has been removed from the R session. The default value is TRUE.

verbose

Integer, the verbose level

Value

A DockerCluster object

Details

This is the core function of the DockerParallel package which defines the cluster object. To user the function, you need to at least provide the cloud provider and worker container. Currently we have ECSFargateProvider and BiocFERContainer, see example.

Examples

# NOT RUN {
## Load the ECS fargate provider
library(ECSFargateProvider)
provider <- ECSFargateProvider()

## Load the bioconductor foreach redis container
container <- BiocFERWorkerContainer()

## Define a cluster with 2 workers,
## each worker use one fourth CPU core and 512 MB memory
cluster <- makeDockerCluster(cloudProvider = provider,
                             workerContainer = container,
                             workerNumber = 2,
                             workerCpu = 256, workerMemory = 512)
## Start the cluster
cluster$startCluster()

## rescale the worker number
cluster$setWorkerNumber(4)

## Use foreach to do the parallel computing
library(foreach)
getDoParWorkers()
foreach(x= 1:4)%dopar%{
    Sys.info()
}
# }