future (version 1.10.0)

makeClusterPSOCK: Create a PSOCK cluster of R workers for parallel processing

Description

The makeClusterPSOCK() function creates a cluster of R workers for parallel processing. These R workers may be background R sessions on the current machine, R sessions on external machines (local or remote), or a mix of such. For external workers, the default is to use SSH to connect to those external machines. This function works similarly to makePSOCKcluster() of the parallel package, but provides additional and more flexibility options for controlling the setup of the system calls that launch the background R workers, and how to connect to external machines.

Usage

makeClusterPSOCK(workers, makeNode = makeNodePSOCK, port = c("auto",
  "random"), ..., autoStop = FALSE, verbose = getOption("future.debug",
  FALSE))
makeNodePSOCK(worker = "localhost", master = NULL, port,
  connectTimeout = getOption("future.makeNodePSOCK.connectTimeout", 2 *
  60), timeout = getOption("future.makeNodePSOCK.timeout", 30 * 24 * 60 *
  60), rscript = NULL, homogeneous = NULL, rscript_args = NULL,
  methods = TRUE, useXDR = TRUE, outfile = "/dev/null",
  renice = NA_integer_,
  rshcmd = getOption("future.makeNodePSOCK.rshcmd", NULL), user = NULL,
  revtunnel = TRUE, rshopts = getOption("future.makeNodePSOCK.rshopts",
  NULL), rank = 1L, manual = FALSE, dryrun = FALSE,
  verbose = FALSE)

Value

An object of class c("SOCKcluster", "cluster") consisting of a list of "SOCKnode" or "SOCK0node" workers.

makeNodePSOCK() returns a "SOCKnode" or "SOCK0node" object representing an established connection to a worker.

Definition of <em>localhost</em>

A hostname is considered to be localhost if it equals:

• "localhost",

• "127.0.0.1", or

• Sys.info()[["nodename"]].

It is also considered localhost if it appears on the same line as the value of Sys.info()[["nodename"]] in file /etc/hosts.

Default values of arguments <code>rshcmd</code> and <code>rshopts</code>

Arguments rshcmd and rshopts are only used when connecting to an external host.

The default method for connecting to an external host is via SSH and the system executable for this is given by argument rshcmd. The default is given by option future.makeNodePSOCK.rshcmd and if that is not set the default is either of ssh and plink -ssh. Most Unix-like systems, including macOS, have ssh preinstalled on the PATH. This is also true for recent Windows 10 (since version 1803; April 2018). Furthermore, when running R from RStudio on Windows, the ssh client that is distributed with RStudio will be used as a fallback if neither of the above two commands are available on the PATH.

For Windows systems prior to Windows 10, which do not have RStudio installed, it is less common to find ssh. Instead it is more likely that such systems have the PuTTY software and its SSH client plink installed. If no SSH-client is found, an informative error message is produced.

It is also possible to specify the absolute path to the SSH client. To do this for PuTTY, specify the absolute path in the first element and option -ssh in the second as in rshcmd = c("C:/Path/PuTTY/plink.exe", "-ssh"). This is because all elements of rshcmd are individually "shell" quoted and element rshcmd[1] must be on the system PATH.

Additional SSH options may be specified via argument rshopts, which defaults to option future.makeNodePSOCK.rshopts. For instance, a private SSH key can be provided as rshopts = c("-i", "~/.ssh/my_private_key"). PuTTY users should specify a PuTTY PPK file, e.g. rshopts = c("-i", "C:/Users/joe/.ssh/my_keys.ppk"). Contrary to rshcmd, elements of rshopts are not quoted.

Reverse SSH tunneling

The default is to use reverse SSH tunneling (revtunnel = TRUE) for workers running on other machines. This avoids the complication of otherwise having to configure port forwarding in firewalls, which often requires static IP address as well as privileges to edit the firewall, something most users don't have. It also has the advantage of not having to know the internal and / or the public IP address / hostname of the master. Yet another advantage is that there will be no need for a DNS lookup by the worker machines to the master, which may not be configured or is disabled on some systems, e.g. compute clusters.

Default value of argument <code>rscript</code>

If homogeneous is FALSE, the rscript defaults to "Rscript", i.e. it is assumed that the Rscript executable is available on the PATH of the worker. If homogeneous is TRUE, the rscript defaults to file.path(R.home("bin"), "Rscript"), i.e. it is basically assumed that the worker and the caller share the same file system and R installation.

Default value of argument <code>homogeneous</code>

The default value of homogeneous is TRUE if and only if either of the following is fulfilled:

• worker is localhost

• revtunnel is FALSE and master is localhost

• worker is neither an IP number nor a fully qualified domain name (FQDN). A hostname is considered to be a FQDN if it contains one or more periods

In all other cases, homogeneous defaults to FALSE.

Connection time out

Argument connectTimeout does not work properly on Unix and macOS due to limitation in R itself. For more details on this, please see R-devel thread 'BUG?: On Linux setTimeLimit() fails to propagate timeout error when it occurs (works on Windows)' on 2016-10-26 (https://stat.ethz.ch/pipermail/r-devel/2016-October/073309.html). When used, the timeout will eventually trigger an error, but it won't happen until the socket connection timeout timeout itself happens.

Communication time out

If there is no communication between the master and a worker within the timeout limit, then the corresponding socket connection will be closed automatically. This will eventually result in an error in code trying to access the connection.

Examples

# NOT RUN {
## Setup of three R workers on two remote machines are set up
workers <- c("n1.remote.org", "n2.remote.org", "n1.remote.org")
cl <- makeClusterPSOCK(workers, dryrun = TRUE)

## Same setup when the two machines are on the local network and
## have identical software setups
cl <- makeClusterPSOCK(
  workers,
  revtunnel = FALSE, homogeneous = TRUE,
  dryrun = TRUE
)

## Setup of remote worker with more detailed control on
## authentication and reverse SSH tunnelling
cl <- makeClusterPSOCK(
  "remote.server.org", user = "johnny",
  ## Manual configuration of reverse SSH tunnelling
  revtunnel = FALSE,
  rshopts = c("-v", "-R 11000:gateway:11942"),
  master = "gateway", port = 11942,
  ## Run Rscript nicely and skip any startup scripts
  rscript = c("nice", "/path/to/Rscript"),
  rscript_args = c("--vanilla"),
  dryrun = TRUE
)

## Setup of 2 Docker workers running rocker/r-base
## (requires installation of future package)
cl <- makeClusterPSOCK(
  rep("localhost", times = 2L),
  ## Launch Rscript inside Docker container
  rscript = c(
    "docker", "run", "--net=host", "rocker/r-base",
    "Rscript"
  ),
  ## Install future package
  rscript_args = c(
    "-e", shQuote("install.packages('future')")
  ),
  dryrun = TRUE
)
                       

## Setup of a single udocker.py worker running rocker/r-base
## (requires installation of future package and extra quoting)
cl <- makeClusterPSOCK(
  "localhost",
  ## Launch Rscript inside Docker container (using udocker)
  rscript = c(
    "udocker.py", "run", "rocker/r-base",
    "Rscript"
  ), 
  ## Install future package and manually launch parallel workers
  ## (need double shQuote():s because udocker.py drops one level)
  rscript_args = c(
    "-e", shQuote(shQuote("install.packages('future')")),
    "-e", shQuote(shQuote("parallel:::.slaveRSOCK()"))
  ),
  dryrun = TRUE
)


## Launching worker on Amazon AWS EC2 running one of the
## Amazon Machine Images (AMI) provided by RStudio
## (http://www.louisaslett.com/RStudio_AMI/)
public_ip <- "1.2.3.4"
ssh_private_key_file <- "~/.ssh/my-private-aws-key.pem"
cl <- makeClusterPSOCK(
  ## Public IP number of EC2 instance
  public_ip,
  ## User name (always 'ubuntu')
  user = "ubuntu",
  ## Use private SSH key registered with AWS
  rshopts = c(
    "-o", "StrictHostKeyChecking=no",
    "-o", "IdentitiesOnly=yes",
    "-i", ssh_private_key_file
  ),
  ## Set up .libPaths() for the 'ubuntu' user and
  ## install future package
  rscript_args = c(
    "-e", shQuote("local({
      p <- Sys.getenv('R_LIBS_USER')
      dir.create(p, recursive = TRUE, showWarnings = FALSE)
      .libPaths(p)
    })"),
    "-e", shQuote("install.packages('future')")
  ),
  dryrun = TRUE
)


## Launching worker on Google Cloud Engine (GCE) running a
## container based VM (with a #cloud-config specification)
public_ip <- "1.2.3.4"
user <- "johnny"
ssh_private_key_file <- "~/.ssh/google_compute_engine"
cl <- makeClusterPSOCK(
  ## Public IP number of GCE instance
  public_ip,
  ## User name (== SSH key label (sic!))
  user = user,
  ## Use private SSH key registered with GCE
  rshopts = c(
    "-o", "StrictHostKeyChecking=no",
    "-o", "IdentitiesOnly=yes",
    "-i", ssh_private_key_file
  ),
  ## Launch Rscript inside Docker container
  rscript = c(
    "docker", "run", "--net=host", "rocker/r-base",
    "Rscript"
  ),
  ## Install future package
  rscript_args = c(
    "-e", shQuote("install.packages('future')")
  ),
  dryrun = TRUE
)


## Connect to remote Unix machine 'remote.server.org' on port 2200
## as user 'bob' from a Windows machine with PuTTY installed
cl <- makeClusterPSOCK(
  "remote.server.org", user = "bob",
  rshopts = c("-P", 2200, "-i", "C:/Users/bobby/.ssh/putty.ppk"),
  dryrun = TRUE
)
# }