batchtools v0.9.12

0

Monthly downloads

0th

Percentile

Tools for Computation on Batch Systems

As a successor of the packages 'BatchJobs' and 'BatchExperiments', this package provides a parallel implementation of the Map function for high performance computing systems managed by schedulers 'IBM Spectrum LSF' (<https://www.ibm.com/us-en/marketplace/hpc-workload-management>), 'OpenLava' (<http://www.openlava.org/>), 'Univa Grid Engine'/'Oracle Grid Engine' (<http://www.univa.com/>), 'Slurm' (<http://slurm.schedmd.com/>), 'TORQUE/PBS' (<https://adaptivecomputing.com/cherry-services/torque-resource-manager/>), or 'Docker Swarm' (<https://docs.docker.com/swarm/>). A multicore and socket mode allow the parallelization on a local machines, and multiple machines can be hooked up via SSH to create a makeshift cluster. Moreover, the package provides an abstraction mechanism to define large-scale computer experiments in a well-organized and reproducible way.

Readme

batchtools

JOSS Publicatoin CRAN Status Badge Build Status Build Status Coverage Status

As a successor of the packages BatchJobs and BatchExperiments, batchtools provides a parallel implementation of Map for high performance computing systems managed by schedulers like Slurm, Sun Grid Engine, OpenLava, TORQUE/OpenPBS, Load Sharing Facility (LSF) or Docker Swarm (see the setup section in the vignette).

Main features:

  • Convenience: All relevant batch system operations (submitting, listing, killing) are either handled internally or abstracted via simple R functions
  • Portability: With a well-defined interface, the source is independent from the underlying batch system - prototype locally, deploy on any high performance cluster
  • Reproducibility: Every computational part has an associated seed stored in a data base which ensures reproducibility even when the underlying batch system changes
  • Abstraction: The code layers for algorithms, experiment definitions and execution are cleanly separated and allow to write readable and maintainable code to manage large scale computer experiments

Installation

Install the stable version from CRAN:

install.packages("batchtools")

For the development version, use devtools:

devtools::install_github("mllg/batchtools")

Next, you need to setup batchtools for your HPC (it will run sequentially otherwise). See the vignette for instructions.

Why batchtools?

The development of BatchJobs and BatchExperiments is discontinued for the following reasons:

  • Maintainability: The packages BatchJobs and BatchExperiments are tightly connected which makes maintenance difficult. Changes have to be synchronized and tested against the current CRAN versions for compatibility. Furthermore, BatchExperiments violates CRAN policies by calling internal functions of BatchJobs.
  • Data base issues: Although we invested weeks to mitigate issues with locks of the SQLite data base or file system (staged queries, file system timeouts, ...), BatchJobs kept working unreliable on some systems with high latency under certain conditions. This made BatchJobs unusable for many users.

BatchJobs and BatchExperiments will remain on CRAN, but new features are unlikely to be ported back. The vignette contains a section comparing the packages.

Resources

Citation

Please cite the JOSS paper using the following BibTeX entry:

@article{,
  doi = {10.21105/joss.00135},
  url = {https://doi.org/10.21105/joss.00135},
  year  = {2017},
  month = {feb},
  publisher = {The Open Journal},
  volume = {2},
  number = {10},
  author = {Michel Lang and Bernd Bischl and Dirk Surmann},
  title = {batchtools: Tools for R to work on batch systems},
  journal = {The Journal of Open Source Software}
}
  • The High Performance Computing Task View lists the most relevant packages for scientific computing with R.
  • clustermq is a similar approach which also supports multiple schedulers. Uses the ZeroMQ network protocol for communication, and shines if you have millions of fast jobs.
  • batch assists in splitting and submitting jobs to LSF and MOSIX clusters.
  • flowr supports LSF, Slurm, TORQUE and Moab and provides a scatter-gather approach to define computational jobs.
  • future.batchtools implements batchtools as backend for future.
  • doFuture together with future.batchtools connects batchtools to foreach.
  • drake uses graphs to define computational jobs. batchtools is used as a backend via future.batchtools.

Contributing to batchtools

This R package is licensed under the LGPL-3. If you encounter problems using this software (lack of documentation, misleading or wrong documentation, unexpected behaviour, bugs, ...) or just want to suggest features, please open an issue in the issue tracker. Pull requests are welcome and will be included at the discretion of the author. If you have customized a template file for your (larger) computing site, please share it: fork the repository, place your template in inst/templates and send a pull request.

Functions in batchtools

Name Description
addExperiments Add Experiments to the Registry
batchtools-package batchtools: Tools for Computation on Batch Systems
estimateRuntimes Estimate Remaining Runtimes
cfReadBrewTemplate Cluster Functions Helper to Parse a Brew Template
doJobCollection Execute Jobs of a JobCollection
cfKillJob Cluster Functions Helper to Kill Batch Jobs
makeClusterFunctionsInteractive ClusterFunctions for Sequential Execution in the Running R Session
makeClusterFunctionsLSF ClusterFunctions for LSF Systems
batchMap Map Operation for Batch Systems
batchExport Export Objects to the Slaves
findJobs Find and Filter Jobs
batchMapResults Map Over Results to Create New Jobs
cfBrewTemplate Cluster Functions Helper to Write Job Description Files
grepLogs Grep Log Files for a Pattern
killJobs Kill Jobs
clearRegistry Remove All Jobs
execJob Execute a Single Jobs
makeClusterFunctionsSlurm ClusterFunctions for Slurm Systems
chunk Chunk Jobs for Sequential Execution
btlapply Synchronous Apply Functions
batchReduce Reduce Operation for Batch Systems
makeClusterFunctionsSocket ClusterFunctions for Parallel Socket Execution
makeClusterFunctionsMulticore ClusterFunctions for Parallel Multicore Execution
makeClusterFunctions ClusterFunctions Constructor
makeClusterFunctionsDocker ClusterFunctions for Docker
findTemplateFile Find a batchtools Template File
makeClusterFunctionsSGE ClusterFunctions for SGE Systems
findConfFile Find a batchtools Configuration File
makeClusterFunctionsSSH ClusterFunctions for Remote SSH Execution
loadRegistry Load a Registry from the File System
loadResult Load the Result of a Single Job
makeClusterFunctionsTORQUE ClusterFunctions for OpenPBS/TORQUE Systems
removeExperiments Remove Experiments
makeClusterFunctionsOpenLava ClusterFunctions for OpenLava
resetJobs Reset the Computational State of Jobs
removeRegistry Remove a Registry from the File System
showLog Inspect Log Files
runOSCommand Run OS Commands on Local or Remote Machines
getJobTable Query Job Information
makeSubmitJobResult Create a SubmitJobResult
cfHandleUnknownSubmitError Cluster Functions Helper to Handle Unknown Errors
getStatus Summarize the Computational Status
makeRegistry Registry Constructor
getErrorMessages Retrieve Error Messages
getDefaultRegistry Get and Set the Default Registry
makeExperimentRegistry ExperimentRegistry Constructor
submitJobs Submit Jobs to the Batch Systems
sweepRegistry Check Consistency and Remove Obsolete Information
saveRegistry Store the Registy to the File System
summarizeExperiments Quick Summary over Experiments
unwrap Unwrap Nested Data Frames
reduceResults Reduce Results
runHook Trigger Evaluation of Custom Function
waitForJobs Wait for Termination of Jobs
reduceResultsList Apply Functions on Results
syncRegistry Synchronize the Registry
testJob Run Jobs Interactively
JoinTables Inner, Left, Right, Outer, Semi and Anti Join for Data Tables
JobNames Set and Retrieve Job Names
Tags Add or Remove Job Tags
Worker Create a Linux-Worker
addAlgorithm Define Algorithms for Experiments
addProblem Define Problems for Experiments
assertRegistry assertRegistry
makeJobCollection JobCollection Constructor
makeJob Jobs and Experiments
No Results!

Vignettes of batchtools

Name
batchtools.Rmd
function_overview.pdf
function_overview.png
function_overview.tex
tikz_prob_algo_simple.pdf
tikz_prob_algo_simple.png
tikz_prob_algo_simple.tex
No Results!

Last month downloads

Details

Include our badge in your README

[![Rdoc](http://www.rdocumentation.org/badges/version/batchtools)](http://www.rdocumentation.org/packages/batchtools)