{"Package":"doRedis","Type":"Package","Title":"'Foreach' Parallel Adapter Using the 'Redis' Database","Version":"2.0.0","Date":"2020-01-26","Author":"B. W. Lewis ","Maintainer":"B. W. Lewis ","Description":"A parallel back end for the 'foreach' package using the 'Redis'\ndatabase.","BugReports":"https://github.com/bwlewis/doRedis/issues","Depends":"R (>= 3.0), foreach(>= 1.3.0), iterators(>= 1.0.0), utils","Imports":"parallel, redux, stats","Suggests":"knitr","VignetteBuilder":"knitr","License":"GPL-2","LazyLoad":"yes","RoxygenNote":"7.0.2","NeedsCompilation":"yes","Packaged":"2020-01-28 17:10:20 UTC; blewis","Repository":"CRAN","Date/Publication":"2020-01-28 17:30:03 UTC","repoType":"cran","tarballUrl":"ftp://cran.r-project.org/pub/R/src/contrib/doRedis_2.0.0.tar.gz","readme":"# doRedis: A parallel backend for foreach using Redis.\n\n## Important change in version 2.0.0\n\nVersion 2 and greater of the doRedis package now depend on the redux package\n(see https://cran.r-project.org/package=redux)\nfor communication with Redis instead of the deprecated rredis package.\n\n## Important Redis configuration notes\n\nSet the following parameter in your redis.conf file before using doRedis:\n\n```\ntimeout 0\n```\n\nExercise caution when using `doRedis` together with `doMC` or any fork-based R\nfunctions like `mclapply`. If you require a local inner parallel code section,\nconsider using `parLapply` and `makePSOCKcluster` or the related `doParallel`\nfunctions instead of fork-based methods. The fork-based functions can work in\nsome cases, but might also lead to trouble because the children share certain\nresources with the parent process like open socket descriptors. I have in\nparticular run in to trouble with some fast BLAS libraries and fork--in\nparticular the AMD ACML can't be used in this way at all. Again, exercise\ncaution with fork and `doRedis`!\n\n## Description\n\nSteve Weston's foreach package is a remarkable parametric evaluation device for\nthe R language. Similarly to lapply-like functions, foreach maps and parameter\nvalues expressions to data and aggregates results. Even better, foreach lets\nyou do this in parallel across multiple CPU cores and computers. And even\nbetter yet, foreach abstracts the parallel computing details away into modular\nback-end code. Code written using foreach works sequentially in the absence of\na parallel back-end, and works uniformly across a variety of back ends.\nThink of foreach as the lingua Franca of parallel computing for R.\n\nRedis is a powerful, fast networked database with many innovative features,\namong them a blocking stack-like data structure (Redis \"lists\"). This feature\nmakes Redis useful as a lightweight backend for parallel computing. The\ndoRedis package relies on the redux package for communication with a Redis\nserver to define a lightweight parallel backend for foreach using Redis that is\nelastic and platform-independent.\n\nHere is a quick example procedure for experimenting with doRedis:\n\n1. Install Redis on your computer.\n2. Install foreach, redux and doRedis packages.\n3. Start the redis server running (see the redis documentation). We assume\n that the server is running on the host \"localhost\" and port 6379 (the\n default Redis port). We assume in the examples below that the worker R\n processes and the master are running on the same machine. In practice,\n they can of course run across a network.\n4. Open one or more R sessions that will act as back-end worker processes. \n Run the following in each session:\n```r\n require('doRedis')\n redisWorker('jobs')\n```\n (The R session will display status messages but otherwise block for\n work.)\n Note: You can add more workers to a work queue at any time. Also note\n that each back-end worker may advertise for work on multiple queues\n simultaneously (see the documentation and examples).\n5. Open another R session that will act as the master process. Run the\n following example (a simple sampling approximation of pi):\n```r\n require('doRedis')\n registerDoRedis('jobs')\n foreach(j=1:10,.combine=sum, .multicombine=TRUE) %dopar%\n 4*sum((runif(1000000) ^ 2 + runif(1000000) ^ 2) < 1) / 10000000\n removeQueue('jobs')\n```\n\nLet's define a few terms before we describe how the above example works:\n\n* A _loop iteration_ is the foreach expression together with a single\n loop parameter value.\n* A _task_ is a collection of loop iterations.\n* Given a foreach expression, a _job_ is the collection of tasks that\n make up the full set of loop iterations.\n* A _work queue_ is a collection of of any number of _tasks_ associated\n with number of _jobs_ submitted by one or more master R processes.\n\nThe \"jobs\" parameter above in the `redisWorker` and `registerDoRedis` function\nnames a Redis key used to transfer data between master and worker processes.\nThink of this name as a reference to a work queue. The master places tasks into\nthe queue, worker R processes pull tasks out of the queue and then return their\nresults to an associated result queue.\n\nThe doRedis parallel backend supports dynamic pools of back-end workers. New\nworkers may be added to work queues at any time and can be immediately used by\nrunning foreach computations.\n\nThe doRedis backend accepts a parameter called `chunkSize` that sets the number\nof loop iterations doled out per task, by default one. Optionally set this with\nthe `setChunkSize` function. Increasing `chunkSize` can improve performance for\nquick-running function evaluations by cutting down on the number of tasks.\nHere is an example that sets `chunkSize` to 100:\n\n```r\nforeach(j=1:500, .options.redis=list(chunkSize=100)) %dopar% ...\n```\n\nSetting `chunkSize` too large will adversely impact load-balancing across\nthe workers. For instance, setting `chunkSize` to the total number of loop\niterations will run everything sequentially on one worker!\n\nThe `redisWorker` function is used to manually invoke worker processes that\nadvertise for job tasks on one or more work queues. The function also has\nparameters for a Redis host, port number and password. For example, if the\nRedis server is running on a host called \"Cazart\" with the default Redis port\n6379:\n```\nredisWorker('jobs', host='Cazart', port=6379)\n```\n\nThe `registerDoRedis` function also contains host and port parameters.\nNeither the worker nor master R session needs to be running on the same\nmachine as the Redis server.\n\nThe `startLocalWorkers` function invokes one or more background R worker\nprocesses on the _local_ machine (internally using the `redisWorker` function).\nIt's a convenient way to invoke several workers at once on your local box.\n\nWorkers self-terminate when their work queues have been deleted with the\n`removeQueue` function.\n\n## Status\n\n $\"Travis$ \n\n\n $\"CRAN$ \n\n","jobInfo":{"package":"doRedis","version":"2.0.0","parsingStatus":"success","parserVersion":1,"parsedAt":"2020-01-28T19:29:53+0000"}}

Description

Usage

Arguments

See Also