{"Type":"Package","Package":"botor","Authors@R":"c(\nperson(\"Gergely\", \"Darczi\", , \"daroczig@rapporter.net\", role = c(\"aut\", \"cre\"), comment = c(ORCID = \"0000-0003-3149-8537\")),\nperson(\"System1\", role = c(\"fnd\"))\n)","Title":"'AWS Python SDK' ('boto3') for R","Description":"Fork-safe, raw access to the 'Amazon Web Services' ('AWS') 'SDK' via the 'boto3' 'Python' module, and convenient helper functions to query the 'Simple Storage Service' ('S3') and 'Key Management Service' ('KMS'), partial support for 'IAM', the 'Systems Manager Parameter Store' and 'Secrets Manager'.","SystemRequirements":"boto3 (https://aws.amazon.com/sdk-for-python)","Version":"0.2.0","Date":"2019-09-23","URL":"https://daroczig.github.io/botor","BugReports":"https://github.com/daroczig/botor/issues","RoxygenNote":"6.1.1","License":"AGPL-3","Encoding":"UTF-8","Imports":"utils, reticulate, checkmate, logger","Suggests":"testthat, covr, digest","NeedsCompilation":"no","Packaged":"2019-09-23 00:39:27 UTC; daroczig","Author":"Gergely Darczi [aut, cre] (),\nSystem1 [fnd]","Maintainer":"Gergely Darczi ","Repository":"CRAN","Date/Publication":"2019-10-21 17:40:02 UTC","repoType":"cran","tarballUrl":"ftp://cran.r-project.org/pub/R/src/contrib/botor_0.2.0.tar.gz","jsonAuthors":[{"name":"Gergely Darczi","email":"daroczig@rapporter.net","maintainer":true},{"name":"System1 ","email":null,"maintainer":false}],"readme":"# botor: Reticulate wrapper on 'boto3'\n\n[![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active) ![CRAN](https://www.r-pkg.org/badges/version/botor) [![Build Status](https://travis-ci.org/daroczig/botor.svg?branch=master)](https://travis-ci.org/daroczig/botor) [![Code Coverage](https://codecov.io/gh/daroczig/botor/branch/master/graph/badge.svg)](https://codecov.io/gh/daroczig/botor)\n\nThis R package provides raw access to the 'Amazon Web Services' ('AWS') 'SDK' via the 'boto3' Python module and some convenient helper functions (currently for S3 and KMS) and workarounds, eg taking care of spawning new resources in forked R processes.\n\n## Installation\n\nThis package requires Python to be installed along with the `boto3` Python module, which can be installed from R via:\n\n```r\nreticulate::py_install('boto3')\n```\n\nIf that might result in technical problems that you cannot solve, then it's probably easier to install a standalone Python along with the system dependencies etc via [`rminiconda`](https://github.com/hafen/rminiconda).\n\nOnce the Python dependencies are resolved, you can either install from CRAN or the most recent (development version) of `botor` can be installed from GitHub:\n\n```r\nremotes::install_github('daroczig/botor')\n```\n\n## Loading the package\n\nLoading the `botor` package might take a while as it will also `import` the `boto3` Python module in the background:\n\n```r\nsystem.time(library(botor))\n#> user system elapsed \n#> 1.131 0.250 1.191\n```\n\n## Getting started\n\nQuick examples:\n\n1. Check the currently used AWS user's name:\n\n ```r\n iam_whoami()\n #> [1] \"gergely-dev\"\n ```\n\n2. Read a `csv` file stored in S3 using a helper function:\n\n ```r\n s3_read('s3://botor/example-data/mtcars.csv', read.csv)\n #> mpg cyl disp hp drat wt qsec vs am gear carb\n #> 1 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4\n #> 2 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4\n #> ...\n ```\n\n3. Encrypt a string via KMS using a helper function:\n\n ```r\n kms_encrypt('alias/key', 'secret')\n #> [1] \"QWERTY...\"\n ```\n\n4. Get more info on the currently used AWS user calling the IAM client directly:\n\n ```r\n iam()$get_user()\n ```\n\n5. Create a new client to a service without helper functions:\n\n ```r\n ec2 <- botor_client('ec2')\n ec2$describe_vpcs()\n ```\n\n## AWS Auth\n\nThe `botor` package by default will use the credentials and related options set in [environmental variables](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#configuration) or in the `~/.aws/config` and `~/.aws/credentials` files. If you need to specify a custom profile or AWS region etc, there are various options with different complexity and flexibility:\n\n* set the related environment variable(s) before loading `botor`\n* call the `botor()` function with the relevant argument to set the config of the default session for the `botor` helper functions, eg\n\n```r\nbotor(region_name = 'eu-west-42')\n```\n\n* if you need to manage multiple sessions, then use the raw `boto3` object from the `botor` package and `boto3.session.Session` to init these custom sessions and the required clients/resources on the top of those, eg\n\n```r\nmy_custom_session1 <- boto3$Session(region_name = 'us-west-1')\nmy_custom_s3_client1 <- my_custom_session1$client('s3')\nmy_custom_session2 <- boto3$Session(region_name = 'us-west-2')\nmy_custom_s3_client2 <- my_custom_session2$client('s3')\n```\n\n## Using the raw `boto3` module\n\nThe `botor` package provides the `boto3` object with full access to the `boto3` Python SDK. Quick example on listing all S3 buckets:\n\n```r\nlibrary(botor)\ns3 <- boto3$resource('s3')\nlibrary(reticulate)\niter_next(s3$buckets$pages())\n```\n\nNote that this approach requires a stable understanding of the `boto3` Python module, plus a decent familiarity with `reticulate` as well (see eg `iter_next`) -- so you might want to rather consider using the helper functions described below.\n\n## Using the default `botor` session\n\nCalling `botor()` will provide you a default `boto3` session that is cached internally. You can always override the default session by calling `botor()` again with new arguments. See eg setting the default `boto3` session to use `us-west-2`:\n\n```r\nbotor(region_name = 'us-west-2')\nbotor()$resource('s3')\n```\n\nA great advantage of using `botor()` instead of custom sessions is that it's fork-safe. See eg:\n\n```r\nattr(botor(), 'pid')\n#> [1] 31225\nattr(botor(), 'pid')\n#> [1] 31225\n\nlapply(1:2, function(i) attr(botor(), 'pid'))\n#> [[1]]\n#> [1] 31225\n#>\n#> [[2]]\n#> [1] 31225\n\nmclapply(1:2, function(i) attr(botor(), 'pid'), mc.cores = 2)\n#> [[1]]\n#> [1] 13209\n#> \n#> [[2]]\n#> [1] 13210\n```\n\n## Convenient helper functions\n\nBesides the `botor` pre-initialized default Boto3 session, the package also provides some further R helper functions for the most common AWS actions, like interacting with S3 or KMS. Note, that the list of these functions is pretty limited for now, but you can always fall back to the raw Boto3 functions if needed. PRs on new helper functions are appreciated :)\n\nExamples:\n\n1. Listing all S3 buckets takes some time as it will first initialize the S3 Boto3 client in the background:\n\n ```r\n system.time(s3_list_buckets())[['elapsed']]\n #> [1] 1.426\n ```\n\n2. But the second query is much faster as reusing the same `s3` Boto3 resource:\n\n ```r\n system.time(s3_list_buckets())[['elapsed']]\n #> [1] 0.323\n ```\n\n3. Unfortunately, sharing the same Boto3 resource between (forked) processes is not ideal, so `botor` will take care of that by spawning new resources in the forked threads:\n\n ```r\n library(parallel)\n simplify2array(mclapply(1:4, function(i) system.time(s3_list_buckets())[['elapsed']], mc.cores = 2))\n #> [1] 1.359 1.356 0.406 0.397\n ```\n\n4. Want to speed it up more?\n\n ```r\n library(memoise)\n s3_list_buckets <- memoise(s3_list_buckets)\n simplify2array(mclapply(1:4, function(i) system.time(s3_list_buckets())[['elapsed']], mc.cores = 2))\n #> [1] 1.330 1.332 0.000 0.000\n ```\n\nThe currently supported resources and features via helper functions: https://daroczig.github.io/botor/reference/index.html\n\n## Error handling\n\nThe convenient helper functions try to suppress the boring Python traceback and provide you only the most relevant information on the error. If you want to see the full tracelog and more details after an error, call `reticulate::py_last_error()`. When working with the raw `boto3` wrapper, you may find `botor:::trypy` useful as well.\n\n```r\ns3_download_file('s3://bottttor/example-data/mtcars.csv', tempfile())\n#> Error in s3_download_file(\"s3://bottttor/example-data/mtcars.csv\", tempfile()) : \n#> Python `ClientError`: An error occurred (404) when calling the HeadObject operation: Not Found\n\ns3_read('s3://botor/example-data/mtcars2.csv', read.csv)\n#> Error in s3_download(object, t) : \n#> Python `ClientError`: An error occurred (403) when calling the HeadObject operation: Forbidden\n\nbotor(region_name = 'us-west-2')\ns3_read('s3://botor/example-data/mtcars.csv', read.csv)\n#> mpg cyl disp hp drat wt qsec vs am gear carb\n#> 1 21.0 6 160.0 110 3.90 2.620 16.46 0 1 4 4\n#> 2 21.0 6 160.0 110 3.90 2.875 17.02 0 1 4 4\n#> ...\n```\n\n## Logging\n\n`botor` uses the [`logger`](https://daroczig.github.io/logger) package to write log messages to the console by default with the following log level standards:\n\n* `TRACE` start of an AWS query (eg just about to start listing all S3 buckets in an AWS account)\n* `DEBUG` summary on the result of an AWS query (eg number of S3 buckets found in an AWS account)\n* `INFO` currently not used\n* `WARN` currently not used\n* `ERROR` something bad happened and logging extra context besides what's being returned in the error message\n* `FATAL` currently not used\n\nThe default log level threshold set to `DEBUG`. If you want to update that, use the package name for the `namespace` argument of `log_threshold` from the `logger` package, eg to enable all log messages:\n\n```r\nlibrary(logger)\nlog_threshold(TRACE, namespace = 'botor')\n\ns3_download_file('s3://botor/example-data/mtcars.csv', tempfile())\n#> TRACE [2019-01-11 14:48:07] Downloading s3://botor/example-data/mtcars.csv to '/tmp/RtmpCPNrOk/file6fac556567d4' ...\n#> DEBUG [2019-01-11 14:48:09] Downloaded 1303 bytes from s3://botor/example-data/mtcars.csv and saved at '/tmp/RtmpCPNrOk/file6fac556567d4'\n```\n\nOr update to not fire the less important messages than warnings:\n\n```r\nlibrary(logger)\nlog_threshold(WARN, namespace = 'botor')\n```\n\nYou can use the same approach to set custom (or more than one) log appenders, eg writing the log messages to files, a database etc -- check the `logger` docs for more details.\n\n## Why the name?\n\n`botor` means \"goofy\" in Hungarian. This is how I feel when looking back to all the dev hours spent on integrating the AWS Java SDK in R -- this includes `AWR.KMS`, where I ended up debugging and fixing many issues in forked processes, but `AWR.Kinesis` still rocks :)\n\nThe name also reminds you that it's not exactly `boto3`, as eg you have to use `$` instead of `.` to access methods.","jobInfo":{"package":"botor","version":"0.2.0","parsingStatus":"success","parserVersion":1,"parsedAt":"2020-02-16T20:29:47+0000"}}

Description

Usage

Arguments

Value

See Also