{"Package":"sparkwarc","Type":"Package","Title":"Load WARC Files into Apache Spark","Version":"0.1.1","Authors@R":"c(\nperson(\"Javier\", \"Luraschi\", email = \"javier@rstudio.com\", role = c(\"aut\", \"cre\"))\n)","Maintainer":"Javier Luraschi ","Description":"Load WARC (Web ARChive) files into Apache Spark using 'sparklyr'. This\nallows to read files from the Common Crawl project .","License":"Apache License 2.0","BugReports":"https://github.com/javierluraschi/sparkwarc","Encoding":"UTF-8","LazyData":"true","Imports":"sparklyr, DBI","RoxygenNote":"5.0.1","NeedsCompilation":"no","Packaged":"2017-01-13 00:49:39 UTC; javierluraschi","Author":"Javier Luraschi [aut, cre]","Repository":"CRAN","Date/Publication":"2017-01-13 06:42:24","repoType":"cran","tarballUrl":"ftp://cran.r-project.org/pub/R/src/contrib/sparkwarc_0.1.1.tar.gz","jsonAuthors":[{"name":"Javier Luraschi","email":"javier@rstudio.com","maintainer":true}],"readme":"sparkwarc - WARC files in sparklyr\n================\n\nInstall\n=======\n\nInstall [sparkwarc from CRAN](https://cran.r-project.org/package=sparkwarc) or the dev version with:\n\n``` r\ndevtools::install_github(\"javierluraschi/sparkwarc\")\n```\n\nIntro\n=====\n\nThe following example loads a very small subset of a WARC file from [Common Crawl](http://commoncrawl.org), a nonprofit 501 organization that crawls the web and freely provides its archives and datasets to the public.\n\n``` r\nlibrary(sparkwarc)\nlibrary(sparklyr)\nlibrary(DBI)\nlibrary(dplyr)\n```\n\n ## \n ## Attaching package: 'dplyr'\n\n ## The following objects are masked from 'package:stats':\n ## \n ## filter, lag\n\n ## The following objects are masked from 'package:base':\n ## \n ## intersect, setdiff, setequal, union\n\n``` r\nsc <- spark_connect(master = \"local\", version = \"2.0.1\")\nspark_read_warc(sc, \"warc\", system.file(\"samples/sample.warc.gz\", package = \"sparkwarc\"))\n```\n\n``` sql\nSELECT count(value)\nFROM WARC\nWHERE length(regexp_extract(value, ' 0\n```\n\n| count(value) |\n|:-------------|\n| 6 |\n\n``` r\nspark_regexp_stats <- function(tbl, regval) {\n tbl %>%\n transmute(language = regexp_extract(value, regval, 1)) %>%\n group_by(language) %>%\n summarize(n = n())\n}\n```\n\n``` r\nregexpLang <- \"http-equiv=\\\"Content-Language\\\" content=\\\"(.*)\\\"\"\ntbl(sc, \"warc\") %>% spark_regexp_stats(regexpLang)\n```\n\n ## Source: query [2 x 2]\n ## Database: spark connection master=local[8] app=sparklyr local=TRUE\n ## \n ## language n\n ## \n ## 1 ru-RU 5\n ## 2 1709\n\n``` r\nspark_disconnect(sc)\n```\n\nScale\n=====\n\nBy [running sparklyr in EMR](https://aws.amazon.com/blogs/big-data/running-sparklyr-rstudios-r-interface-to-spark-on-amazon-emr/), one can configure an EMR cluster and load about **~5GB** of data using:\n\n``` r\nsc <- spark_connect(master = \"yarn-client\")\nspark_read_warc(sc, \"warc\", cc_warc(1, 1))\n\ntbl(sc, \"warc\") %>% summarize(n = n())\nspark_disconnect_all()\n```\n\nTo read the first 200 files, or about **~1TB** of data, first scale the cluster, consider maximizing resource allocation with the followin EMR config:\n\n [\n {\n \"Classification\": \"spark\",\n \"Properties\": {\n \"maximizeResourceAllocation\": \"true\"\n }\n }\n ]\n\nFollowed by loading the `[1, 200]` file range with:\n\n``` r\nsc <- spark_connect(master = \"yarn-client\")\nspark_read_warc(sc, \"warc\", cc_warc(1, 200))\n\ntbl(sc, \"warc\") %>% summarize(n = n())\nspark_disconnect_all()\n```\n\nTo read the entire crawl, about **~1PB**, a custom script would be needed to load all the WARC files.","jobInfo":{"package":"sparkwarc","version":"0.1.1","parsingStatus":"success","parserVersion":1,"parsedAt":"2020-12-16T00:31:23+0000"}}

Description

Usage

Arguments

Examples