Learn R
sparkwarc (version 0.1.1)

spark_read_warc: Reads a WARC File into Apache Spark

Description

Reads a WARC (Web ARChive) file into Apache Spark using sparklyr.

Usage

spark_read_warc(sc, name, path, repartition = 0L, memory = TRUE,
  overwrite = TRUE, group = FALSE, parse = FALSE, ...)

Arguments

sc

An active spark_connection.

name

The name to assign to the newly generated table.

path

The path to the file. Needs to be accessible from the cluster. Supports the "hdfs://", "s3n://" and "file://" protocols.

repartition

The number of partitions used to distribute the generated table. Use 0 (the default) to avoid partitioning.

memory

Boolean; should the data be loaded eagerly into memory? (That is, should the table be cached?)

overwrite

Boolean; overwrite the table with the given name if it already exists?

group

TRUE to group by warc segment. Currently supported only in HDFS and uncompressed files.

parse

TRUE to parse warc into tags, attribute, value, etc.

...

Additional arguments reserved for future use.

Examples

Run this code
# NOT RUN {
library(sparklyr)
sc <- spark_connect(master = "spark://HOST:PORT")
df <- spark_read_warc(
  sc,
  system.file("samples/sample.warc", package = "sparkwarc"),
  repartition = FALSE,
  memory = FALSE,
  overwrite = FALSE
)

spark_disconnect(sc)

# }

Run the code above in your browser using DataCamp Workspace