Last chance! 50% off unlimited learning
Sale ends in
Reads a parquet stream as a Spark dataframe stream.
stream_read_parquet(sc, path, name = NULL, columns = NULL,
options = list(), ...)
A spark_connection
.
The path to the file. Needs to be accessible from the cluster. Supports the "hdfs://", "s3a://" and "file://" protocols.
The name to assign to the newly generated stream.
A vector of column names or a named vector of column types.
A list of strings with additional options.
Optional arguments; currently unused.
Other Spark stream serialization: stream_read_csv
,
stream_read_json
,
stream_read_kafka
,
stream_read_orc
,
stream_read_scoket
,
stream_read_text
,
stream_write_console
,
stream_write_csv
,
stream_write_json
,
stream_write_kafka
,
stream_write_memory
,
stream_write_orc
,
stream_write_parquet
,
stream_write_text
# NOT RUN {
sc <- spark_connect(master = "local")
sdf_len(sc, 10) %>% spark_write_parquet("parquet-in")
stream <- stream_read_parquet(sc, "parquet-in") %>% stream_write_parquet("parquet-out")
stream_stop(stream)
# }
# NOT RUN {
# }
Run the code above in your browser using DataLab