livy_config

0th

Percentile

Create a Spark Configuration for Livy

Create a Spark Configuration for Livy

Usage
livy_config(config = spark_config(), username = NULL, password = NULL,
  negotiate = FALSE, custom_headers = list(`X-Requested-By` = "sparklyr"),
  ...)
Arguments
config

Optional base configuration

username

The username to use in the Authorization header

password

The password to use in the Authorization header

negotiate

Whether to use gssnegotiate method or not

custom_headers

List of custom headers to append to http requests. Defaults to list("X-Requested-By" = "sparklyr").

...

additional Livy session parameters

Details

Extends a Spark "spark_config" configuration with settings for Livy. For instance, "username" and "password" define the basic authentication settings for a Livy session.

The default value of "custom_headers" is set to list("X-Requested-By" = "sparklyr") in order to facilitate connection to Livy servers with CSRF protection enabled.

Additional parameters for Livy sessions are:

proxy_user

User to impersonate when starting the session

jars

jars to be used in this session

py_files

Python files to be used in this session

files

files to be used in this session

driver_memory

Amount of memory to use for the driver process

driver_cores

Number of cores to use for the driver process

executor_memory

Amount of memory to use per executor process

executor_cores

Number of cores to use for each executor

num_executors

Number of executors to launch for this session

archives

Archives to be used in this session

queue

The name of the YARN queue to which submitted

queue

The name of this session

heartbeat_timeout

Timeout in seconds to which session be orphaned

Value

Named list with configuration data

Aliases
  • livy_config
Documentation reproduced from package sparklyr, version 0.8.2, License: Apache License 2.0 | file LICENSE

Community examples

Looks like there are no examples yet.