livy_config
Create a Spark Configuration for Livy
Create a Spark Configuration for Livy
Usage
livy_config(
config = spark_config(),
username = NULL,
password = NULL,
negotiate = FALSE,
custom_headers = list(`X-Requested-By` = "sparklyr"),
proxy = NULL,
curl_opts = NULL,
...
)
Arguments
- config
Optional base configuration
- username
The username to use in the Authorization header
- password
The password to use in the Authorization header
- negotiate
Whether to use gssnegotiate method or not
- custom_headers
List of custom headers to append to http requests. Defaults to
list("X-Requested-By" = "sparklyr")
.- proxy
Either NULL or a proxy specified by httr::use_proxy(). Defaults to NULL.
- curl_opts
List of CURL options (e.g., verbose, connecttimeout, dns_cache_timeout, etc, see httr::httr_options() for a list of valid options) -- NOTE: these configurations are for libcurl only and separate from HTTP headers or Livy session parameters.
- ...
additional Livy session parameters
Details
Extends a Spark spark_config()
configuration with settings
for Livy. For instance, username
and password
define the basic authentication settings for a Livy session.
The default value of "custom_headers"
is set to list("X-Requested-By" = "sparklyr")
in order to facilitate connection to Livy servers with CSRF protection enabled.
Additional parameters for Livy sessions are:
proxy_user
User to impersonate when starting the session
jars
jars to be used in this session
py_files
Python files to be used in this session
files
files to be used in this session
driver_memory
Amount of memory to use for the driver process
driver_cores
Number of cores to use for the driver process
executor_memory
Amount of memory to use per executor process
executor_cores
Number of cores to use for each executor
num_executors
Number of executors to launch for this session
archives
Archives to be used in this session
queue
The name of the YARN queue to which submitted
name
The name of this session
heartbeat_timeout
Timeout in seconds to which session be orphaned
Note that queue
is supported only by version 0.4.0 of Livy or newer.
If you are using the older one, specify queue via config
(e.g.
config = spark_config(spark.yarn.queue = "my_queue")
).
Value
Named list with configuration data