livy_config

From sparklyr v1.5.0 by Yitao Li

Create a Spark Configuration for Livy

Usage

livy_config(
  config = spark_config(),
  username = NULL,
  password = NULL,
  negotiate = FALSE,
  custom_headers = list(`X-Requested-By` = "sparklyr"),
  proxy = NULL,
  curl_opts = NULL,
  ...
)

Arguments

config: Optional base configuration
username: The username to use in the Authorization header
password: The password to use in the Authorization header
negotiate: Whether to use gssnegotiate method or not
custom_headers: List of custom headers to append to http requests. Defaults to list("X-Requested-By" = "sparklyr").
proxy: Either NULL or a proxy specified by httr::use_proxy(). Defaults to NULL.
curl_opts: List of CURL options (e.g., verbose, connecttimeout, dns_cache_timeout, etc, see httr::httr_options() for a list of valid options) -- NOTE: these configurations are for libcurl only and separate from HTTP headers or Livy session parameters.
...: additional Livy session parameters

Details

Extends a Spark spark_config() configuration with settings for Livy. For instance, username and password define the basic authentication settings for a Livy session.

The default value of "custom_headers" is set to list("X-Requested-By" = "sparklyr") in order to facilitate connection to Livy servers with CSRF protection enabled.

Additional parameters for Livy sessions are:

proxy_user: User to impersonate when starting the session
jars: jars to be used in this session
py_files: Python files to be used in this session
files: files to be used in this session
driver_memory: Amount of memory to use for the driver process
driver_cores: Number of cores to use for the driver process
executor_memory: Amount of memory to use per executor process
executor_cores: Number of cores to use for each executor
num_executors: Number of executors to launch for this session
archives: Archives to be used in this session
queue: The name of the YARN queue to which submitted
name: The name of this session
heartbeat_timeout: Timeout in seconds to which session be orphaned

Note that queue is supported only by version 0.4.0 of Livy or newer. If you are using the older one, specify queue via config (e.g. config = spark_config(spark.yarn.queue = "my_queue")).

Value

Named list with configuration data

Documentation reproduced from package sparklyr, version 1.5.0, License: Apache License 2.0 | file LICENSE

Community examples

Looks like there are no examples yet.