spark_config_kubernetes

0th

Percentile

Kubernetes Configuration

Convenience function to initialize a Kubernetes configuration instead of spark_config(), exposes common properties to set in Kubernetes clusters.

Usage
spark_config_kubernetes(master, image = "spark:sparklyr",
  driver = random_string("sparklyr-"), account = "spark",
  jar = "local:///opt/sparklyr/sparklyr-2.3-2.11.jar", forward = TRUE,
  ...)
Arguments
master

Kubernetes url to connect to, found by running kubectl cluster-info.

image

Container image to use to launch Spark and sparklyr. Also known as spark.kubernetes.container.image.

driver

Name of the driver pod. If not set, the driver pod name is set to "sparklyr" suffixed by id to avoid name conflicts. Also known as spark.kubernetes.driver.pod.name.

account

Service account that is used when running the driver pod. The driver pod uses this service account when requesting executor pods from the API server. Also known as spark.kubernetes.authenticate.driver.serviceAccountName.

jar

Path to the sparklyr jars; either, a local path inside the container image with the sparklyr jars copied when the image was created or, a path accesible by the container where the sparklyr jars were copied. You can find a path to the sparklyr jars by running system.file("java/", package = "sparklyr").

forward

Should ports used in sparklyr be forwarded automatically through Kubernetes? Default to TRUE which runs kubectl port-forward.

...

Additional parameters, currently not in use.

Aliases
  • spark_config_kubernetes
Documentation reproduced from package sparklyr, version 0.9.2, License: Apache License 2.0 | file LICENSE

Community examples

Looks like there are no examples yet.