Boolean to distribute .libPaths()
packages to each node,
a list of packages to distribute, or a package bundle created with
spark_apply_bundle()
.
Defaults to TRUE
or the sparklyr.apply.packages
value set in
spark_config()
.
For clusters using Yarn cluster mode, packages
can point to a package
bundle created using spark_apply_bundle()
and made available as a Spark
file using config$sparklyr.shell.files
. For clusters using Livy, packages
can be manually installed on the driver node.
For offline clusters where available.packages()
is not available,
manually download the packages database from
https://cran.r-project.org/web/packages/packages.rds and set
Sys.setenv(sparklyr.apply.packagesdb = "<pathl-to-rds>")
. Otherwise,
all packages will be used by default.
For clusters where R packages already installed in every worker node,
the spark.r.libpaths
config entry can be set in spark_config()
to the local packages library. To specify multiple paths collapse them
(without spaces) with a comma delimiter (e.g., "/lib/path/one,/lib/path/two"
).