Spark Jar Task
spark_jar_task(main_class_name, parameters = list())The full name of the class containing the main method
to be executed. This class must be contained in a JAR provided as a library.
The code must use SparkContext.getOrCreate to obtain a Spark context;
otherwise, runs of the job fail.
Named list. Parameters passed to the main method. Use Task parameter variables to set parameters containing information about job runs.
Other Task Objects:
email_notifications(),
libraries(),
new_cluster(),
notebook_task(),
pipeline_task(),
python_wheel_task(),
spark_python_task(),
spark_submit_task()