Spark Machine Learning Library (MLlib)

Overview

sparklyr provides bindings to Spark's distributed machine learning library. In particular, sparklyr allows you to access the machine learning routines provided by the spark.ml package. Together with sparklyr's dplyr interface, you can easily create and tune machine learning workflows on Spark, orchestrated entirely within R.

sparklyr provides three families of functions that you can use with Spark machine learning:

• Machine learning algorithms for analyzing data (ml_*)
• Feature transformers for manipulating individual features (ft_*)
• Functions for manipulating Spark DataFrames (sdf_*)

An analytic workflow with sparklyr might be composed of the following stages. For an example see Example Workflow.

1. Perform SQL queries through the sparklyr dplyr interface,
2. Use the sdf_* and ft_* family of functions to generate new columns, or partition your data set,
3. Choose an appropriate machine learning algorithm from the ml_* family of functions to model your data,
4. Inspect the quality of your model fit, and use it to make predictions with new data.
5. Collect the results for visualization and further analysis in R

Algorithms

Spark's machine learning library can be accessed from sparklyr through the ml_* set of functions:

Function	Description
ml_kmeans	K-Means Clustering
ml_linear_regression	Linear Regression
ml_logistic_regression	Logistic Regression
ml_survival_regression	Survival Regression
ml_generalized_linear_regression	Generalized Linear Regression
ml_decision_tree	Decision Trees
ml_random_forest	Random Forests
ml_gradient_boosted_trees	Gradient-Boosted Trees
ml_pca	Principal Components Analysis
ml_naive_bayes	Naive-Bayes
ml_multilayer_perceptron	Multilayer Perceptron
ml_lda	Latent Dirichlet Allocation
ml_one_vs_rest	One vs Rest

Formulas {.toc-ignore}

The ml_* functions take the arguments response and features. But features can also be a formula with main effects (it currently does not accept interaction terms). The intercept term can be omitted by using -1.

Options {.toc-ignore}

The Spark model output can be modified with the ml_options argument in the ml_* functions. The ml_options is an experts only interface for tweaking the model output. For example, model.transform can be used to mutate the Spark model object before the fit is performed.

Transformers

A model is often fit not on a dataset as-is, but instead on some transformation of that dataset. Spark provides feature transformers, facilitating many common transformations of data within a Spark DataFrame, and sparklyr exposes these within the ft_* family of functions. These routines generally take one or more input columns, and generate a new output column formed as a transformation of those columns.

Function	Description
ft_binarizer	Threshold numerical features to binary (0/1) feature
ft_bucketizer	Bucketizer transforms a column of continuous features to a column of feature buckets
ft_discrete_cosine_transform	Transforms a length NN real-valued sequence in the time domain into another length NN real-valued sequence in the frequency domain
ft_elementwise_product	Multiplies each input vector by a provided weight vector, using element-wise multiplication.
ft_index_to_string	Maps a column of label indices back to a column containing the original labels as strings
ft_quantile_discretizer	Takes a column with continuous features and outputs a column with binned categorical features
ft_sql_transformer	Implements the transformations which are defined by a SQL statement
ft_string_indexer	Encodes a string column of labels to a column of label indices
ft_vector_assembler	Combines a given list of columns into a single vector column

Utilities

Functions for interacting with Spark ML model fits.

Function	Description
ml_binary_classification_eval	Calculates the area under the curve for a binary classification model.
ml_classification_eval	Calculates performance metrics (i.e. f1, precision, recall, weightedPrecision, weightedRecall, and accuracy) for binary and multiclass classification model.
ml_tree_feature_importance	Calculates variable importance for decision trees (i.e. decision trees, random forests, gradient boosted trees).
ml_saveload	Save and load model fits. For use with scoring models across platforms (e.g. using a model as an estimator in a Spark application). These functions are currently experimental and not yet ready for production use.

Extensions

Functions for creating custom wrappers to other Spark ML algorithms.

Function	Description
ensure	Enforces Specific Structure for R Objects.
ml_create_dummy_variables	Given a column in a Spark DataFrame, generate a new Spark DataFrame containing dummy variable columns.
ml_model	Creates an ML Model Object.
ml_options	Provides Options for Spark.ML Routines.
ml_prepare_dataframe	Prepares a Spark DataFrame for Spark ML Routines.
ml_prepare_response_features_intercept	Pre-process / normalize the inputs typically passed to a Spark ML routine.

Examples

We will use the iris data set to examine a handful of learning algorithms and transformers. The iris data set measures attributes for 150 flowers in 3 different species of iris.

K-Means Clustering

Use Spark's K-means clustering to partition a dataset into groups. K-means clustering partitions points into k groups, such that the sum of squares from points to the assigned cluster centers is minimized.

Linear Regression

Use Spark's linear regression to model the linear relationship between a response variable and one or more explanatory variables.

Logistic Regression

Use Spark's logistic regression to perform logistic regression, modeling a binary outcome as a function of one or more explanatory variables.

PCA

Use Spark's Principal Components Analysis (PCA) to perform dimensionality reduction. PCA is a statistical method to find a rotation such that the first coordinate has the largest variance possible, and each succeeding coordinate in turn has the largest variance possible.

Random Forest

Use Spark's Random Forest to perform regression or multiclass classification.

SDF Partitioning

Split a Spark DataFrame into training, test datasets.

FT String Indexing

Use ft_string_indexer and ft_index_to_string to convert a character column into a numeric column and back again.

SDF Mutate

sdf_mutate is provided as a helper function, to allow you to use feature transformers. For example, the previous code snippet could have been written as:

Example Workflow

Let's walk through a simple example to demonstrate the use of Spark's machine learning algorithms within R. We'll use ml_linear_regression to fit a linear regression model. Using the built-in mtcars dataset, we'll try to predict a car's fuel consumption (mpg) based on its weight (wt), and the number of cylinders the engine contains (cyl).

First, we will copy the mtcars dataset into Spark.

Transform the data with Spark SQL, feature transformers, and DataFrame functions.

1. Use Spark SQL to remove all cars with horsepower less than 100
2. Use Spark feature transformers to bucket cars into two groups based on cylinders
3. Use Spark DataFrame functions to partition the data into test and training

Then fit a linear model using spark ML. Model MPG as a function of weight and cylinders.

The summary() suggests that our model is a fairly good fit, and that both a cars weight, as well as the number of cylinders in its engine, will be powerful predictors of its average fuel consumption. (The model suggests that, on average, heavier cars consume more fuel.)

Let's use our Spark model fit to predict the average fuel consumption on our test data set, and compare the predicted response with the true measured fuel consumption. We'll build a simple ggplot2 plot that will allow us to inspect the quality of our predictions.

Although simple, our model appears to do a fairly good job of predicting a car's average fuel consumption.

As you can see, we can easily and effectively combine feature transformers, machine learning algorithms, and Spark DataFrame functions into a complete analysis with Spark and R.