ft_sql_transformer

From sparklyr v0.8.2 by Javier Luraschi

Feature Transformation -- SQLTransformer

Implements the transformations which are defined by SQL statement. Currently we only support SQL syntax like 'SELECT ... FROM __THIS__ ...' where '__THIS__' represents the underlying table of the input dataset. The select clause specifies the fields, constants, and expressions to display in the output, it can be any select clause that Spark SQL supports. Users can also use Spark SQL built-in function and UDFs to operate on these selected columns.

Usage

ft_sql_transformer(x, statement, uid = random_string("sql_transformer_"), ...)
ft_dplyr_transformer(x, tbl, uid = random_string("dplyr_transformer_"), ...)

Arguments

x: A spark_connection, ml_pipeline, or a tbl_spark.
statement: A SQL statement.
uid: A character string used to uniquely identify the feature transformer.
...: Optional arguments; currently unused.
tbl: A tbl_spark generated using dplyr transformations.

Details

ft_dplyr_transformer() is a wrapper around ft_sql_transformer() that takes a tbl_spark instead of a SQL statement. Internally, the ft_dplyr_transformer() extracts the dplyr transformations used to generate tbl as a SQL statement then passes it on to ft_sql_transformer(). Note that only single-table dplyr verbs are supported and that the sdf_ family of functions are not.

Value

The object returned depends on the class of x.

spark_connection: When x is a spark_connection, the function returns a ml_transformer, a ml_estimator, or one of their subclasses. The object contains a pointer to a Spark Transformer or Estimator object and can be used to compose Pipeline objects.
ml_pipeline: When x is a ml_pipeline, the function returns a ml_pipeline with the transformer or estimator appended to the pipeline.
tbl_spark: When x is a tbl_spark, a transformer is constructed then immediately applied to the input tbl_spark, returning a tbl_spark

Other feature transformers: ft_binarizer, ft_bucketizer, ft_chisq_selector, ft_count_vectorizer, ft_dct, ft_elementwise_product, ft_feature_hasher, ft_hashing_tf, ft_idf, ft_imputer, ft_index_to_string, ft_interaction, ft_lsh, ft_max_abs_scaler, ft_min_max_scaler, ft_ngram, ft_normalizer, ft_one_hot_encoder, ft_pca, ft_polynomial_expansion, ft_quantile_discretizer, ft_r_formula, ft_regex_tokenizer, ft_standard_scaler, ft_stop_words_remover, ft_string_indexer, ft_tokenizer, ft_vector_assembler, ft_vector_indexer, ft_vector_slicer, ft_word2vec

Documentation reproduced from package sparklyr, version 0.8.2, License: Apache License 2.0 | file LICENSE

Community examples

Looks like there are no examples yet.