ft_regex_tokenizer
Feature Tranformation -- RegexTokenizer (Transformer)
A regex based tokenizer that extracts tokens either by using the provided
regex pattern to split the text (default) or repeatedly matching the regex
(if gaps is false). Optional parameters also allow filtering tokens using a
minimal length. It returns an array of strings that can be empty.
Usage
ft_regex_tokenizer(x, input_col, output_col, gaps = TRUE,
min_token_length = 1L, pattern = "\\s+", to_lower_case = TRUE,
uid = random_string("regex_tokenizer_"), ...)Arguments
- x
A
spark_connection,ml_pipeline, or atbl_spark.- input_col
The name of the input column.
- output_col
The name of the output column.
- gaps
Indicates whether regex splits on gaps (TRUE) or matches tokens (FALSE).
- min_token_length
Minimum token length, greater than or equal to 0.
- pattern
The regular expression pattern to be used.
- to_lower_case
Indicates whether to convert all characters to lowercase before tokenizing.
- uid
A character string used to uniquely identify the feature transformer.
- ...
Optional arguments; currently unused.
Value
The object returned depends on the class of x.
spark_connection: Whenxis aspark_connection, the function returns aml_transformer, aml_estimator, or one of their subclasses. The object contains a pointer to a SparkTransformerorEstimatorobject and can be used to composePipelineobjects.ml_pipeline: Whenxis aml_pipeline, the function returns aml_pipelinewith the transformer or estimator appended to the pipeline.tbl_spark: Whenxis atbl_spark, a transformer is constructed then immediately applied to the inputtbl_spark, returning atbl_spark
See Also
See http://spark.apache.org/docs/latest/ml-features.html for more information on the set of transformations available for DataFrame columns in Spark.
Other feature transformers: ft_binarizer,
ft_bucketizer,
ft_chisq_selector,
ft_count_vectorizer, ft_dct,
ft_elementwise_product,
ft_feature_hasher,
ft_hashing_tf, ft_idf,
ft_imputer,
ft_index_to_string,
ft_interaction, ft_lsh,
ft_max_abs_scaler,
ft_min_max_scaler, ft_ngram,
ft_normalizer,
ft_one_hot_encoder, ft_pca,
ft_polynomial_expansion,
ft_quantile_discretizer,
ft_r_formula,
ft_sql_transformer,
ft_standard_scaler,
ft_stop_words_remover,
ft_string_indexer,
ft_tokenizer,
ft_vector_assembler,
ft_vector_indexer,
ft_vector_slicer, ft_word2vec