`step_tfidf` creates a *specification* of a recipe step that will convert a list of tokens into multiple variables containing the Term frequency-inverse document frequency of tokens.
step_tfidf(recipe, ..., role = "predictor", trained = FALSE,
columns = NULL, vocabulary = NULL, res = NULL, smooth_idf = TRUE,
norm = "l1", sublinear_tf = FALSE, prefix = "tfidf",
skip = FALSE, id = rand_id("tfidf"))# S3 method for step_tfidf
tidy(x, ...)
A recipe object. The step will be added to the sequence of operations for this recipe.
One or more selector functions to choose variables. For `step_tfidf`, this indicates the variables to be encoded into a list column. See [recipes::selections()] for more details. For the `tidy` method, these are not currently used.
For model terms created by this step, what analysis role should they be assigned?. By default, the function assumes that the new columns created by the original variables will be used as predictors in a model.
A logical to indicate if the recipe has been baked.
A list of tibble results that define the encoding. This is `NULL` until the step is trained by [recipes::prep.recipe()].
A character vector of strings to be considered.
The words that will be used to calculate the term frequency will be stored here once this preprocessing step has be trained by [prep.recipe()].
TRUE smooth IDF weights by adding one to document frequencies, as if an extra document was seen containing every term in the collection exactly once. This prevents division by zero.
A character, defines the type of normalization to apply to term vectors. "l1" by default, i.e., scale by the number of words in the document. Must be one of c("l1", "l2", "none").
A logical, apply sublinear term-frequency scaling, i.e., replace the term frequency with 1 + log(TF). Defaults to FALSE.
A character string that will be the prefix to the resulting new variables. See notes below.
A logical. Should the step be skipped when the recipe is baked by [recipes::bake.recipe()]? While all operations are baked when [recipes::prep.recipe()] is run, some operations may not be able to be conducted on new data (e.g. processing the outcome variable(s)). Care should be taken when using `skip = TRUE` as it may affect the computations for subsequent operations.
A character string that is unique to this step to identify it.
A `step_tfidf` object.
An updated version of `recipe` with the new step added to the sequence of existing steps (if any).
Term frequency-inverse document frequency is the product of two statistics. The term frequency (TF) and the inverse document frequency (IDF).
Term frequency is a weight of how many times each token appear in each observation.
Inverse document frequency is a measure of how much information a word gives, in other words, how common or rare is the word across all the observations. If a word appears in all the observations it might not give us that much insight, but if it only appear in some it might help us differentiate the observations.
The IDF is defined as follows: idf = log(# documents in the corpus) / (# documents where the term appears + 1)
The new components will have names that begin with `prefix`, then the name of the variable, followed by the tokens all seperated by `-`. The new variables will be created alphabetically according to token.
[step_hashing()] [step_tf()] [step_tokenize()]
# NOT RUN {
library(recipes)
data(okc_text)
okc_rec <- recipe(~ ., data = okc_text) %>%
step_tokenize(essay0) %>%
step_tfidf(essay0)
okc_obj <- okc_rec %>%
prep(training = okc_text, retain = TRUE)
bake(okc_obj, okc_text)
tidy(okc_rec, number = 2)
tidy(okc_obj, number = 2)
# }
Run the code above in your browser using DataLab