unnest_tokens

0th

Percentile

Split a column into tokens using the tokenizers package

Split a column into tokens using the tokenizers package

Usage
unnest_tokens_(tbl, output_col, input_col, token = "words", to_lower = TRUE, drop = TRUE, collapse = NULL, ...)
unnest_tokens(tbl, output, input, token = "words", to_lower = TRUE, drop = TRUE, collapse = NULL, ...)
Arguments
tbl
Data frame
output_col
Output column to be created
input_col
Input column that gets split
token
Unit for tokenizing, or a custom tokenizing function. Built-in options are "words" (default), "characters", "ngrams", "skip_ngrams", "sentences", "lines", "paragraphs", and "regex". If a function, should take a character vector and return a list of character vectors of the same length.
to_lower
Whether to turn column lowercase
drop
Whether original input column should get dropped. Ignored if the original input and new output column have the same name.
collapse
Whether to combine text with newlines first in case tokens (such as sentences or paragraphs) span multiple lines. If NULL, collapses when token method is "ngrams", "skip_ngrams", "sentences", "lines", "paragraphs", or "regex"
...
Extra arguments passed on to the tokenizer, such as n and k for "ngrams" and "skip_ngrams" or pattern for "regex"
output
Output column to be created as bare name
input
Input column that gets split as bare name
Details

If the unit for tokenizing is ngrams, skip_ngrams, sentences, lines, paragraphs, or regex, the entire input will be collapsed together before tokenizing.

Aliases
  • unnest_tokens
  • unnest_tokens_
Examples

library(dplyr)
library(janeaustenr)

d <- data_frame(txt = prideprejudice)
d

d %>%
  unnest_tokens(word, txt)

d %>%
  unnest_tokens(sentence, txt, token = "sentences")

d %>%
  unnest_tokens(ngram, txt, token = "ngrams", n = 2)

d %>%
  unnest_tokens(ngram, txt, token = "skip_ngrams", n = 4, k = 2)

d %>%
  unnest_tokens(chapter, txt, token = "regex", pattern = "Chapter [\\d]")

# custom function
d %>%
  unnest_tokens(word, txt, token = stringr::str_split, pattern = " ")

Documentation reproduced from package tidytext, version 0.1.1, License: MIT + file LICENSE

Community examples

Looks like there are no examples yet.