tidytext (version 0.1.1)

unnest_tokens: Split a column into tokens using the tokenizers package

Description

Split a column into tokens using the tokenizers package

Usage

unnest_tokens_(tbl, output_col, input_col, token = "words", to_lower = TRUE, drop = TRUE, collapse = NULL, ...)
unnest_tokens(tbl, output, input, token = "words", to_lower = TRUE, drop = TRUE, collapse = NULL, ...)

Arguments

tbl
Data frame
output_col
Output column to be created
input_col
Input column that gets split
token
Unit for tokenizing, or a custom tokenizing function. Built-in options are "words" (default), "characters", "ngrams", "skip_ngrams", "sentences", "lines", "paragraphs", and "regex". If a function, should take a character vector and return a list of character vectors of the same length.
to_lower
Whether to turn column lowercase
drop
Whether original input column should get dropped. Ignored if the original input and new output column have the same name.
collapse
Whether to combine text with newlines first in case tokens (such as sentences or paragraphs) span multiple lines. If NULL, collapses when token method is "ngrams", "skip_ngrams", "sentences", "lines", "paragraphs", or "regex"
...
Extra arguments passed on to the tokenizer, such as n and k for "ngrams" and "skip_ngrams" or pattern for "regex"
output
Output column to be created as bare name
input
Input column that gets split as bare name

Details

If the unit for tokenizing is ngrams, skip_ngrams, sentences, lines, paragraphs, or regex, the entire input will be collapsed together before tokenizing.

Examples


library(dplyr)
library(janeaustenr)

d <- data_frame(txt = prideprejudice)
d

d %>%
  unnest_tokens(word, txt)

d %>%
  unnest_tokens(sentence, txt, token = "sentences")

d %>%
  unnest_tokens(ngram, txt, token = "ngrams", n = 2)

d %>%
  unnest_tokens(ngram, txt, token = "skip_ngrams", n = 4, k = 2)

d %>%
  unnest_tokens(chapter, txt, token = "regex", pattern = "Chapter [\\d]")

# custom function
d %>%
  unnest_tokens(word, txt, token = stringr::str_split, pattern = " ")