tidytext (version 0.1.0)

unnest_tokens: Split a column into tokens using the tokenizers package

Description

Split a column into tokens using the tokenizers package

Usage

unnest_tokens_(tbl, output_col, input_col, token = "words", to_lower = TRUE,
  drop = TRUE, ...)
unnest_tokens(tbl, output, input, token = "words", to_lower = TRUE,
  drop = TRUE, ...)

Arguments

tbl

Data frame

output_col

Output column to be created

input_col

Input column that gets split

token

Unit for tokenizing. Options are "characters", "words", "sentences", "lines", "paragraphs", and "regex". Default is "words".

to_lower

Whether to turn column lowercase

drop

Whether original input column should get dropped. Ignored if the original input and new output column have the same name.

...

Extra arguments passed on to the tokenizer

output

Output column to be created as bare name

input

Input column that gets split as bare name

Details

If the unit for tokenizing is sentences, lines, paragraphs, or regex, the entire input will be collapsed together before tokenizing.

Examples

library(dplyr)
library(janeaustenr)

d <- data_frame(txt = prideprejudice)
d

d %>%
  unnest_tokens(word, txt)

d %>%
  unnest_tokens(word, txt, token = "sentences")