Learn R Programming

syuzhet (version 1.0.0)

get_tokens: Word Tokenization

Description

Parses a string into a vector of word tokens.

Usage

get_tokens(text_of_file, pattern = "\\W")

Arguments

text_of_file
A Text String
pattern
A regular expression for token breaking

Value

A Character Vector of Words