text_tokenize_words

text_tokenize_words.default

string

<p>whether or not token as well as non tokens shall be returned.</p>

non_token

<p>A wrapper to text_tokenize that tokenizes text into words.
Since using text_tokenize()'s option non_token might slow things
down considerably this one purpose wrapper is a little more clever
than the general implementation and hence much faster.</p>

Base R already ships with string handling capabilities 'out-
of-the-box' but lacks streamlined function names and workflow. The
'stringi' ('stringr') package on the other hand has well named functions,
extensive Unicode support and allows for a streamlined workflow. On the other
hand it adds dependencies and regular expression interpretation between base R
functions and 'stringi' functions might differ. This packages aims at providing
a solution to the use case of unwanted dependencies on the one hand but the need
for streamlined text processing on the other. The packages' functions are solely
based on wrapping base R functions into 'stringr'/'stringi' like function names.
Along the way it adds one or two extra functions and last but not least provides
all functions as generics, therefore allowing for adding methods for other text
structures besides plain character vectors.

text_tokenize_words: generic to tokenize text into words

Description

Usage

Arguments