text_tokenize_lines

text_tokenize_lines.default

string

whether or not token as well as non tokens shall be returned.

non_token

generic to tokenize text into lines
text_tokenize default

Base R already ships with string handling capabilities 'out-
of-the-box' but lacks streamlined function names and workflow. The
'stringi' ('stringr') package on the other hand has well named functions,
extensive Unicode support and allows for a streamlined workflow. On the other
hand it adds dependencies and regular expression interpretation between base R
functions and 'stringi' functions might differ. This packages aims at providing
a solution to the use case of unwanted dependencies on the one hand but the need
for streamlined text processing on the other. The packages' functions are solely
based on wrapping base R functions into 'stringr'/'stringi' like function names.
Along the way it adds one or two extra functions and last but not least provides
all functions as generics, therefore allowing for adding methods for other text
structures besides plain character vectors.

text_tokenize_lines: generic to tokenize text into lines

Description

Usage

Arguments