RBERT (version 0.1.11)

FullTokenizer: Construct objects of FullTokenizer class.

Description

Construct objects of FullTokenizer class.

Usage

FullTokenizer(vocab_file, do_lower_case = TRUE)

Arguments

vocab_file

Path to text file containing list of vocabulary tokens.

do_lower_case

Logical: do we convert everything to lowercase?

Value

An object of class FullTokenizer.

Examples

# NOT RUN {
f_tokenizer <- FullTokenizer("vocab.txt", TRUE)
# }