.new_morphemepiece_vocabulary

Character vector; the "actual" vocabulary.

vocab

List of character vectors; the split vocabulary.

vocab_split

Logical; whether the vocabulary is cased.

is_cased

Constructor for Class morphemepiece_vocabulary

internal

Tokenize text into morphemes. The morphemepiece algorithm uses a
lookup table to determine the morpheme breakdown of words, and falls back on a
modified wordpiece tokenization algorithm for words not found in the lookup
table.

.new_morphemepiece_vocabulary: Constructor for Class morphemepiece_vocabulary

Description

Usage

Arguments

Value