.process_mp_vocab

.process_mp_vocab.default

.process_mp_vocab.morphemepiece_vocabulary

.process_mp_vocab.integer

.process_mp_vocab.character

<p>An object of class <code>morphemepiece_vocabulary</code>.</p>

<p>Process a Morphemepiece Vocabulary for Tokenization</p>

internal

Tokenize text into morphemes. The morphemepiece algorithm uses a
lookup table to determine the morpheme breakdown of words, and falls back on a
modified wordpiece tokenization algorithm for words not found in the lookup
table.

Jonathan Bratt

morphemepiece

Morpheme Tokenization

Jon Harmon

Bedford Freeman & Worth Pub Grp LLC DBA Macmillan Learning 

.process_mp_vocab function

Process a Morphemepiece Vocabulary for Tokenization — .process_mp_vocab

Process a Morphemepiece Vocabulary for Tokenization

.process_mp_vocab: Process a Morphemepiece Vocabulary for Tokenization

Description

Usage

Arguments

Value