txt_remove_

Remove prefixed underscore unicode character 'LOWER ONE EIGHTH BLOCK' (U+2581)

Unsupervised text tokenizer allowing to perform byte pair encoding and unigram modelling.
Wraps the 'sentencepiece' library <https://github.com/google/sentencepiece> which provides a language independent tokenizer to split text in words and smaller subword units.
The techniques are explained in the paper "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing" by Taku Kudo and John Richardson (2018) <doi:10.18653/v1/D18-2012>.
Provides as well straightforward access to pretrained byte pair encoding models and subword embeddings trained on Wikipedia using 'word2vec',
as described in "BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages" by Benjamin Heinzerling and Michael Strube (2018) <http://www.lrec-conf.org/proceedings/lrec2018/pdf/1049.pdf>.

Jan Wijffels

sentencepiece

Text Tokenization using Byte Pair Encoding and Unigram Modelling

BNOSAC 

Google Inc. 

The Abseil Authors 

Kenton Varda (Google Inc.) 

Sanjay Ghemawat (Google Inc.) 

Jeff Dean (Google Inc.) 

Laszlo Csomor (Google Inc.) 

Wink Saville (Google Inc.) 

Jim Meehan (Google Inc.) 

Chris Atenasio (Google Inc.) 

Jason Hsueh (Google Inc.) 

Anton Carver (Google Inc.) 

Maxim Lifantsev (Google Inc.) 

Susumu Yata 

Daisuke Okanohara 

Yuta Mori 

Benjamin Heinzerling 

txt_remove_ function

<dl><dt>x</dt>
<dd>a character vector</dd>
<dt>replacement</dt>
<dd>character string how to replace the underscore. Defaults to the empty string.</dd></dl>

Arguments

Remove prefixed underscore — txt_remove_

<dl>

<dt>x</dt>
<dd>a character vector</dd>


<dt>replacement</dt>
<dd>character string how to replace the underscore. Defaults to the empty string.</dd>

</dl>

Remove prefixed underscore

txt_remove_: Remove prefixed underscore

Description

Usage

Value

Arguments

Examples