{"Package":"wordpiece","Type":"Package","Title":"R Implementation of Wordpiece Tokenization","Version":"2.1.3","Authors@R":"c(\nperson(given = \"Jonathan\",\nfamily = \"Bratt\",\nrole = c(\"aut\", \"cre\"),\nemail = \"jonathan.bratt@macmillan.com\",\ncomment = c(ORCID = \"0000-0003-2859-0076\")),\nperson(given = \"Jon\",\nfamily = \"Harmon\",\nrole = c(\"aut\"),\nemail = \"jonthegeek@gmail.com\",\ncomment = c(ORCID = \"0000-0003-4781-4346\")),\nperson(given = \"Bedford Freeman & Worth Pub Grp LLC DBA Macmillan Learning\",\nrole = c(\"cph\"))\n)","Description":"Apply 'Wordpiece' () tokenization to input text,\ngiven an appropriate vocabulary. The 'BERT' () tokenization\nconventions are used by default.","Encoding":"UTF-8","URL":"https://github.com/macmillancontentscience/wordpiece","BugReports":"https://github.com/macmillancontentscience/wordpiece/issues","Depends":"R (>= 3.3.0)","License":"Apache License (>= 2)","RoxygenNote":"7.1.2","Imports":"dlr (>= 1.0.0), fastmatch (>= 1.1), memoise (>= 2.0.0),\npiecemaker (>= 1.0.0), rlang, stringi (>= 1.0), wordpiece.data\n(>= 1.0.2)","Suggests":"covr, knitr, rmarkdown, testthat (>= 3.0.0)","VignetteBuilder":"knitr","Config/testthat/edition":"3","NeedsCompilation":"no","Packaged":"2022-03-03 14:19:39 UTC; jonathan.bratt","Author":"Jonathan Bratt [aut, cre] (),\nJon Harmon [aut] (),\nBedford Freeman & Worth Pub Grp LLC DBA Macmillan Learning [cph]","Maintainer":"Jonathan Bratt ","Repository":"CRAN","Date/Publication":"2022-03-03 15:10:02 UTC","repoType":"cran","tarballUrl":"ftp://cran.r-project.org/pub/R/src/contrib/wordpiece_2.1.3.tar.gz","jsonAuthors":[{"name":"Jonathan Bratt","email":"jonathan.bratt@macmillan.com","maintainer":true},{"name":"Jon Harmon","email":"jonthegeek@gmail.com","maintainer":false},{"name":"Bedford Freeman & Worth Pub Grp LLC DBA Macmillan Learning ","email":null,"maintainer":false}],"readme":"\n\n\n# wordpiece\n\n\n\n[![Lifecycle:\nexperimental](https://img.shields.io/badge/lifecycle-experimental-orange.svg)](https://lifecycle.r-lib.org/articles/stages.html#experimental)\n\n\nThe goal of wordpiece is to allow for easy text tokenization using a\nwordpiece vocabulary.\n\n## Installation\n\nYou can install the released version of wordpiece from\n[CRAN](https://CRAN.R-project.org) with:\n\n``` r\ninstall.packages(\"wordpiece\")\n```\n\nAnd the development version from [GitHub](https://github.com/) with:\n\n``` r\n# install.packages(\"devtools\")\ndevtools::install_github(\"macmillancontentscience/wordpiece\")\n```\n\n# Examples\n\nThis package can be used to tokenize text for modeling. A common usecase\nwould be to tokenize all text in a data.frame or other tibble.\n\n``` r\nlibrary(wordpiece)\nlibrary(dplyr, warn.conflicts = FALSE)\ndf_tokenized <- tibble(\n text = c(\n \"I like tacos.\",\n \"I like apples with cheese.\",\n \"The unaffable coder wrote incorrect examples.\"\n )\n) %>% \n mutate(\n tokens = wordpiece_tokenize(text)\n )\n\ndf_tokenized\n#> # A tibble: 3 x 2\n#> text tokens \n#> \n#> 1 I like tacos. \n#> 2 I like apples with cheese. \n#> 3 The unaffable coder wrote incorrect examples. \ndf_tokenized$tokens[[1]]\n#> i like ta ##cos . \n#> 1045 2066 11937 13186 1012\n```\n\n## Code of Conduct\n\nPlease note that the wordpiece project is released with a [Contributor\nCode of\nConduct](https://contributor-covenant.org/version/2/0/CODE_OF_CONDUCT.html).\nBy contributing to this project, you agree to abide by its terms.\n\n## Disclaimer\n\nThis is not an officially supported Macmillan Learning product.\n\n## Contact information\n\nQuestions or comments should be directed to Jonathan Bratt\n() and Jon Harmon\n().","jobInfo":{"package":"wordpiece","version":"2.1.3","parsingStatus":"success","parserVersion":1,"parsedAt":"2022-03-03T16:32:13+0000"}}

Description

Usage

Arguments

Value