stri_enc_detect(str, filter_angle_brackets = FALSE)raw vectorsstr.
Each list element is a list with the following three named vectors
representing all guesses:
Encoding-- string; guessed encodings;NAon failure,Language-- string; guessed languages;NAif the language could
not be determined (e.g. in case of UTF-8),Confidence-- numeric in [0,1]; the higher the value,
the more confidence there is in the match;NAon failure.str and filter_angle_brackets.This is, at best, an imprecise operation using statistics and heuristics. Because of this, detection works best if you supply at least a few hundred bytes of character data that's mostly in a single language. However, Because the detection only looks at a limited amount of the input byte data, some of the returned charsets may fail to handle the all of input data. Note that in some cases, the language can be determined along with the encoding.
Several different techniques are used for character set detection. For multi-byte encodings, the sequence of bytes is checked for legal patterns. The detected characters are also check against a list of frequently used characters in that encoding. For single byte encodings, the data is checked against a list of the most commonly occurring three letter groups for each language that can be written using that encoding.
The detection process can be configured to optionally ignore
HTML or XML style markup (using
This function should most often be used for byte-marked input strings,
especially after loading them from text files and before the main
conversion with stri_encode.
The input encoding is of course not taken into account here, even
if marked.
The following table shows all the encodings that can be detected:
If you have some initial guess on language and encoding, try with
stri_enc_detect2.
stri_enc_detect2;
stri_enc_isascii;
stri_enc_isutf16be,
stri_enc_isutf16le,
stri_enc_isutf32be,
stri_enc_isutf32le;
stri_enc_isutf8;
stringi-encodingf <- rawToChar(readBin("test.txt", "raw", 100000))
stri_enc_detect(f)Run the code above in your browser using DataLab