word.alignment (version 1.0.9)

word_alignIBM1: Computing One-to-Many Word Alignment Using IBM Model 1 for a Given Parallel Corpus

Description

For a given sentence-aligned parallel corpus, it aligns words in each sentence pair. Moreover, it calculates the expected length and vocabulary size of each language (source and taget language) and also shows word translation probability as a data.table.

Usage

word_alignIBM1(file_train1, file_train2, nrec = -1, 
              encode.sorc = 'unknown', encode.trgt = 'unknown',
              iter = 5, minlen = 5, maxlen = 40, 
              removePt = TRUE, all = FALSE, 
              dtfile_path = NULL, f1 = "fa", e1 = "en", 
              result_file = 'myResultIBM1', input = FALSE)
# S3 method for alignment
print(x, ...)

Arguments

file_train1

the name of source language file in training set.

file_train2

the name of target language file in training set.

nrec

the number of sentences to be read. If -1, it considers all sentences.

encode.sorc

encoding to be assumed for the source language. If the value is "latin1" or "UTF-8" it is used to mark character strings as known to be in Latin-1 or UTF-8. For more details please see scan function.

encode.trgt

encoding to be assumed for the target language. If the value is "latin1" or "UTF-8" it is used to mark character strings as known to be in Latin-1 or UTF-8. For more details please see scan function.

iter

the number of iterations for IBM Model 1.

minlen

a minimum length of sentences.

maxlen

a maximum length of sentences.

removePt

logical. If TRUE, it removes all punctuation marks.

all

logical. If TRUE, it considers the third argument (lower = TRUE) in culf function.

dtfile_path

if NULL (usually for the first time), a data.table will be created contaning cross words of all sentences with their matched probabilities. It saves into a file named as a combination of f1, e1, nrec and iter as 'f1.e1.nrec.iter.RData'.

If specific file name is set, it will be read and continue the rest of the function, i.e. : finding the word alignments.

f1

it is a notation for the source language (default = 'fa').

e1

it is a notation for the target language (default = 'en').

result_file

the output results file name.

input

logical. If TRUE, the output can be used by mydictionary and align_test.set functions.

x

an object of class 'alignment'.

…

further arguments passed to or from other methods.

Value

word_alignIBM1 returns an object of class 'alignment'.

An object of class 'alignment' is a list containing the following components:

If input = TRUE

dd1

A data.table.

Otherwise,

n1

An integer.

n2

An integer.

time

A number. (in second/minute/hour)

iterIBM1

An integer.

expended_l_source

A non-negative real number.

expended_l_target

A non-negative real number.

VocabularySize_source

An integer.

VocabularySize_target

An integer.

word_translation_prob

A data.table.

word_align

A list of one-to-many word alignment for each sentence pair (it is as word by word).

number_align

A list of one-to-many word alignment for each sentence pair (it is as numbers).

aa

A matrix (n*2), where n is the number of remained sentence pairs after preprocessing.

Details

Here, word alignment is a map of the target language to the source language.

The results depend on the corpus. As an example, we have used English-Persian parallel corpus named Mizan which consists of more than 1,000,000 sentence pairs with a size of 170 Mb. If all sentences are considered, it takes about 50.96671 mins using a computer with cpu: intel Xeon X5570 2.93GHZ and Ram: 8*8 G = 64 G and word alignment is good. But for the 10,000 first sentences, the word alignment might not be good. In fact, it is sensitive to the original translation type (lexical or conceptual). The results can be found at

http://www.um.ac.ir/~sarmad/word.a/example_wordalignIBM1.pdf

References

Koehn P. (2010), "Statistical Machine Translation.", Cambridge University, New York.

Lopez A. (2008), "Statistical Machine Translation.", ACM Computing Surveys, 40(3).

Peter F., Brown J. (1990), "A Statistical Approach to Machine Translation.", Computational Linguistics, 16(2), 79-85.

Supreme Council of Information and Communication Technology. (2013), Mizan English-Persian Parallel Corpus. Tehran, I.R. Iran. Retrieved from http://dadegan.ir/catalog/mizan.

http://statmt.org/europarl/v7/bg-en.tgz

See Also

align_test.set, Symmetrization, mydictionary, scan

Examples

# NOT RUN {
# Since the extraction of  bg-en.tgz in Europarl corpus is time consuming, 
# so the aforementioned unzip files have been temporarily exported to 
# http://www.um.ac.ir/~sarmad/... .
# }
# NOT RUN {
w1 = word_alignIBM1 ('http://www.um.ac.ir/~sarmad/word.a/euro.bg',
                     'http://www.um.ac.ir/~sarmad/word.a/euro.en',
                      nrec = 30, encode.sorc = 'UTF-8')
                 
w2 = word_alignIBM1 ('http://www.um.ac.ir/~sarmad/word.a/euro.bg',
                     'http://www.um.ac.ir/~sarmad/word.a/euro.en',
                      nrec = 30, encode.sorc = 'UTF-8', removePt = FALSE)
# }