These functions build record pairs and finally comparison patterns
by which these pairs are later classified as links or non-links. They make up
the initial stage in a Record Linkage process after possibly
normalizing the data. Two general
scenarios are reflected by the two functions: compare.dedup
works on a
single data set which is to be deduplicated, compare.linkage
is intended
for linking two data sets together.
Data sets are represented as data frames or matrices (typically of type
character), each row representing one record, each column representing one
field or attribute (like first name, date of birth...). Row names are not
retained in the record pairs. If an identifier other than row number is
needed, it should be supplied as a designated column and excluded from
comparison (see note on exclude
below).
Each element of blockfld
specifies a set of columns in which two
records must agree to be included in the output. Each blocking definition in
the list is applied individually, the sets obtained
thereby are combined by a union operation.
If blockfld
is FALSE
, no blocking will be performed,
which leads to a large number of record pairs
(\(\frac{n(n-1)}{2}\) where \(n\) is the number of
records).
As an alternative to blocking, a determined number of n_match
matches
and n_non_match
non-matches can be drawn if identity
or
identity1
and identity2
are supplied. This is relevant for generating training sets for the supervised classificators (see trainSupv
).
Fields can be excluded from the linkage process by supplying their column index in the vector exclude
, which is especially useful for external identifiers. Excluded fields can still be used for blocking, also with phonetic code.
Phonetic codes and string similarity measures are supported for enhanced detection of misspellings. Applying a phonetic code leads to a binary values, where 1 denotes equality of the generated phonetic code. A string comparator leads to a similarity value in the range \([0,1]\).
String comparison is not allowed on a field for which a phonetic code is generated. For phonetic encoding functions included in the package, see phonetics. For the included string comparators, see jarowinkler
and levenshteinSim
.
Please note that phonetic code and string metrics can slow down the generation of comparison patterns significantly.
User-defined functions for phonetic code and string comparison can be supplied via the arguments phonfun
and strcmpfun
. phonfun
is expected to have a single character argument (the string to be transformed) and must return a character value with the encoded string.
strcmpfun
must have as arguments the two strings to be compared and return a similarity value in the range \([0,1]\), with 0 denoting the lowest and 1 denoting the highest degree of similarity. Both functions must be fully vectorized to work on matrices.