link_records() and links_wf_probabilistic() are functions to implement deterministic, fuzzy or probabilistic record linkage.
link_records() compares every record-pair in one instance,
while links_wf_probabilistic() is a wrapper function of links and so compares batches of record-pairs in iterations.
link_records() is more thorough in the sense that it compares every combination of record-pairs.
This makes it faster but is memory intensive, particularly if there's no blocking_attribute.
In contrast, links_wf_probabilistic() is less memory intensive but takes longer since it does it's checks in batches.
The implementation of probabilistic record linkage is based on Fellegi and Sunter (1969) model for deciding if two records belong to the same entity.
In summary, record-pairs are created and categorised as matches and non-matches (attr_threshold) with user-defined functions (cmp_func).
Two probabilities (m and u) are then estimated for each record-pair to score the matches and non-matches.
The m-probability is the probability that matched records are actually from the same entity i.e. a true match,
while u-probability is the probability that matched records are not from the same entity i.e. a false match.
By default, u-probabilities are calculated as the frequency of each value of an attribute however,
they can also be supplied along with m-probabilities.
Record-pairs whose total score are above a certain threshold (score_threshold) are assumed to belong to the same entity.
Agreement (match) and disagreement (non-match) scores are calculated as described by Asher et al. (2020).
For each record pair, an agreement for attribute \(i\) is calculated as;
$$\log_{2}(m_{i}/u_{i})$$
For each record pair, a disagreement score for attribute \(i\) is calculated as;
$$\log_{2}((1-m_{i})/(1-u_{i}))$$
where \(m_{i}\) and \(u_{i}\) are the m and u-probabilities for each value of attribute \(i\).
Note that each probability is calculated as a combined probability for the record pair.
For example, if the values of the record-pair have u-probabilities of 0.1 and 0.2 respectively,
then the u-probability for the pair will be 0.02.
Missing data (NA) are considered non-matches and assigned a u-probability of 0.
By default, matches and non-matches for each attribute are determined as an exact_match with a binary outcome.
Alternatively, user-defined functions (cmp_func) are used to create similarity scores.
Pairs with similarity scores within (attr_threshold) are then considered matches for the corresponding attribute.
If probabilistic is FALSE,
the sum of all similarity scores is used as the score_threshold instead of deriving one from the m and u-probabilities.
A blocking_attribute can be used to reduce the processing time by restricting comparisons to subsets of the dataset.
In link_records(), score_threshold is a convenience argument because every combination of record-pairs are returned
therefore, a new score_threshold can be selected after reviewing the final scores.
However, in links_wf_probabilistic(), the score_threshold is more important
because a final selection is made at each iteration.
As a result, links_wf_probabilistic() requires an acceptable score_threshold in advance.
To help with this, prob_score_range() can be used to return the range of scores attainable for a given set of attribute, m and u-probabilities.
Additionally, id_1 and id_2 can be used to link specific records pairs, aiding the review of potential scores.