Skip to main content

Using a probabilistic model to assist merging of large-scale administrative records

Author(s): Enamorado, Ted; Fifield, Benjamin; Imai, Kosuke

To refer to this page use:
Abstract: © 2019 American Political Science Association. Since most social science research relies on multiple data sources, merging data sets is an essential part of researchers' workflow. Unfortunately, a unique identifier that unambiguously links records is often unavailable, and data may contain missing and inaccurate information. These problems are severe especially when merging large-scale administrative records. We develop a fast and scalable algorithm to implement a canonical model of probabilistic record linkage that has many advantages over deterministic methods frequently used by social scientists. The proposed methodology efficiently handles millions of observations while accounting for missing data and measurement error, incorporating auxiliary information, and adjusting for uncertainty about merging in post-merge analyses. We conduct comprehensive simulation studies to evaluate the performance of our algorithm in realistic scenarios. We also apply our methodology to merging campaign contribution records, survey data, and nationwide voter files. An open-source software package is available for implementing the proposed methodology.
Publication Date: May-2019
Citation: Enamorado, T, Fifield, B, Imai, K. (2019). Using a probabilistic model to assist merging of large-scale administrative records. American Political Science Review, 113 (2), 353 - 371. doi:10.1017/S0003055418000783
DOI: doi:10.1017/S0003055418000783
ISSN: 0003-0554
EISSN: 1537-5943
Pages: 1 - 40
Type of Material: Journal Article
Journal/Proceeding Title: American Political Science Review
Version: Author's manuscript

Items in OAR@Princeton are protected by copyright, with all rights reserved, unless otherwise indicated.