Racial bias and policing made headlines last year after a study examining records of fatal police shootings claimed white officers were no more likely to shoot racial minorities than nonwhite officers. There was one problem: The study was based on a logical fallacy.
The original research counted the numbers of fatal shootings, but never considered how often civilians encounter police officers, an essential ingredient to justifying its central claim.
The findings sparked a fiery debate among other academics, including two professors from Princeton University, who raised mathematical concerns about the study’s approach. Today, they published their critique as a letter in the Proceedings of the National Academy of Sciences (PNAS).
The pair — Dean Knox, assistant professor of politics, and Jonathan Mummolo, assistant professor of politics and public affairs— outline a number of serious flaws in the original study, which was featured in PNAS on Aug. 6, 2019.
For the original study, researchers from Michigan State University and the University of Maryland compiled data on 900 fatal U.S. police shootings from crowdsourced databases. They then contacted each police department, gathering information about the race of the police officers responsible for each fatality.
The researchers then used the shootings data to predict the race of victims. Specifically, they showed that when the shooting officer was black, the civilian who was shot was more likely to be black than white. And controlling for attributes of the county in which shootings occurred, “the relationship between officer and civilian race was attenuated or eliminated.” The authors interpreted these results as evidence that white officers are not biased against black civilians.
Yet, Knox and Mummolo show that the authors' conclusions hinge on the assumption that black and white officers encounter black and white civilians in equal numbers. Knox and Mummolo show this formally, but a simple thought experiment also illustrates the conceptual problem.
Imagine a white officer encounters 90 white civilians and 10 black, while a black officer encounters 90 black civilians and 10 white, both under identical circumstances. If both officers shot five black and nine white civilians, the results would — according to the reasoning of the original study — appear to show no racial bias.
However, once encounter rates are taken into account, one would see the white officer shot 50% of the black civilians he or she saw while the black officer shot 5.6%. Therefore, failing to incorporate information on encounter rates masks racial bias.
The data from the original study also only includes records of shootings, ignoring all other police-civilian encounters. And it doesn’t take into account that all police officers — white and nonwhite — could, in theory, be biased in shooting black men.
These critiques have a number of implications on the way data is collected for research and the benchmarks used for analysis.
“New data on police behavior are coming online all the time, and that is great from a research standpoint,” Mummolo said. “But all the data in the world do not negate the need to adhere to basic tenets of statistical theory and causal inference. Studies of racial bias demand the utmost rigor, and when blatant mistakes are made, they need to be quickly corrected. To allow provably false results to stand unchallenged risks confusing the public and lawmakers on one of the most pressing policy issues of our time.”
After their critique was initially rejected by PNAS, Mummolo published a Twitter thread highlighting the mathematical problems associated with the original study in August 2019. The team also posted their analysis on the preprint server SSRN.
Responding to the critique, the authors of the original paper released a formal response, stating their claim about the relative probability of white and black officers shooting racial minorities was not supported, but adding that the original findings, “as described in that manuscript, largely stand unchanged.”
Knox and Mummolo then appealed the rejection at PNAS, and their critique was accepted.
The letter, “Making inferences about racial disparities in political violence,” appeared in PNAS on Jan. 21.