Authors: Kevin T. Greene, (Princeton University), Jacob N. Shapiro (Princeton University), Nilima Pisharody (New York University), Lucas Augusto Meyer (Microsoft), Mayana Pereira (Microsoft), Juan Lavista Ferres (Microsoft)
Title: Current engagement with unreliable sites from web search driven by navigational search
Journal: Science Advances

At a time when 66% of Americans surveyed in a Gallup poll have little to no trust in mass media, and with the rise of the phrase “fake news,” a team of researchers from Princeton University, New York University, and Microsoft set out to understand whether search engine algorithms systematically expose users to content from unreliable sites.

The study was led by Jake Shapiro, professor of politics and international affairs and Kevin Greene, an associate research scholar in the Empirical Studies of Conflict Project at the Princeton School of Public and International Affairs.

“There are growing concerns that search engines expose unwitting users to large volumes of unreliable information, leading to easy engagement with this content,” Greene said. “As search engines are among the key ways people acquire information and are highly trusted sources, these concerns are particularly important.”

In collaboration with Microsoft and its search engine, Bing, the research team analyzed approximately 14 billion search results organized into two collections. The first contained all the results returned between June and August 2022 for a weighted random sample of more than 100,000 information-seeking queries. The set included queries that exposed users at least once to any of the more than 8,000 domains rated for reliability by NewsGuard. The second collection was a simple random sample of search results between April and June 2023, leading to the same set of sites as the first sample.

NewsGuard uses a nine-point system to conclude whether websites are reliable or unreliable, including if they publish false content, correct factual errors, and effectively separate news from opinion. Sites that receive an overall score that falls below 60 out of 100 are classified as unreliable information sources.

User preferences play a role in whether an individual engages with unreliable information sources. More than 80% of engagements with unreliable information sources were because users specifically searched for these sites. The team did find that reliable information sources appeared 19-45 times more often than unreliable information sources in the algorithms.

“We found that for general informational searches, the results expose users to few unreliable information sources,” Greene said. “After identifying searches where users specifically search for the name of an unreliable domain, we are able to directly measure the role of user preferences in engagement with unreliable domains.”

As much of the engagement with unreliable sites comes from user preferences for this content, attempts to combat "infodemics" should include efforts to create demand for quality information.

The researchers suggest a need to better understand the interactions between user preferences and search engine algorithms to better solve the disinformation problem.

“Debates about the harms from online systems have largely been driven by anecdotal evidence that bad things can happen, but when researchers get their hands on large-scale data we often see that the picture is much more complicated,” Shapiro said. “Here we learned that when it comes to topics covered heavily by low-quality information sources, search engines are largely helping people get to the sources they are already looking for. That's good news, in that these systems are not taking people places they don't want to go. And it's also bad news because so many are now seeking out news sources that fail to meet the journalistic standards which supported the flowering of democracy around the world after the end of the Cold War.”