Fact-checking System

User study with AI-powered fact-checking system for COVID-19 related topics | Oct 2023

HIGHLIGHTS

RESEARCH GOAL

Misinformation is an important topic in the Information Retrieval (IR) context and has implications for both system-centered and user-centered IR. While it has been established that the performance in discerning misinformation is affected by a person’s cognitive load, the variation in cognitive load in judging the veracity of news is less understood.

Previous research in cognition and discerning misinformation typically displayed one news headline per trial. However, in realistic information search scenarios, people often encounter several pieces of information in one search. Moreover, the headlines shown in the previous experiments were mostly related to political topics. Currently, misinformation not only threatens democracy but also public health regarding the situation of the COVID-19 pandemic. The aim of the study was to:

• Understand how users interact with the fact-checking system via two different user interfaces: an interactive interface and a non-interactive interface.

• Examine how the cognitive load is impacted when users interact with the fact-checking content, and how it is related to their belief change and misinformation judgment.

EXPERIMENTAL DESIGN

A controlled, within-subjects eye-tracking study was conducted in the Information eXperience usability lab at the University of Texas at Austin (N=40, 22 females). Voluntary participants interacted with two versions (user-interfaces) of a fact-checking system. Participants were pre-screened for native-level English familiarity, 20/20 vision (uncorrected or corrected), and non-expert topic familiarity of the content being shown. A Tobii TX-300 eye-tracker was used to record the participants’ eye movements. Upon completion of the experiment, each participant was compensated with USD 25.

Fact-checking system (interactive interface on the left, non-interactive on the right)

Participants interacted with a mock fact checking system, and examined 24 COVID-19 related claims in the system (12 for each interface). Each claim was shown at the top of the interface. Surrogates of five related news articles were presented below the claim, each with its corresponding news source, source reputation, news headline, and the article’s stance towards the claim. Based on the article’s stance and news source reputation, the system provided a prediction of the claim’s correctness at the bottom. The news headlines were clickable, and upon clicking, opened the news article in a new browser tab. Each claim examination consisted of viewing the claim, the headlines of the news articles, and, optionally, clicking the news articles to read them in detail. To mitigate the effect of background luminance of pupil dilation, the color and luminance of the fact-checking-system interface was kept constant during the experimental session.

FINDINGS

The participants generally spent more time

reading news than using the interface. Overall, in the interactive

interface, participants spent more time using the fact-checking

system (interface dwell time), as well as reading the underlying

news articles (news dwell time).

Total fixation count (left), total fixation duration (right) for different AOIs in two interfaces. (A: stance of article, C: predicted correctness, H: news article headlines, R:source reputation, S: news source, T: claim text).

There were more fixations and overall longer eye-dwell time in the interactive interface. The figure illustrates that most fixations and longer fixation durations were on the ‘news article headlines’ area of interest (AOI), followed by the ‘source reputation’, ‘news source’, and ‘stance of article’ AOIs. The ‘predicted correctness’ AOI had fewer fixations and shorter durations, while the ‘claim text’ AOI had the least.

Figure(a) Distribution of RPD of the news headline AOIs as

a function of headline stance (-1: headline denies the claim; 1:

headline supports the claim) and claim correctness (TRUE or

FALSE claim). (b) Distribution of RPD of the news headline

AOIs as a function of the evidence correctness (correct or

incorrect) and claim correctness (TRUE or FALSE claim).

Higher mental workload was on the news headline AOIs that denied the claims. Higher mental workload when checking TRUE claims compared to FALSE claims. Mental workload was higher when participants were checking the incorrect evidence for TRUE claims, and when checking the correct evidence for FALSE claims.

Figure(a) Distribution of RPD of the news headline AOIs

as a function of the evidence correctness (correct or incor-

rect) and the belief change in both claims (stay/to-right in

TRUE/FALSE claims). (b) Distribution of RPD of the news

headline AOIs as a function of belief-headline-consistency

(consistent or inconsistent) and the belief change in both

claims (stay/to-right in TRUE/FALSE claims).

Mental workload when reading correct evidence had small differences between belief change and claim correctness combination groups, while mental workload when reading incorrect evidence had larger differences between belief change and claim correctness combination groups. when reading the incorrect evidence, the mental workload differed more between change trend groups. Additionally, the differences between the headline-belief consistent group and inconsistent group. Mental workload in headline-belief consistent group were larger than headline-belief inconsistent group when participants maintained their correct beliefs (stay-right) in FALSE claims. Mental workload in headline-belief inconsistent group were larger than headline-belief consistent group when participants changed to the correct belief (to-right) in both TRUE and FALSE claims, and when participants maintained their correct beliefs (stay-right) in TRUE claims.