Large language models (LLMs) are helping millions of users to learn and write about a diversity of issues. In doing so, LLMs may expose users to new ideas and perspectives, or reinforce existing knowledge and user opinions. This creates concerns about political bias in LLMs, and how these biases might influence LLM users and society. In his talk, Rottger will first discuss why measuring political biases in LLMs is difficult, and why most evidence so far should be approached with skepticism. Using the Political Compass Test as a case study, he will demonstrate critical issues of robustness and ecological validity when applying such tests to LLMs.

Second, he will present his team’s approach to building a more meaningful evaluation dataset called IssueBench, to measure biases in how LLMs write about political issues. He will describe the steps they took to make IssueBench realistic and robust. Then, he will outline their results from testing state-of-the-art LLMs with IssueBench, including clear evidence for issue bias, striking similarities in biases across models, and strong alignment with Democrat over Republican voter positions on a subset of issues.

Bio:

Paul Rottger is a postdoctoral researcher in the MilaNLP Lab at Bocconi University, working on evaluating and improving the alignment and safety of large language models (LLMs), as well as measuring their societal impacts. For his recent work in this area, he won Outstanding Paper at ACL and Best Paper at NeurIPS D&B. Before coming to Milan, he completed his Ph.D. at the University of Oxford, where he worked on LLMs for hate speech detection. During his Ph.D., he also co-founded Rewire, a start-up building AI for content moderation, which was acquired by another large online safety company in 2023.

In-person attendance is open to Princeton University faculty, staff and students. This talk will be open to the public via Zoom(Link is external) (Link opens in new window)(Link is external) (Link opens in new window). It will be recorded and posted here, on the CITP YouTube channel, and on the Princeton University Media Central channel.

If you need an accommodation for a disability please contact Jean Butcher at butcher@princeton.edu at least one week before the event.

Sponsorship of an event does not constitute institutional endorsement of external speakers or views presented.