How well can crowd workers judge whether individual news articles contain harmful misinformation? How can the task be structured to improve the quality and reduce partisanship of their judgments? And how should quality be measured when journalist judgments, the best gold standard that is available, are not uniform? Three university research teams independently conducted empirical studies. They met in two private workshops in 2019 and 2020, to share study designs and initial findings. Some articles and journalist judgments on those articles were shared between studies to enhance comparability of results. Facebook convened the two workshops, provided some of the articles, and funded one of the three studies.
This panel will include a member of each study team and will be moderated by the former product manager at Facebook who led the project that inspired this research. Panelists will share and compare results. The panel will also reflect on the process of convening multiple research teams to conduct related but distinct studies addressing a common question and frictions between industry and academic needs and approaches in a partnership of this kind.
Chaired by Henry Silverman
Jenny Allen (MIT)
Zeve Sanderson (NYU)
Paul Resnick (University of Michigan)