Prior Setting in Practice: Strategies and Rationales Used in Choosing Prior Distributions for Bayesian Analysis
Abhraneel Sarma, Matthew Kay
ACM Human Factors in Computing Systems (CHI) 2020
Different types of visualizations shown to participants for prior elicitation. We visualize the prior density (left), prior density on the response scale (center) and prior predictive density (right); the chosen prior is also written in the common parametric form (for example, Normal(0, 1)). Users' use a widget interfaceto specify their prior.
Abstract
Bayesian statistical analysis is steadily growing in popularity and use. Choosing priors is an integral part of Bayesian inference. While there exist extensive normative recommendations for prior setting, little is known about how priors are chosen in practice. We conducted a survey (N = 50) and interviews (N = 9) where we used interactive visualizations to elicit prior distributions from researchers experienced with Bayesian statistics and asked them for rationales for those priors. We found that participants’ experience and philosophy influence how much and what information they are willing to incorporate into their priors, manifesting as different levels of informativeness and skepticism. We also identified three broad strategies participants use to set their priors: centrality matching, interval matching, and visual probability mass allocation. We discovered that participants’ understanding of the notion of “weakly informative priors”—a commonly-recommended normative approach to prior setting—manifests very differently across participants. Our results have implications both for how to develop prior setting recommendations and how to design tools to elicit priors in Bayesian analysis.