New paper in JEP General: Is representative design the key to valid assessments of people's risk preferences?

Posted on Friday, Apr 30 2021

A large body of research has documented the relatively poor psychometric properties of behavioral measures of risk taking, such as low convergent validity and poor test–retest reliability. In this project we examined the extent to which these issues may be related to violations of "representative design" – the idea that experimental stimuli should be sampled or designed such that they represent the environments to which measured constructs are supposed to generalize.

To this end, we focused on one of the most prominent behavioral measures of risk taking, the Balloon Analogue Risk Task (BART). Our analyses demonstrate that the typical implementation of the BART violates the principle of representative design, and strongly conflicts with the expectations people might have formed from real balloons. We conducted two extensive empirical studies (N = 772 and N = 632), aimed at testing the effects of improved representative designs. Indeed, participants acquired more accurate beliefs about the optimal behavior in the BART due to these task adaptions. Yet strikingly, these improvements proved to be insufficient to enhance the task's psychometric properties (e.g., convergent validity with other measures of risk preference and related constructs). We conclude that for the development of valid behavioral measurement instruments, our field has to overcome the philosophy of the "repair program" (i.e., fixing existing tasks). Instead, the development of valid task designs may require ecological assessments that identify those real-life behaviors and associated psychological processes that lab tasks are supposed to capture and generalize to.

This is a joint project with my (now former) PhD student Markus Steiner (see picture below), who has successfully defended his thesis last week – congratulations, Dr. Steiner!

Steiner, M., & Frey, R. (2021). Representative design in psychological assessment: A case study using the Balloon Analogue Risk Task (BART). Journal of Experimental Psychology: General, 150, 2117-2136. doi:10.1037/xge0001036 | PDF