Using parameter contours to achieve more robust model estimation
Many current practices in parameter estimation and model evaluation rely on fit statistics, calculated on the basis of estimated parameterizations of competing models. However, the design of an experiment can influence the conclusions a modeler draws about the parameterizations and relative performance of these models (Broomell, Sloman, Blaha, and Chelen, 2019). We highlight the importance of mapping the model-stimulus space, i.e. understanding how the parameter-dependent predictions of a model change across different stimuli. To achieve this goal, we represent models as a topography across the stimulus space, in which adjacent contour lines are defined by adjacent parameter values. Using data simulated from models of decision-making, we show how our proposed techniques can identify conditions under which traditional parameter estimation techniques will lead to inconclusive and inconsistent results. We also discuss ways in which modelers could exploit these insights to develop experimental designs for more robust parameter estimation. In addition, we demonstrate how a better understanding of the model-stimulus space can help researchers design powerful experiments to diagnose data generated by hypothesized models. Finally, we explore the conceptual implications of representing cognitive models as a topography of the stimulus space.
Great project. Topographical consistency could be used for multiple purposes, and thus possibly have multiple formal definitions depending on the application. It could help developers evaluate design decisions of models. As in your example, probing for consistency and contours can reveal new behaviors that should be explored with models.
One question: I have problems understanding what defines a parameter contour. Is it any line or region in a plot that follows from a model with fixed parameter values? In other words, can the axes be anything as long as a fixed set of parameter values produces more than one predicted data point? Thanks for the interesting presentation!