Analyzing variability in instance-based learning model predictions using recurrence quantification analysis

Model variability is important: if systematic variation in model predictions does not reflect systematic variation in human behavior, the model's ability to describe, predict, and explain behavior is in question. We demonstrate a method to compare variation in model predictions to variation in human behavior in a dynamic decision making task. Dynamic decisions are a sequence of inter-dependent choices in changing environments, where human choices may systematically change over time. We can characterize these changes with a qualitative and quantitative visual analytics approach, recurrence quantification analysis (RQA). RQA visualizes (with recurrence plots) and describes (with recurrence statistics) recurring states in sequences of observations. We compared human choice sequences in a dynamic decision making task to predictions of an instance-based learning (IBL) model, a memory-based model of choice with two parameters (noise and decay). Specifically, we generated predictions using two parameterizations of the IBL model: one using default noise and decay parameters from the ACT-R cognitive architecture, another using the average of noise and decay parameters from IBL models fit to human data at the individual level. We compared the recurrence statistic distributions of the human data and both parameterizations. We find ACT-R default parameters predict more decision makers with less trial-to-trial change in choices than in human data. In contrast, the averaged parameters predict more decision makers with more trial-to-trial change in choices than in human data. RQA provides new tools for assessing model predictions, and a new source of evidence for demonstrating that models successfully characterize sequences of human choice.