Prior predictive entropy as a measure of model complexity

In science, when we are faced with the problem of choosing between two different accounts of a phenomenon, we are told to choose the simplest one. However, it is not always clear what a “simple” model is. Model selection criteria (\emph{e.g.,} the BIC) typically define model complexity as a function of the number of parameters in a model, or some other function of the parameter space. Here we present an alternative based on the prior predictive distribution. We argue that we can measure the complexity of a model by the entropy of its predictions before looking at the data. This can lead to surprising findings that are not well explained by thinking of model complexity in terms of parameter spaces. In particular, we use a simple choice rule as an example to show that the predictions of a nested model can have a higher entropy in comparison to its more general counterpart. Finally, we show that the complexity in a model’s predictions is a function of the experimental design.