Comparing hypothesis tests using regions of support

Hypothesis testing is one of the most widely used tools in inferential statistics. Yet, hypothesis tests — be it frequentist or Bayesian — have their respective problems and can cause sever misinterpretations. We argue that one reason for these persistent problems is the following discrepancy: While hypothesis tests are explicit on which parameter-values are theoretically contained in each hypothesis, they are usually not explicit on which parameter-values would in a practical setting lead (most likely) to which test outcome. For example, certain small 'true' effects although deviating from the typical point-null hypothesis will in most cases lead to Bayes Factors supporting the null hypothesis depending on the sample size (or, more generally, precision). To make these test-characteristics explicit we introduce the concept of Regions of Support (ROS). ROS can serve both as a check for researchers’ expectations as well as a comparison of different tests. We evaluate standard Bayesian and frequentist point-null tests as well as interval (equivalence) tests on a simple, two independent samples setting. Interestingly, for interval tests our ROS analysis finds that Bayes factors suffer from an undesirable bias towards the equivalence hypothesis. We argue that other methods such as the Bayesian highest density interval (HDI) with region of practical equivalence (ROPE) or its frequentist analogue (confidence interval with ROPE) do not show this bias and might be preferable. With that, we demonstrate the diagnostic value ROS can have and hope that — due to its general applicability to any test — it will find its way into researchers' statistical toolboxes.