Safe anytime live and leading interim meta-analysis

In this talk we introduce the Anytime Live and Leading Interim (ALL-IN) meta-analysis approach that allows experimenters to adaptively design multi-lab replication experiments in a safe manner. In this context, safe refers to the fact that the procedure comes with explicit guarantees regarding the tolerable type I error rate *during* data collection. This approach to meta-analysis is based on a so-called meta e-value that combines the currently available evidence across all studies. At any moment in time, experimenters can safely consult this meta e-value to decide whether it is worthwhile to (i) extend the meta-analysis with another (replication) study, (ii) continue recruiting participants in the currently active studies, or (iii) stop the data collection of all studies, because the combined evidence is already compelling. It is important to note, that despite this data-driven approach leading to interdependent studies, the statistical inferences from ALL-IN meta-analyses remain valid. This is unfortunately not the case for conventional meta-analyses based on p-values and confidence intervals, because, when used adaptively, they over-inflate the type I error, thus, have a high chance of mistaking random noise for structural effects. Statistical reliable use of conventional methods, therefore, requires experimenters to confine themselves to rigid designs, in which a meta-analysis is only conducted retrospectively once all --either too few, or too many-- studies are completed. The ALL-IN procedure frees experimenters from the statistical shackles imposed by conventional methods and empowers them with a more flexible, efficient, and safe approach to conducting meta-analyses.