Active experiment design in crowdsourced experiments

As experimentation in the behavioral and social sciences moves from brick-and-mortar laboratories to the web, new opportunities arise in the design of experiments. By taking advantage of the new medium, experimenters can write complex computationally mediated adaptive procedures for gathering data: algorithms. Here, we explore the consequences of adopting an algorithmic approach to experiment design. We review several active experiment designs, describing their interpretation as algorithms. We then discuss software platforms for the efficient execution of these algorithms with people. Finally, we consider how machine learning can optimize crowdsourced experiments and form the foundation of next-generation experiment design.