How to Improve the World by Scaling Good IdeasIGIER VISITING STUDENT ALBERTO BINETTI REPORTS ON THE RECENT IGIER POLICY SEMINAR WITH JOHN LIST (CHICAGO UNIVERSITY)
Empirical economic research typically focuses on establishing causal links. In his new book, The Voltage Effect, Professor John List (Chicago University) argues that to change the world research needs to do much more: to systematically assess the applicability and scalability of its findings. The book was presented at the IGIER Policy Seminar of October 3rd 2022. It provides insights on the science of scaling up ideas by combining academic research with List’s experience in decision-making roles at firms such as Uber, Lyft, and Walmart. Professors Björn Bartling (University of Zurich) and Alfonso Gambardella (Bocconi University) acted as discussants and Professor Alexia Delfino (Bocconi University) as moderator.
A pioneer in using field experiments in economics, Professor List started thinking about the science of scaling in 2014. In 2008, List and his coauthors built a pre-K school (called CHECC) in Chicago Heights, a disadvantaged neighborhood. The goal was to find new ways to improve student achievement. The results in 2014 were encouraging: there was a significant improvement in children’s cognitive skills after just 6 months. In what has always been a recurring theme in his research, the use of rigorous empirical findings to devise better policies, Professor List tried to convince policymakers to scale up the program in an attempt to improve the conditions of disadvantaged children worldwide. He received a sound “no” based on the idea that “this policy cannot work at scale.” List then asked himself: how do we determine whether a policy can or cannot work at scale? Was there a scientific approach that could go beyond anecdotal evidence and reliably explain what happens to the cost-benefit profile as scale increases? The Voltage Effect summarizes the research that List and others have carried out on these important questions.
The answer to the first question is yes: effects change when moving from a small scale to a big one, and, perhaps more surprisingly, they do so in a predictable manner. Professor List highlights 5 elements called “Vital Signs” that determine what makes an idea scalable or not. First, we need to see whether the initial results are valid and whether they are obtained on samples that reliably represent the population targeted by the intervention. Even if the results are robust, though, they may not scale if the key ingredients are not scalable themselves. Think of restaurants: those that can truly scale up base their success on their ingredients, not their chef, as the latter is not replicable. Scalable ideas must also integrate potential spillovers: children who did not attend CHECC but whose neighbors did and showed improvements comparable to those enrolled in the program. Finally, we should keep in mind that ideas can be implemented at a large scale only if economically sustainable.
What are then the secrets to “high-voltage” scaling that translate a scalable idea into a successfully applicable one? To answer such a question, Professor List first illustrates how scalability should be interpreted along with the findings of behavioral sciences, bringing scalable incentives as an example. Following the backlash of the #DeleteUber trend in 2017, which led to an overnight jump of Lyft’s market share from 5% to 30%, Uber needed a way to regain the interest of drivers. Hence, the option to tip drivers was introduced. At first, when it was rolled only for a small portion of drivers, the results were a success: drivers earned more and worked more. Then, when the rollout was completed, the increase in earnings disappeared entirely. What could explain this? Drivers simply reacted to this incentive, which could not work at a large scale, while new drivers started working less, as their hourly rate was now higher thanks to tipping. Behavioral insights also span other areas, including organizational culture and framing of problems in terms of margins instead of averages.
The discussion continued with Professor Bartling, who provided a summary and overview of the findings of the book and emphasized its reliance on scientific research and its value as teaching material. He praised the many examples present in the book, many of which were drawn from Professor List’s personal experience, as an illustration of the consequences of misunderstanding how scalability works.
By leveraging his expertise in strategic management, Professor Gambardella provided an insightful perspective on scaling as model misspecification. Many ideas fail to scale up as they do not take into account the actual complexity of the world beforehand. Following this interpretation of model misspecification, the science of scaling resides in designing theories that explain decisions for which no data is available and in generating experiments that can validate (or disprove) these theories.
In a world of contradictions, where poverty and education disparities persist despite the efforts of countless scholars in coming up with mechanisms in an attempt to improve such conditions, the missing key of the puzzle seems to be scalability. The Voltage Effect aims at providing a scientific guide to it. As Professor List said when introducing the book, “the only way to make the world better is to do it at scale.”
by Alberto Binetti