Economics: The Empirical Revolution Needs a Dose of Theory

(Bloomberg View) -- A lot of my economist colleagues are enthusiastic about the promise of the empirical revolution in our field. They acknowledge the methodological problems of the young movement, but see the solution as more and better empirics. I, on the other hand, think that what’s needed is a larger dose of theory.

The benefits of the empirical revolution are clear. Prior to the 1980s, researchers spent most of their energy erecting an edifice of clever but untested theories. The advent of cheap desktop computers gave the next generation the power to put those theories to the test.

The edifice cracked. In 1997, the scholars David Card and Alan Krueger presented research showing that minimum-wage increases did not lead to lower employment, as prevailing economic theory would have predicted. The subtext was clear. Theory was fiction. Empirics were fact.

At roughly the same time, worries had been cropping up about the techniques of empirical science. These concerns first arose in the field of nutrition, where researchers typically used widely accepted double-blind placebo trials to test their theories. Subjects were randomly assigned to say take, say, a fish oil supplement or a placebo. Neither the subjects nor the doctors studying them would be told who received the real fish oil.

Over the course of the study, doctors would measure cardiovascular risk factors like blood pressure or cholesterol count. Any differences that emerged between the groups could plausibly be attributed to the nutritional effects of the fish oil. 

Studies similar to this were indeed performed. The fish oil groups were found to have improved risk factors and fish oil supplements were heralded as a new wonder treatment for heart disease. Only they weren’t. Subsequent trials repeatedly revealed no significant effect of fish oil on health. 

One might argue that this disappointing result actually showed that the scientific system was working. Faulty early results were contradicted by subsequent research. The problem of emprics was cured by more empirics.

Except it’s not always so simple. Brian Wansink of Cornell University has been one of the most successful food researchers in history, studying what worked and didn’t work to change eating behavior. His papers have been cited over 24,000 times and he was appointed by the administration of President Barack Obama to oversee the production of 2010 agriculture department dietary guidelines.

His reputation as a researcher was spectacular, as measured by his h-index, a statistic used to gauge how influential a scholar has been. Wansink’s rating at the time was over 70; by comparison, the London School of Economics estimates that the average professor of economics has a score of 7.6 and that the highest feasible score in economics is probably 45 to 50.

Few people were more influential in nutrition policy or research. Yet in March of last year, the Netherlands-based researcher Tim van der Zee released the Wansink Dossier, a list of errors and improprieties in Wansink’s published work. The list alleges a variety of infractions in at least 50 of Wansink’s papers, which were collectively cited 4,000 times.

Van der Zee’s investigation began not because Wansink’s work was refuted by later work, but because Wansink wrote a blog post in 2016 praising a young researcher for refusing to give up in her attempt to confirm his discoveries. Reporting later revealed that Wansink coached her to parse and re-parse her data, running statistical experiment after statistical experiment until she produced the finding he expected.

She compiled and published four papers based on that work, all of which cited and supported Wansink’s claims. What makes this such a crucial case is that Wansink wasn’t intentionally ginning up bad science. He was bragging to the world about what he thought was good science.

Experts in scientific methodology have called Wansink’s work “storytelling rather than science.” Indeed, Wansink had a story to tell. Almost all dedicated scientists do. And while Wansink is an extreme outlier, it’s quite common for researchers to report findings consistent with their favored stories. This is how schools of thought grow up.

It’s also why theory performs such an important function. The purpose of theory is to lay one’s story bare; to present it to the world in a common, often mathematical framework that any other trained theorist can interpret and criticize. Those criticisms help refine the theory, but eventually they lead to disputes that can only be settled by empirical investigations.

The effort to confirm or disprove a theory is what drives the most powerful empirical investigations. For example, quantum theory posited the existence of an elusive subatomic particle dubbed the Higgs boson. Efforts to demonstrate its existence inspired construction of the world’s most powerful particle accelerator, the Large Hadron Collider. In 2012, the particle was indeed discovered.

Card and Krueger’s type of findings have been reproduced by one school of economists. An opposing school led by David Neumark of the University of California at Irvine has argued for the opposite conclusion, that a minimum wage does result in job losses. The camps continue to pump out research, mostly confirming their respective points of view.

What’s needed is a solid new theory to replace the conventional wisdom that was upended by Card and Krueger. It would outline how the two schools’ methods are consistent with the results that they find, and derive a third test that could theoretically distinguish between the two. The next generation of empirical researchers could dedicate their talents to making that test a reality.

This column does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.

Karl W. Smith is director of economic research at the Niskanen Center and founder of the blog Modeled Behavior.

To contact the author of this story: Karl W. Smith at ksmith@niskanencenter.org.

To contact the editor responsible for this story: Jonathan Landman at jlandman4@bloomberg.net.

For more columns from Bloomberg View, visit http://www.bloomberg.com/view.

©2018 Bloomberg L.P.