No Wonder Scientists Ask Statisticians to Cook the Data

(Bloomberg Opinion) -- You won't believe the things scientists ask statisticians to do.

When statisticians responded to a survey about the kinds of requests they received from biomedical scientists, a number of them reported “inappropriate requests” like throwing out or ignoring inconvenient data points and otherwise finding ways to make things look like the scientist got a desired result, rather than the truth.

The survey was created by an epidemiologist at NYU, who, according to a story in the CBC, was shocked when a statistician told him of frequent requests to help cook the books. The epidemiologist decided to investigate whether this sort of thing was widespread.

Four particularly popular types of inappropriate requests were reported by at least 20 percent of the 390 statisticians who responded to the survey:

1. Removing or altering some data records to better support the research hypothesis
2. Interpreting the statistical findings on the basis of expectation, rather than actual results
3. Not reporting the presence of key missing data that might bias the results
4. Ignoring violations of assumptions that would change results from positive to negative

The statisticians didn’t have to say whether they accepted any of these requests. But the results did show that many medical researchers are open about such transgressions, despite being antithetical to the progress of science, and likely to contribute to misleading medical information.

The survey helps explain why so many results in biomedical research can’t be reproduced, said Steven Goodman, a professor of medicine and policy at Stanford University, and co-author of an editorial piece that ran with the survey results in the Annals of Internal Medicine. Does it mean scientists are committing misconduct, or trying to?

Goodman said that the definition of scientific misconduct is currently limited to fabricating or falsifying data and plagiarism, but the public might benefit from a wider definition that included deliberate manipulation of data and statistics. Statistical tricks may be a lot more common than outright falsification. If they both lead to false outcomes and impressions, the harm to potential patients is the same.

The survey was published in the wake of a new scandal in behavioral science — another field troubled with flawed findings. Brian Wansink, a business professor at Cornell, had been a star of academia and the TED talk circuit for his research on overeating — with lots of catchy claims, such as the notion that people eat more rice from a white plate than a black one.

Wansink had long been the target of criticism over sloppy statistics. But he aroused more concerted scrutiny when he boasted on a blog post that he’d persuaded a post-doctoral fellow to rework data from a negative study — one that didn’t find an effect they’d thought might exist. With his guidance, the fellow massaged the data until they got several results that appeared interesting enough to be published. “He put out a recipe for data dredging that is a textbook violation of basic principles,” said Goodman. Last month Cornell accused him of misconduct and fired him. He conceded minor flaws but said he had committed no serious transgressions.

Goodman says there’s tremendous pressure on scientists to get standout findings and to avoid investing time in studies that don’t show some desired effect. From scientists’ point of view, a study is a failure if it gets an answer that doesn’t help their careers. For the rest of us, a failed study is one whose results don’t correspond to the truth. Something needs to change so their interests line up better with ours.

To contact the editor responsible for this story: Philip Gray at philipgray@bloomberg.net

This column does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.

Faye Flam is a Bloomberg Opinion columnist. She has written for the Economist, the New York Times, the Washington Post, Psychology Today, Science and other publications. She has a degree in geophysics from the California Institute of Technology.

©2018 Bloomberg L.P.