The journal Science has announced that it is checking the statistical analyses reported in papers that are published in the journal. Effective July 1 a Statistical Board of Reviewing Editors will be performing oversight on the statistics and data analysis reported in papers that are submitted for publication in Science. The purpose is to provide better assurance that the interpretation of data is substantiated correctly.
Not every paper will be subjected to the additional scrutiny of statistical analyses. The journal’s Board of Reviewing Editors will recommend some papers to be reviewed in-depth by the Statistical Board of Reviewing Editors. It is expected that the papers receiving the extra scrutiny will be those with more complicated data analysis needs. Papers that are reporting results based on a simple T-test, for example, will likely not receive the extra review. A T-test is a relatively simple test to determine if the means of two groups of data are significantly different.
In the beginnings of the scientific era many scientific investigations were observational and did not test hypotheses. For example, the early biologists, then called naturalists, were observing different life forms and classifying them without setting up formal experiments to determine how or why life systems worked. Research being carried out in the modern age of electronic technology usually produces vast quantities of numerical data and these numbers require statistical analysis. The advent of electronic calculators allowed more sophisticated quantitative analysis, usually carried out by graduate students under the supervision of their professors. The development of computers with large capacities for data storage and data processing have greatly accelerated the number and scale of scientific studies that could be implemented.
Since statistical software packages have become available, many scientists who are not trained in statistical analysis can crunch numbers and produce a statistical result. With the push of a button, a p value can be obtained in a few seconds. A p value is a number that is calculated to indicate the probability of rejecting the null hypothesis of a study question when the hypothesis is actually true. Failure to reject the null hypothesis means there is no statistical significance. The editors of Science have had too many experiences with authors making honest mistakes or analyzing their data incorrectly and have come to the realization that there needs to be an additional reviewing step for data analysis.
In the era of “big data,” the number of numbers that can be part of an analysis is mind-boggling. Most researchers have the means to collect data that is stored directly by a computer and then they just need to push a few buttons to get a statistical result. If one way does not produce the expected result, the pressing of a few more buttons just might do the trick. There may not be any idea of cheating or “massaging” the numbers but trial and error of what the statistical package offers might be too tempting.
Sometimes a study in a published paper is the first of its kind and subsequent attempts to replicate the study fail to produce the same result. Improper data analysis may be the reason behind this. The way that a scientific study is validated is through a peer review process. Some scientific journals employ peers to review a paper and decide whether the study deserves publication. The peer review process in scientific journals was put in place to offer some legitimacy to research studies.
Some journals are considered to be more respected than others and Science is thought of as one of the most respected journals published in the U.S. It is not easy to get a paper published in the journal Science and now the extra checking of stats in papers it publishes will help to ensure its position as a top scientific journal.
By Margaret Lutze