Against “the study”

A new study shows…

A new study shows a dog’s heart rate increases by 46% when the owner says “I love you.” A new study shows that pink eye may be a symptom of COVID-19 in children. A new study shows an iceberg might not have sunk the Titanic.

Science journalists are always announcing the results of the latest study. The more bizarre and controversial, the better. A recent study is, almost by definition, cutting-edge research — what better way to tap into the pulse of science? Except, the latest and greatest research is just as often wrong.

The concern is not simply with hype. Rather, the problem is the “study.” As a unit of scientific research, it leaves much to be desired, and for those who are unfamiliar with the practices of the scientific community, how to interpret a lone study can be deeply confusing. A study refers to both a research project and to the reporting of one or more research findings in a scientific journal. By design, a scientific study is a research result that represents an important enough contribution to an area of investigation to be published. That’s all it is. It does not need to be highly likely to be correct, and it does not represent the current sum total of the evidence.

The most novel and exciting research in a field is also often the most speculative. Scientists are attracted to such topics because of their potential, not because they have undergone extensive confirmation. And it is, of course, impossible for new research to have stood the test of time.

The picture one gets if they follow research from study to study is that the scientific community has commitment issues. For instance, scientists can never make up their mind whether eggs are good for you, or how much cardio you should be doing. It is easy to get the impression that science moves serially, with each study representing the current state of the evidence until a newer study comes along and supersedes it. However, studies are in fact not serial, but cumulative. Each study gets added to the total pile of the evidence, and no single study sums up the current state of the science.¹ 

One of the greatest advantages trained scientists have in interpreting research is not the ability to evaluate the specific study at hand, but to contextualize the study within a broader view of the field. This is what makes it so hard for Google warriors to approach scientific questions the way a trained scientist can, whatever their general level of education. Consider a study testing a pharmaceutical treatment for cancer done on laboratory mice. Using knowledge of the biology of the mice used and other background facts, a cancer researcher might know that the results of this study do not yet imply anything about the treatment of cancer in humans. A person without this background cannot contextualize the significance of the findings in the same way, even if they can get through the scientific paper.  

However, the study is not necessarily just a problem for non-scientists — scientists have their own concerns. In the social and biomedical sciences especially, research is extremely “study-based.” What I mean by this is that research proceeds through somewhat isolated projects looking for interesting results, wherever they may find them. Consider the vaccine trials currently underway with COVID-19. The aim is not to prove a general theory about coronavirus immunization, but to find workable vaccines. This contrasts with more theory-driven fields like physics, in which experimental research generally serves to confirm an already well-established guiding theory. This is why the physics community can agree on a 9 billion dollar piece of equipment like the Large Hadron Collider to search for specific particles. 

There is nothing inherently wrong with the more free-flowing research that occurs in fields like biomedicine and psychology, and it drives creative and productive science. But, there is a statistical concern. The more separate hypotheses not dictated by theory that are tested, the more likely it is that scientists accumulate false positives by chance alone. Relatedly, if scientists are spreading their research so wide, it is hard to get the deep and focused accumulation of evidence which occurs in more narrow, theory-driven disciplines.

Acknowledging the limitations of the study are more important now than ever, especially in the context of a global pandemic, when there is a storm of studies on COVID-19 for which science journalists feel obligated to inform the public. However, hopping from latest finding to latest finding obscures the fact that when it comes to a novel phenomenon like COVID-19, uncertainty sometimes can be the cutting edge. 

We need to place our trust not in the veracity of individual studies, but in the global community of researchers. We need to be able to trust that public health institutions and associated scientists are appropriately contextualizing their studies, extracting enduring findings, forming recommendations based on these findings, and then revising their recommendations when their understanding improves. 

Scientists can always be wrong, but there’s not a better place to bet.

¹This is at least true over the short term. Historians and philosophers of science have more serious concerns about how well this accretionary model of science holds over longer periods of time. The most famous book on the topic is Thomas Kuhn’s The Structure of Scientific Revolutions, although it is now a bit dated.

Edited by Vaishnavi Muralikrishnan and Lana Ruck