We were with some friends the other night, and one of the topics of discussion was about food. My friend said a particular food was good for you, but everyone did not agree. Then he said, “a study showed” his claim to be true. We all read about studies. They are in the news every day. A study shows this, a study shows that. The question we should be asking is whether the conclusion of any particular study is actually true. A recent study of studies found most published research findings are in fact false.
Richard Harris, a longtime science correspondent at NPR News, published an excellent article in the Wall Street Journal on this topic titled The Breakdown in Biomedical Research. The article was based on his new book, Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope and Wastes Billions.
Mr. Harris cited numerous reasons why most studies are flawed. In cancer research, cells of a particular type of caner are used to study the effect of various drugs. Frequently, the cancer cells used are contaminated. In a large study of breast cancer with widely published results, researchers later found the cells used in the research were actually from melanoma, not breast cancer. Some research turns out to be fraudulent, but in most cases researchers are honest and work very diligently. However, government and academia apply a lot of pressure to achieve research results which can make headlines. Getting research studies published often leads to competitive positions in academia or government, and money is often involved one way or another.
An additional factor which can lead to false results is the N, which means the number of samples or participants are in the study. For studies to be statistically valid, they must have an adequate sample size. For numerous reasons, getting an adequate number of samples or research participants can be difficult. In addition to the N, the samples must be randomly chosen so no bias is introduced in the study. A group of studies on promising drugs found the research often used fewer than a dozen mice per experiment. Compounding the inadequate N, the studies did not properly avoid significant sources of bias, such as genetic variability in the mice. An independent research institute redid the studies properly and found “none of the dozen or so drugs, despite the initial findings, showed any real promise”.
Mr. Harris believes real change in professional habits and culture in biomedical research is underway. He believes the result will be accelerated progress toward new drugs and improved treatments. I hope he is right. He also believes, “It’s still smart to cast a wary eye on sensational results from the latest study.”
