Category Archives: Experimental Psychology

The Importance of Replication in Psychological Science [2/2]

THE OCCURRENCE OF REPLICATION

“ONLY 1.07% of published psychology articles were replications!”

Recent accusations of scientific fraud and shoddy methodology increased public scrutiny of psychological research. This scrutiny revitalized awareness within the field. Researchers discussed and debated various solutions, hoping to reduce error and scientific fraud. These discussions frequently pointed to one solution, replication.

I previously introduced a recent study directed by Dr. Matthew Makel: Replications in Psychology Research: How Often Do They Really Occur? Dr. Makel and a team of Duke investigators researched the previously unmeasured prevalence of replication in psychology. He acknowledges the recent public condemnation, and a need to redirect the field towards more rigorous science. Consistently testing research results reveals errors and strengthens procedural rigor. Frequent replication may provide psychology the key course correction needed to reverse public opinion and repair research methods.

The study conducted an exploratory investigation. The team hoped to provide an overview and the starting point for future discussion and direction. The Duke team investigated the entire publication history of 100 psychology journals. The team examined the journals for the total number of replications, and changes in the amount of replication over time. Dr. Makel also looked for factors contributing to successful replications.

The investigation team uncovered an alarmingly low frequency of replication in psychology. They also uncovered a substantial difference in successful reproductions when the replication was conducted by a member of the original research team.

Of the 320,000 articles, only 5,000 articles used a variation of the term replication, about 1.6%. The investigation examined a randomly selected sample of 500 articles using the term. Of those 500, only 68% were actual replications. Dr. Makel adjusted the data with this  new information, it revealed only 1.07% of published psychology articles were replications! While there seems to be an increase in the last ten years, the total number of replication studies was still low, at 1.8%. Dr. Makel believes this low occurrence of replication may indicate psychology actively deters replication.

NOT ALL REPLICATIONS ARE CREATED EQUAL

Like original research, replications must provide proper methodology and an accurate analysis of the data. By expanding the original data, replications assist researchers in recognizing clumps of data perhaps leading to false-positive results. However, replication research is not without its own inherent concerns. An important aspect of replication that the research is conducted by an independent team. The Duke study provides a clear distinction between the success rate of replications involving a a member of the original research team, and  those conducted by completely independent research teams.

Replications involving an overlap of team members from the original research presents increased risk of confirmation bias. The Duke team found 92% of replications with overlapping team members successfully reproduced the findings of the original study. Only 65% of replications by independent research teams confirmed the results of the orignal study. If a key aspect of preventing bias is independent replication, using Dr. Makel’s data, only .49% of psychological studies are independent research replications!

Dr. Makel expressed other minor concerns affecting replication. Confirmation bias also affects independent researchers (Moonesinghe, et al. 2007). Researchers may confirm an expected result due to perceived consensus. Further, variations in research resources and researcher capabilities creates concern about replication equivalency, and the precision provided by research replication. However, these concerns due not provide cause to disregard replication. A little bit of replication goes a long way. Improving the frequency of independent replications provides “self-correction” for original research, as well as the replicated research.

THE POSITIVE EFFECTS OF BIAS

Why does a low occurrence replication matter?

Most psychological scientist positively regard replication. Replication seems to be a simple enough concept, and has been valued by all of science as a sort of standard candle, or what Dr. Makel refers to as the scientific “Supreme Court.” Replication expands the original data and examines research methodology. Reproducibility helps research achieve validity, diminishing concerns about false-positive results, and decreasing experimenter and measurement biases. Replication contributes to consensus.

Unfortunately, the field emphasizes a conflicting ideal for original research. Researchers must fill gaps in the literature in order to advance knowledge. Further, Researchers involved in replication are not looked on as advancing knowledge, even though many agree,“a field that replicates its work is rigorous and scientifically sound.”  Even as a graduate student contemplating my theses, I am expected to find a gap in the literature, and develop original research. Just exploring new information ignores the benefit of independent replications. Reproducing or dismissing past results serves the literature by strengthening conclusions or revealing false-positive results. It is like  filling a pot-hole with stone, and failing to pave the cavity with cement. The ideal of originality provides a greater abundance of knowledge, but also generates greater uncertainty.

The low replication rate provides few opportunities to correct false findings, and may explain the high rate of successful research in psychology.

As I discussed in my last post,  the ESP research by Stanford psychologist, Dr.Daryl Blem, exemplifies intentional scientific fraud in contemporary psychology (and exacerbated the current public scrutiny). However, not all fraud is intentional. Psychologist tend to find significance in the research more frequently than any other discipline (Fanelli 2010 and Ioannidis 2005).  A year before the Duke study, an article in the Wall Street Journal explored the difficultly various disciplines experienced in reproducing research results. When replications do occur, they often fail to reproduce results from the original study, which may lead to a file-draw effect, in which researcher will not submit findings for publication in order to maintain the status quo. Scientist are pressured to succeed. Money and recognition flows more freely towards research establishing significance. Further research is needed, but these findings may be indicative of pervasive confirmation bias, supported by Makel’s findings about researcher overlap.

The infograph below displays the amount of published positive findings throughout science. Psychology publishes more studies supporting the tested hypothesis tan any other discipline. (Faneli 2010)

CONCLUSIONS

Dr. Makel suggests researching methods for determining what research should be replicated. I believe it is important instead to attempt to alter the attitude of the field towards replication. Any scientific field ignoring replication may enter a state where most published research is false (Ioannidis 2005). Psychology must emerge from the current scrutiny with a more rigorous adherence to empirical research and replication. Psychologist acknowledge the importance of replication, yet maintain an environment overwhelmed by the pressure to publish original research, and expand the literature. Readjustment is needed alongside a redirection of resources. Replication as part of increasing rigorous adherence to the scientific method, will provide a more precise measure of psychological therapies and practices.

Psychological science seems pointed in the right direction and as a discipline is in a unique position. The Duke reported the replication rate doubled since 2000. As a young science psychological science may be more open to a revolutionary redirection emphasizing replication. The historical dismissal of psychological science by more established disciplines, and the recent scrutiny and concerns of scientific fraud, further encourages response and a rigorous redirection. Further, Dr. Makel’s investigation provides the field the initial measurement necessary to encourage and compare future results from replication. His research establishes a baseline and will allow the field to test methods supporting replication, as well as the effectiveness of replication frequency.  By establishing a baseline and an emphasis for research replication, the field establishes a foundation best able to distinguish the signals from an overwhelming cacophony of white noise.

QUESTIONS FOR DISCUSSION?

• How do the rates of replication vary throughout the different disciplines within the field?
• Should schools encourage research replication for undergrads and graduate students, as their capstone or thesis?
  • Will this remove the stigma of replication as un-original and failing to advance knowledge?

[TL;DR]

• There is increased public scrutiny of psychology
• Dr. Makel investigated 100 years of psychology publications; only 1.07% of  studies were replications
• Replication is not encouraged in psychology
• Psychological studies achieve significance more than any other discipline
• Replication diminishes coincidence and intentional/unintentional error
• Psychology requires replication to reverse public concern, and recover as a more rigorous scientific endeavor.

 

Interested in more information about replication…?

CHECK OUT THE SCIENCE EXCHANGE REPRODUCIBLITY PROJECT 

STAY TUNED! Look for NEW posts every MONDAY! For updates, follow GETPSYCHED @getpsychedblog or on google+!